Walkthrough
llamagithub_triageAIvllmmachine-learningend-to-end-use-casesllama2LLMllama-cookbookPythonfinetuningpytorchlangchain
Export
[1]
Automatic Issues Triaging with Llama
We utilize an off-the-shelf Llama model to analyze, generate insights, and create a report for better understanding of the state of a repository.
This notebook walks you through the tool's working.
Setup
[ ]
[2]
Set access keys and tokens
Set your GitHub token for API calls. Some privileged information may not be available if you don't have push-access to the target repository.
Set your groq token for inference. Get one at https://console.groq.com/keys
[3]
Set target repo and period to analyze
[4]
Repo name: pytorch/pytorch Period: 2024-08-24 - 2024-08-29
Fetch issues from the repository
Use the github API to retrieve issues (including the full discussion on them) and store it in a dataframe.
[5]
Fetching issues on pytorch/pytorch from 2024-08-24 to 2024-08-29
Fetched 112 issues on pytorch/pytorch from 2024-08-24 to 2024-08-29 Data saved to output/pytorch/pytorch/2024-08-24_2024-08-29/issues.csv
[Showing 5 out of 112 rows]
repo_name number html_url \
0 pytorch/pytorch 134384 https://github.com/pytorch/pytorch/issues/134384
1 pytorch/pytorch 134385 https://github.com/pytorch/pytorch/issues/134385
2 pytorch/pytorch 134388 https://github.com/pytorch/pytorch/issues/134388
3 pytorch/pytorch 134390 https://github.com/pytorch/pytorch/issues/134390
4 pytorch/pytorch 134391 https://github.com/pytorch/pytorch/issues/134391
closed num_comments created_at \
0 False 3 2024-08-24T00:26:26Z
1 False 2 2024-08-24T00:43:18Z
2 False 2 2024-08-24T05:04:42Z
3 False 2 2024-08-24T07:10:28Z
4 True 1 2024-08-24T07:54:45Z
discussion
0 Torch compile stochastically fails with FileNo...
1 FlopCounterMode doesn't support HOP\n### 🐛 Des...
2 Remove redundant copy in functional collective...
3 This send_object_list and recv_object_list met...
4 ProcessGroupNCCL::barrier()'s device id guessi...
Use Llama to generate the annotations for this data
We use 2 prompts defined in config.yaml to annotate the issues with additional information that can help triagers and repo maintainers:
-
parse_issues: generate annotations and other metadata basd on the contents in the issue thread. -
assign_categorytags each issue with the most relevant category (from a list of categories specified in the prompt's output schema).
[8]
Prompt for generating annotations: You are an expert maintainer of an open source project. Given some discussion threads, you must respond with a report in JSON. Your response should only contain English, and you may translate if you can. Prompt for categorizing issues: You are the lead maintainer of an open source project. Given a list of issues, generate a JSON that categorizes the issues by common themes. For every theme include a description and cite the relevant issue numbers. All issues must be categorized into at least one theme. Issues can be categorized into: ['Cloud Compute', 'Installation and Environment', 'Model Loading', 'Model Fine-tuning and Training', 'Model Conversion', 'Model Inference', 'Distributed Training and Multi-GPU', 'Performance and Optimization', 'Quantization and Mixed Precision', 'Documentation', 'CUDA Compatibility', 'Model Evaluation and Benchmarking', 'Miscellaneous', 'Invalid']
We run inference on these prompts along with the issues data in issues_df
[11]
Generating annotations for 112 Data saved to output/pytorch/pytorch/2024-08-24_2024-08-29/annotated_issues.csv Data saved to output/pytorch/pytorch/2024-08-24_2024-08-29/annotated_issues.csv
-
The annotations include new metadata like
summary,possible_causes,remediationsthat can help triagers quickly understand and diagnose the issue. -
Annotations like
issue_type,component,themescan help identify the right POC / maintainer to address the issue. -
Annotations like
severity,op_expertiseandsentimentcan help gauge the general pulse of developers.
[13]
metadata generated by LLM: {'possible_causes', 'summary', 'severity', 'issue_type', 'sentiment', 'themes', 'component', 'op_expertise', 'remediations'}
[Showing 5 out of 112 rows]
repo_name number html_url \
0 pytorch/pytorch 134384 https://github.com/pytorch/pytorch/issues/134384
1 pytorch/pytorch 134385 https://github.com/pytorch/pytorch/issues/134385
2 pytorch/pytorch 134388 https://github.com/pytorch/pytorch/issues/134388
3 pytorch/pytorch 134390 https://github.com/pytorch/pytorch/issues/134390
4 pytorch/pytorch 134391 https://github.com/pytorch/pytorch/issues/134391
closed num_comments created_at \
0 False 3 2024-08-24T00:26:26Z
1 False 2 2024-08-24T00:43:18Z
2 False 2 2024-08-24T05:04:42Z
3 False 2 2024-08-24T07:10:28Z
4 True 1 2024-08-24T07:54:45Z
discussion \
0 Torch compile stochastically fails with FileNo...
1 FlopCounterMode doesn't support HOP\n### 🐛 Des...
2 Remove redundant copy in functional collective...
3 This send_object_list and recv_object_list met...
4 ProcessGroupNCCL::barrier()'s device id guessi...
summary \
0 Torch compile stochastically fails with FileNo...
1 FlopCounterMode does not support HOP, causing ...
2 Discussion on removing redundant copy operation...
3 Performance issue with send_object_list and re...
4 ProcessGroupNCCL::barrier() device id guessing...
possible_causes \
0 [Race condition due to concurrent.futures, Fil...
1 [Missing registration of FlopCounterMode formu...
2 [Current implementation of functional collecti...
3 [Serialization and deserialization overhead, D...
4 [Insufficient device id guessing logic in Proc...
remediations \
0 [Check for file existence before compiling, Us...
1 [Register a FlopCounterMode formula via HOP.py...
2 [Implement in-place version of functional coll...
3 [Use send/recv methods instead of send_object_...
4 [Improve the device id guessing logic in Proce...
component sentiment issue_type severity \
0 torch._dynamo negative bug_report major
1 torch.utils.flop_counter.FlopCounterMode neutral bug_report major
2 PyTorch Distributed neutral discussion minor
3 PyTorch NCCL communication module negative bug_report major
4 ProcessGroupNCCL negative bug_report major
op_expertise themes
0 advanced [Distributed Training and Multi-GPU, CUDA Comp...
1 advanced [Miscellaneous]
2 advanced [Miscellaneous]
3 intermediate [Distributed Training and Multi-GPU, Performan...
4 advanced [Distributed Training and Multi-GPU, Performan...
Use Llama to generate high-level insights
The above data is good for OSS maintainers and developers to quickly address any issues. The next section will synthesize this data into high-level insights about this repository.
[31]
Prompt for generating high-level overview:
{'system': 'You are not only an experienced Open Source maintainer, but also an expert at paraphrasing raw data into clear succinct reports. Draft a concise report about the issues in this open source repository. Include an executive summary that provides an overview of the challenges faced, any open questions or decisions to be made, or actions that we can take. Group issues together if they ladder up to the same overall challenge, summarize the challenges and include any actionable resolutions we can take (more information in the \\"remediations\\" sections). Use your experience and judgement to ignore issues that are clearly unrelated to the open source project. Ensure the output is in JSON.', 'json_schema': '{ "type": "object", "properties": { "executive_summary": { "description": "An executive summary of the analysis", "type": "string" }, "open_questions": { "description": "Any open questions or decisions that the product team needs to make in light of these issues", "type": "array", "items": { "type": "string" } }, "issue_analysis": { "type": "array", "items": { "type": "object", "properties": { "key_challenge": { "description": "A description of the challenge reported in these issues", "type": "string" }, "affected_issues": { "description": "A list of issues that are related to this challenge", "type": "array", "items": { "type": "number" } }, "possible_causes": { "description": "A list of possible causes or reasons for this challenge to occur", "type": "array", "items": { "type": "string" } }, "remediations": { "description": "Steps we can take to address this challenge", "type": "array", "items": { "type": "string" } } }, "required": ["key_challenge", "affected_issues", "possible_causes", "remediations"] } } }, "required": ["issue_analysis", "open_questions", "actions", "executive_summary"] }'}
[ ]
Generating high-level summaries from annotations... Identifying key-challenges faced by users... Data saved to output/pytorch/pytorch/2024-08-28_2024-08-28/challenges.csv Data saved to output/pytorch/pytorch/2024-08-28_2024-08-28/overview.csv
Key Challenges data
We identify key areas that users are challenged by along with the relevant issues.
[ ]
[Showing 5 out of 5 rows]
repo_name start_date end_date key_challenge \
0 pytorch/pytorch 2024-08-28 2024-08-28 Performance Regressions
1 pytorch/pytorch 2024-08-28 2024-08-28 Compatibility Issues
2 pytorch/pytorch 2024-08-28 2024-08-28 Security Vulnerabilities
3 pytorch/pytorch 2024-08-28 2024-08-28 Tensor Parallelism and Autograd
4 pytorch/pytorch 2024-08-28 2024-08-28 CUDA Support
affected_issues \
0 [134679, 134686]
1 [134640, 134682, 134684]
2 [134664]
3 [134668, 134676]
4 [134682, 134684]
possible_causes \
0 [Changes in the torch or torchvision libraries...
1 [Incompatible version of caffe2 with the curre...
2 [Using the affected version of protobuf (3.20....
3 [Lack of understanding of tensor parallelism i...
4 [cuDNN version incompatibility between PyTorch...
remediations
0 [Investigate the suspected guilty commit and r...
1 [Try installing an older version of caffe2 tha...
2 [Update protobuf to a version mentioned in the...
3 [Improve documentation on tensor parallelism a...
4 [Ensure PyTorch can find the bundled cuDNN by ...
Overview Data
As the name suggests, the overview dataframe contains columns that provide information about the overall activity in the repository during this period, including:
- an executive summary of all the issues seen during this period
- how many issues were created, discussed and closed
- what are some open questions that the maintainers should address
- how many issues were seen for each theme etc.
[32]
[Showing 5 out of 1 rows]
repo_name start_date end_date issues_created open_discussion \
0 pytorch/pytorch 2024-08-28 2024-08-28 26 8
closed_discussion open_no_discussion closed_no_discussion \
0 0 18 0
open_questions \
0 [What is the root cause of the performance reg...
executive_summary ... \
0 The PyTorch repository is facing various chall... ...
themes_count_model_loading themes_count_model_fine_tuning_and_training \
0 3 3
themes_count_model_inference \
0 3
themes_count_distributed_training_and_multi_gpu \
0 3
themes_count_performance_and_optimization \
0 5
themes_count_quantization_and_mixed_precision themes_count_documentation \
0 1 1
themes_count_cuda_compatibility \
0 2
themes_count_model_evaluation_and_benchmarking themes_count_miscellaneous
0 1 14
[1 rows x 28 columns]
Visualizing the data
Based on this data we can easily create some plots to graphically understand the activity in the repo.
Some additional data can be accessed via the github API, but this requires you to have push-access to this repo.
The generated plots are saved as images in plot_folder
[33]
Plotting traffic trends...
Github fetch failed for <function plot_views_clones at 0x13837a3e0>. Make sure you have push-access to pytorch/pytorch! Github fetch failed for <function plot_high_traffic_resources at 0x13ed59580>. Make sure you have push-access to pytorch/pytorch! Github fetch failed for <function plot_high_traffic_referrers at 0x13ed84860>. Make sure you have push-access to pytorch/pytorch!
Plotting issue trends...
Putting it together
Now that we have all the data and insights, we can create a PDF report
[34]
Creating PDF report at output/pytorch/pytorch/2024-08-28_2024-08-28/report.pdf
[ ]