Add Github Action for Benchmarking on Merge #119
Conversation
![sweep-nightly[bot]](https://avatars.githubusercontent.com/in/324098?s=60&v=4){kind=small}
There was a problem hiding this comment.
Great work on these changes! Here's a summary of what I found:
-
.github/workflows/benchmark_on_merge.yml: This new file correctly sets up a Github Action workflow for running benchmarks, creating sample issues, and measuring issue quality whenever there's a push to the main branch. No changes required. -
src/utils/issue_quality_checker.py: This new file introduces a function to calculate the quality of a GitHub issue based on the length of its title and body. It also includes a main block to test this function with a specific issue from the "sweepai/sweep" repository. No changes required. -
tests/create_sample_issue.py: Thecreate_issuefunction now returns the created issue object. Also, the script now creates issues in the "sweepai/bot-internal" repository when run. No changes required.
Keep up the good work!
This PR introduces a new Github action that runs benchmarks on code whenever a merge to the main branch occurs. The action also creates sample issues and measures their quality.
Changes
.github/workflows/benchmark_on_merge.ymlthat triggers on merge to the main branch. This workflow runs benchmarks on the merged code and creates sample issues.src/utils/issue_quality_checker.pythat measures the quality of an issue. This script is called by the Github action to evaluate the quality of the sample issues it creates.tests/create_sample_issue.pyto extract the logic for creating sample issues into a function. This function can now be imported by other scripts.Testing
Please ensure the Github action triggers correctly on merge and that it successfully runs benchmarks, creates sample issues, and measures their quality.
Fixes #118.
To checkout this PR branch, run the following command in your terminal: