Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sweep: Modify posthog to log on_ticket successes separately from failures src/handlers/on_ticket.py #11

Open
wwzeng1 opened this issue Jun 16, 2023 · 4 comments
Open

Sweep: Modify posthog to log on_ticket successes separately from failures src/handlers/on_ticket.py #11

wwzeng1 opened this issue Jun 16, 2023 · 4 comments
Labels
sweep Assigns Sweep to an issue or pull request. test_issue

Comments

@wwzeng1
Copy link
Contributor

wwzeng1 commented Jun 16, 2023

No description provided.

@sweep-nightly sweep-nightly bot added the sweep Assigns Sweep to an issue or pull request. label Jun 16, 2023
@sweep-nightly
Copy link
Contributor

sweep-nightly bot commented Jun 16, 2023

Hey @wwzeng1,

I've started working on your request. The plan is to modify the logging in the on_ticket function to differentiate between success and failure scenarios. This will involve adjusting the posthog.capture method calls to use different event names based on whether the function executed successfully or encountered an error.

Give me a minute!

Some code snippets I looked at (click to expand). If some file is missing from here, you can mention the path in the ticket description.

sweep/src/api.py

Lines 97 to 118 in c77f0ec

current_issue.add_to_labels(LABEL_NAME)
case "issues", "labeled":
request = IssueRequest(**request_dict)
if request.issue is not None and (
"sweep" in [label.name.lower() for label in request.issue.labels]
):
request.issue.body = request.issue.body or ""
request.repository.description = (
request.repository.description or ""
)
# Update before we handle the ticket to make sure index is up to date
# other ways suboptimal
handle_ticket.spawn(
request.issue.title,
request.issue.body,
request.issue.number,
request.issue.html_url,
request.issue.user.login,
request.repository.full_name,
request.repository.description,
request.installation.id,
)

sweep/src/api.py

Lines 1 to 57 in c77f0ec

from loguru import logger
import modal
from pydantic import ValidationError # type: ignore
from src.handlers.on_ticket import on_ticket
from src.handlers.on_comment import on_comment
from src.utils.constants import API_NAME, BOT_TOKEN_NAME, DB_NAME, LABEL_COLOR, LABEL_DESCRIPTION, LABEL_NAME, SWEEP_LOGIN
from src.events import (
CommentCreatedRequest,
InstallationCreatedRequest,
IssueCommentRequest,
IssueRequest,
ReposAddedRequest,
)
from src.utils.event_logger import posthog
from src.utils.github_utils import get_github_client, index_full_repository
from fastapi import HTTPException, Request
stub = modal.Stub(API_NAME)
image = (
modal.Image.debian_slim()
.apt_install("git")
.pip_install(
"openai",
"anthropic",
"PyGithub",
"loguru",
"docarray",
"backoff",
"tiktoken",
"highlight-io",
"GitPython",
"posthog",
"tqdm"
)
)
secrets = [
modal.Secret.from_name(BOT_TOKEN_NAME),
modal.Secret.from_name("openai-secret"),
modal.Secret.from_name("anthropic"),
modal.Secret.from_name("posthog"),
modal.Secret.from_name("highlight"),
]
FUNCTION_SETTINGS = {
"image": image,
"secrets": secrets,
"timeout": 15 * 60,
}
retries = modal.Retries(
max_retries=2,
backoff_coefficient=2,
initial_delay=60,
)
handle_ticket = stub.function(**FUNCTION_SETTINGS, retries=retries)(on_ticket)
handle_comment = stub.function(**FUNCTION_SETTINGS, retries=retries)(on_comment)

sweep/src/utils/github_utils.py

Lines 1 to 54 in c77f0ec

import shutil
import modal
import os
import time
import re
import github
from github import Github
from github.Repository import Repository
from loguru import logger
from git import Repo
from jwt import encode
import requests
from tqdm import tqdm
from src.core.models import Snippet
from src.utils.config import SweepConfig
from src.utils.constants import APP_ID, DB_NAME
from src.utils.event_logger import posthog
# from src.utils.event_logger import log_info_event # type: ignore
def make_valid_string(string: str):
pattern = r"[^\w./-]+"
return re.sub(pattern, "_", string)
def get_jwt():
signing_key = os.environ["GITHUB_APP_PEM"]
app_id = APP_ID
payload = {"iat": int(time.time()), "exp": int(time.time()) + 600, "iss": app_id}
return encode(payload, signing_key, algorithm="RS256")
def get_token(installation_id: int):
jwt = get_jwt()
headers = {
"Accept": "application/vnd.github+json",
"Authorization": "Bearer " + jwt,
"X-GitHub-Api-Version": "2022-11-28",
}
response = requests.post(
f"https://api.github.com/app/installations/{int(installation_id)}/access_tokens",
headers=headers,
)
return response.json()["token"]
def get_github_client(installation_id: int):
token = get_token(installation_id)
return Github(token)

sweep/src/api.py

Lines 140 to 151 in c77f0ec

)
elif request.issue.pull_request and request.issue.user.login == SWEEP_LOGIN: # TODO(sweep): set a limit
logger.info(f"Handling comment on PR: {request.issue.pull_request}")
handle_comment.spawn(
repo_full_name=request.repository.full_name,
repo_description=request.repository.description,
comment=request.comment.body,
pr_path=None,
pr_line_position=None,
username=request.comment.user.login,
installation_id=request.installation.id,
pr_number=request.issue.number,

sweep/src/handlers/on_ticket.py

Lines 1 to 286 in c77f0ec

"""
On Github ticket, get ChatGPT to deal with it
"""
# TODO: Add file validation
import os
import openai
from loguru import logger
import modal
from src.core.models import Snippet
from src.core.prompts import (
reply_prompt,
)
from src.core.sweep_bot import SweepBot
from src.core.prompts import issue_comment_prompt
from src.handlers.on_review import review_pr
from src.utils.event_logger import posthog
from src.utils.github_utils import get_github_client, search_snippets
from src.utils.prompt_constructor import HumanMessagePrompt
from src.utils.constants import DB_NAME, PREFIX
github_access_token = os.environ.get("GITHUB_TOKEN")
openai.api_key = os.environ.get("OPENAI_API_KEY")
update_index = modal.Function.lookup(DB_NAME, "update_index")
bot_suffix = "I'm a bot that handles simple bugs and feature requests \
but I might make mistakes. Please be kind!"
collapsible_template = """
<details>
<summary>{summary}</summary>
{body}
</details>
"""
chunker = modal.Function.lookup("utils", "Chunking.chunk")
num_of_snippets_to_query = 10
max_num_of_snippets = 5
def on_ticket(
title: str,
summary: str,
issue_number: int,
issue_url: str,
username: str,
repo_full_name: str,
repo_description: str,
installation_id: int,
comment_id: int = None
):
# Flow:
# 1. Get relevant files
# 2: Get human message
# 3. Get files to change
# 4. Get file changes
# 5. Create PR
organization, repo_name = repo_full_name.split("/")
metadata = {
"issue_url": issue_url,
"issue_number": issue_number,
"repo_full_name": repo_full_name,
"organization": organization,
"repo_name": repo_name,
"repo_description": repo_description,
"username": username,
"installation_id": installation_id,
"function": "on_ticket",
"mode": PREFIX,
}
posthog.capture(username, "started", properties=metadata)
g = get_github_client(installation_id)
if comment_id:
logger.info(f"Replying to comment {comment_id}...")
logger.info(f"Getting repo {repo_full_name}")
repo = g.get_repo(repo_full_name)
current_issue = repo.get_issue(number=issue_number)
item_to_react_to = current_issue.get_comment(comment_id) if comment_id else current_issue
eyes_reaction = item_to_react_to.create_reaction("eyes")
def comment_reply(message: str):
current_issue.create_comment(message + "\n\n---\n" + bot_suffix)
comments = current_issue.get_comments()
replies_text = ""
if comment_id:
replies_text = "\nComments:\n" + "\n".join(
[
issue_comment_prompt.format(
username=comment.user.login,
reply=comment.body,
) for comment in comments
]
)
def fetch_file_contents_with_retry():
retries = 3
error = None
for i in range(retries):
try:
logger.info(f"Fetching relevant files for the {i}th time...")
return search_snippets(
repo,
f"{title}\n{summary}\n{replies_text}",
num_files=num_of_snippets_to_query,
branch=None,
installation_id=installation_id,
)
except Exception as e:
error = e
continue
posthog.capture(
username, "fetching_failed", properties={"error": error, **metadata}
)
raise error
# update_index.call(
# repo_full_name,
# installation_id=installation_id,
# )
logger.info("Fetching relevant files...")
try:
snippets, tree = fetch_file_contents_with_retry()
assert len(snippets) > 0
except Exception as e:
logger.error(e)
comment_reply(
"It looks like an issue has occured around fetching the files. Perhaps the repo has not been initialized: try removing this repo and adding it back. I'll try again in a minute. If this error persists contact team@sweep.dev."
)
raise e
# reversing to put most relevant at the bottom
snippets: list[Snippet] = snippets[::-1]
num_full_files = 2
num_extended_snippets = 2
most_relevant_snippets = snippets[-num_full_files:]
snippets = snippets[:-num_full_files]
logger.info("Expanding snippets...")
for snippet in most_relevant_snippets:
current_snippet = snippet
_chunks, metadatas, _ids = chunker.call(
current_snippet.content,
current_snippet.file_path
)
segmented_snippets = [
Snippet(
content=current_snippet.content,
start=metadata["start"],
end=metadata["end"],
file_path=metadata["file_path"],
) for metadata in metadatas
]
index = 0
while index < len(segmented_snippets) and segmented_snippets[index].start <= current_snippet.start:
index += 1
index -= 1
for i in range(index + 1, min(index + num_extended_snippets + 1, len(segmented_snippets))):
current_snippet += segmented_snippets[i]
for i in range(index - 1, max(index - num_extended_snippets - 1, 0), -1):
current_snippet = segmented_snippets[i] + current_snippet
snippets.append(current_snippet)
# snippet fusing
i = 0
while i < len(snippets):
j = i + 1
while j < len(snippets):
if snippets[i] ^ snippets[j]: # this checks for overlap
snippets[i] = snippets[i] | snippets[j] # merging
snippets.pop(j)
else:
j += 1
i += 1
snippets = snippets[:min(len(snippets), max_num_of_snippets)]
human_message = HumanMessagePrompt(
repo_name=repo_name,
issue_url=issue_url,
username=username,
repo_description=repo_description,
title=title,
summary=summary + replies_text,
snippets=snippets,
tree=tree, # TODO: Anything in repo tree that has something going through is expanded
)
sweep_bot = SweepBot.from_system_message_content(
human_message=human_message, repo=repo, is_reply=bool(comments)
)
try:
logger.info("CoT retrieval...")
if sweep_bot.model == "gpt-4-32k-0613":
sweep_bot.cot_retrieval()
logger.info("Fetching files to modify/create...")
file_change_requests = sweep_bot.get_files_to_change()
logger.info("Getting response from ChatGPT...")
reply = sweep_bot.chat(reply_prompt, message_key="reply")
sweep_bot.delete_messages_from_chat("reply")
logger.info("Sending response...")
new_line = '\n'
comment_reply(
reply
+ "\n\n"
+ collapsible_template.format(
summary="Some code snippets I looked at (click to expand). If some file is missing from here, you can mention the path in the ticket description.",
body="\n".join(
[
f"https://github.com/{organization}/{repo_name}/blob/{repo.get_commits()[0].sha}/{snippet.file_path}#L{max(snippet.start, 1)}-L{min(snippet.end, snippet.content.count(new_line))}\n"
for snippet in snippets[::-1]
]
),
)
)
logger.info("Generating PR...")
pull_request = sweep_bot.generate_pull_request()
logger.info("Making PR...")
pull_request.branch_name = sweep_bot.create_branch(pull_request.branch_name)
sweep_bot.change_files_in_github(file_change_requests, pull_request.branch_name)
# Include issue number in PR description
pr_description = f"{pull_request.content}\n\nFixes #{issue_number}."
pr = repo.create_pull(
title=pull_request.title,
body=pr_description,
head=pull_request.branch_name,
base=repo.default_branch,
)
current_issue.create_reaction("rocket")
try:
review_pr(repo=repo, pr=pr, issue_url=issue_url, username=username,
repo_description=repo_description, title=title,
summary=summary, replies_text=replies_text, installation_id=installation_id, snippets=snippets, tree=tree)
except Exception as e:
logger.error(e)
except openai.error.InvalidRequestError as e:
logger.error(e)
comment_reply(
"I'm sorry, but it looks our model has ran out of context length. We're trying to make this happen less, but one way to mitigate this is to code smaller files. I'll try again in a minute. If this error persists contact team@sweep.dev."
)
posthog.capture(
username,
"failed",
properties={
"error": str(e),
"reason": "Invalid request error / context length",
**metadata,
},
)
raise e
except Exception as e:
logger.error(e)
comment_reply(
"I'm sorry, but it looks like an error has occured. Try removing and re-adding the sweep label. I'll try again in a minute. If this error persists contact team@sweep.dev."
)
posthog.capture(
username,
"failed",
properties={"error": str(e), "reason": "Generic error", **metadata},
)
raise e
else:
try:
eyes_reaction.delete()
except:
pass
item_to_react_to.create_reaction("rocket")
posthog.capture(username, "success", properties={**metadata})
logger.info("on_ticket success")
return {"success": True}

I'm a bot that handles simple bugs and feature requests but I might make mistakes. Please be kind!

@sweep-nightly sweep-nightly bot mentioned this issue Jun 16, 2023
Closed
@sweep-nightly
Copy link
Contributor

sweep-nightly bot commented Jun 16, 2023

I'm sorry, but it looks like an error has occured. Try removing and re-adding the sweep label. I'll try again in a minute. If this error persists contact team@sweep.dev.

I'm a bot that handles simple bugs and feature requests but I might make mistakes. Please be kind!

2 similar comments
@sweep-nightly
Copy link
Contributor

sweep-nightly bot commented Jun 16, 2023

I'm sorry, but it looks like an error has occured. Try removing and re-adding the sweep label. I'll try again in a minute. If this error persists contact team@sweep.dev.

I'm a bot that handles simple bugs and feature requests but I might make mistakes. Please be kind!

@sweep-nightly
Copy link
Contributor

sweep-nightly bot commented Jun 16, 2023

I'm sorry, but it looks like an error has occured. Try removing and re-adding the sweep label. I'll try again in a minute. If this error persists contact team@sweep.dev.

I'm a bot that handles simple bugs and feature requests but I might make mistakes. Please be kind!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
sweep Assigns Sweep to an issue or pull request. test_issue
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Separate Success and Failure Logging in on_ticket Function sweepai/sweep
1 participant
@wwzeng1