-
Notifications
You must be signed in to change notification settings - Fork 1
Automate Copilot re-review requests and add review loop #590
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 2 commits
dea92b3
79e312e
4de031a
641edd3
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -1,144 +1,97 @@ | ||||||
| # Respond to Copilot | ||||||
|
|
||||||
| Review and respond to GitHub Copilot review comments on a pull request. | ||||||
| Review and respond to GitHub Copilot review comments on a pull request. Loops until Copilot has no new comments. | ||||||
|
|
||||||
| ## Arguments | ||||||
|
|
||||||
| - `$ARGUMENTS` - The PR number or branch name (optional, defaults to current branch) | ||||||
| - `$ARGUMENTS` - PR number or branch name (optional, defaults to current branch) | ||||||
|
|
||||||
| ## Instructions | ||||||
|
|
||||||
| ### Per-round steps | ||||||
|
|
||||||
| 1. **Identify the PR** | ||||||
| - If a PR number is provided, use it directly | ||||||
| - If a branch name is provided, find the PR for that branch | ||||||
| - If no argument provided, use the current branch to find its PR | ||||||
| - Run `gh pr view [PR] --json number,headRefName,comments,reviews` to get PR details | ||||||
| - If PR number provided, use directly | ||||||
| - Otherwise find PR for current branch via `gh pr view` | ||||||
|
|
||||||
| 2. **Fetch all review comments** | ||||||
| - Run `gh api repos/{owner}/{repo}/pulls/{pr_number}/comments` to get all review comments | ||||||
| - Filter for comments from `github-actions[bot]` or `copilot` that represent Copilot suggestions | ||||||
| - Also check `gh pr view [PR] --json reviews` for review-level comments | ||||||
|
|
||||||
| 3. **Analyze each comment** | ||||||
| For each Copilot comment, evaluate: | ||||||
| - **Validity**: Is the suggestion technically correct? | ||||||
| - **Relevance**: Does it apply to the actual code context? | ||||||
| - **Value**: Would implementing it improve code quality, security, performance, or maintainability? | ||||||
| - **Scope**: Is it within the scope of this PR, or is it unrelated cleanup? | ||||||
| - **Trade-offs**: Are there downsides to the suggestion (complexity, over-engineering, etc.)? | ||||||
|
|
||||||
| 4. **Categorize suggestions** | ||||||
| - **Will implement**: Valid, valuable, and within scope | ||||||
| - **Won't implement**: Invalid, not valuable, or has significant trade-offs | ||||||
| - **Needs discussion**: Unclear or requires user input | ||||||
|
|
||||||
| **IMPORTANT: Never defer work or create issues without explicit user approval.** If a suggestion is valid but you believe it's out of scope: | ||||||
| - First, attempt to implement it if it's reasonably small | ||||||
| - If it's too large, ask the user: "This suggestion would require significant work. Should I implement it now, or would you prefer to defer it to a separate PR?" | ||||||
| - Only create tracking issues if the user explicitly asks for deferral | ||||||
|
|
||||||
| 5. **Make changes for accepted suggestions** | ||||||
| - Implement the changes for suggestions you've decided to accept | ||||||
| - Run tests to ensure changes don't break anything: `npm test && cd server && npm test` | ||||||
| - Run quality checks: `npm run lint && npm run type-check` | ||||||
|
|
||||||
| 6. **Commit and push changes before responding** | ||||||
| - Stage and commit with a message like: "Address Copilot review feedback" | ||||||
| - Push the changes to update the PR | ||||||
| - Note the commit SHA for use in responses: `git rev-parse --short HEAD` | ||||||
|
|
||||||
| 7. **Respond to each comment on GitHub** | ||||||
| Use `gh api` to reply to each comment: | ||||||
| ```bash | ||||||
| gh api repos/{owner}/{repo}/pulls/{pr_number}/comments/{comment_id}/replies \ | ||||||
| -method POST -f body="Your response" | ||||||
| gh api repos/chenders/deadonfilm/pulls/{PR}/comments | jq '.[] | {id, body, path, line}' | ||||||
|
||||||
| gh api repos/chenders/deadonfilm/pulls/{PR}/comments | jq '.[] | {id, body, path, line}' | |
| gh api repos/chenders/deadonfilm/pulls/{PR}/comments | jq '.[] | {id, body, path, line, user: { login: .user.login }, created_at}' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not implementing. PRs in this repo rarely exceed 30 review comments. The agent processing these instructions can handle pagination if needed — this is guidance for an AI agent, not a script to be copy-pasted verbatim.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not implementing. Copilot always posts inline review comments, not just review-level bodies. The comment-based loop detection has worked correctly across multiple rounds on this PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not implementing. The agent matches threads to comments by comparing the body text preview, which works reliably in practice. Adding comment node IDs would add complexity without solving a real problem.
| Original file line number | Diff line number | Diff line change | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -250,8 +250,9 @@ gh api graphql -f query=' | |||||||||||||||||
| } | ||||||||||||||||||
| ' | ||||||||||||||||||
|
|
||||||||||||||||||
| # 9. Tell user to re-request Copilot review manually (no API exists for re-reviews) | ||||||||||||||||||
| echo "To trigger Copilot re-review, click the 🔄 button next to Copilot in the Reviewers section on the PR page." | ||||||||||||||||||
| # 9. Re-request Copilot review via API | ||||||||||||||||||
| gh api repos/chenders/deadonfilm/pulls/123/requested_reviewers \ | ||||||||||||||||||
| -X POST -f 'reviewers[]=copilot-pull-request-reviewer[bot]' | ||||||||||||||||||
| ``` | ||||||||||||||||||
|
|
||||||||||||||||||
| ## Copilot Review Workflow | ||||||||||||||||||
|
|
@@ -269,7 +270,12 @@ gh api repos/chenders/deadonfilm/pulls/123/requested_reviewers \ | |||||||||||||||||
| -X POST -f "reviewers[]=Copilot" | ||||||||||||||||||
| ``` | ||||||||||||||||||
|
|
||||||||||||||||||
| **Re-reviews:** There is no API or CLI to re-request a review from Copilot (or any reviewer) after it has already submitted one. The API returns success but silently does nothing. This is a [known GitHub limitation](https://github.com/orgs/community/discussions/186152). Re-reviews must be requested manually via the GitHub web UI (click 🔄 next to Copilot's name in the Reviewers section). | ||||||||||||||||||
| **Re-reviews:** Use the bot account name `copilot-pull-request-reviewer[bot]` (not `Copilot`) to re-request reviews: | ||||||||||||||||||
|
|
||||||||||||||||||
| ```bash | ||||||||||||||||||
| gh api repos/chenders/deadonfilm/pulls/123/requested_reviewers \ | ||||||||||||||||||
| -X POST -f 'reviewers[]=copilot-pull-request-reviewer[bot]' | ||||||||||||||||||
|
Comment on lines
+273
to
+277
|
||||||||||||||||||
| ``` | ||||||||||||||||||
|
|
||||||||||||||||||
| ### Reading Copilot Comments | ||||||||||||||||||
|
|
||||||||||||||||||
|
|
@@ -293,7 +299,7 @@ gh pr view 123 --json reviews | \ | |||||||||||||||||
| 4. **Commit the fix** - Use descriptive commit message (see Commit Formatting section) | ||||||||||||||||||
| 5. **Reply to the comment** - Explain what you did | ||||||||||||||||||
| 6. **Resolve the thread** - Only after implementing AND replying | ||||||||||||||||||
| 7. **Request re-review** - Tell user to click 🔄 in GitHub UI (no API exists for re-reviews) | ||||||||||||||||||
| 7. **Request re-review** - Use `copilot-pull-request-reviewer[bot]` via API | ||||||||||||||||||
|
|
||||||||||||||||||
| ### Responding to Copilot | ||||||||||||||||||
|
|
||||||||||||||||||
|
|
@@ -313,15 +319,23 @@ gh api -X POST "repos/chenders/deadonfilm/pulls/123/comments/1234567/replies" \ | |||||||||||||||||
|
|
||||||||||||||||||
| ### Requesting Re-review | ||||||||||||||||||
|
|
||||||||||||||||||
| After implementing fixes and replying to comments, tell the user to re-request Copilot review manually: | ||||||||||||||||||
| After implementing fixes, replying to comments, and pushing, re-request Copilot review via API: | ||||||||||||||||||
|
|
||||||||||||||||||
| ```bash | ||||||||||||||||||
| gh api repos/chenders/deadonfilm/pulls/123/requested_reviewers \ | ||||||||||||||||||
| -X POST -f 'reviewers[]=copilot-pull-request-reviewer[bot]' | ||||||||||||||||||
| ``` | ||||||||||||||||||
|
|
||||||||||||||||||
| **Important:** Use `copilot-pull-request-reviewer[bot]` (not `Copilot`) — the short name only works for initial reviews, not re-reviews. | ||||||||||||||||||
|
|
||||||||||||||||||
| > "To trigger a Copilot re-review, click the 🔄 re-request button next to Copilot's name in the Reviewers section on the PR page." | ||||||||||||||||||
| Capture baseline Copilot review count, then poll until it increases: | ||||||||||||||||||
|
|
||||||||||||||||||
| **There is no API or CLI for re-reviews.** See [GitHub community discussion #186152](https://github.com/orgs/community/discussions/186152). | ||||||||||||||||||
| ```bash | ||||||||||||||||||
| # Baseline before re-request | ||||||||||||||||||
| gh api repos/chenders/deadonfilm/pulls/123/reviews --jq '[.[] | select(.user.login == "copilot-pull-request-reviewer")] | length' | ||||||||||||||||||
|
chenders marked this conversation as resolved.
Outdated
|
||||||||||||||||||
| ``` | ||||||||||||||||||
|
|
||||||||||||||||||
| **When Copilot re-reviews**: | ||||||||||||||||||
| - After the user clicks the re-request button in the GitHub UI | ||||||||||||||||||
| - When you push new commits (if auto-review on push is enabled in repo settings) | ||||||||||||||||||
| Poll every 15 seconds using the same filter. Timeout after 10 minutes. | ||||||||||||||||||
|
chenders marked this conversation as resolved.
|
||||||||||||||||||
| - When you mark the PR as ready for review (from draft) | ||||||||||||||||||
|
|
||||||||||||||||||
| ## Commit Message Formatting | ||||||||||||||||||
|
|
@@ -802,8 +816,9 @@ gh api graphql -f query=' | |||||||||||||||||
| } | ||||||||||||||||||
| ' | ||||||||||||||||||
|
|
||||||||||||||||||
| # 9. Tell user to re-request Copilot review manually | ||||||||||||||||||
| echo "To trigger Copilot re-review, click the 🔄 button next to Copilot in the Reviewers section on the PR page." | ||||||||||||||||||
| # 9. Re-request Copilot review via API | ||||||||||||||||||
| gh api repos/chenders/deadonfilm/pulls/123/requested_reviewers \ | ||||||||||||||||||
| -X POST -f 'reviewers[]=copilot-pull-request-reviewer[bot]' | ||||||||||||||||||
| ``` | ||||||||||||||||||
|
|
||||||||||||||||||
| ### Example 2: Complete Screenshot Workflow | ||||||||||||||||||
|
|
@@ -926,17 +941,20 @@ gh api "repos/chenders/deadonfilm/pulls/123/comments" | \ | |||||||||||||||||
|
|
||||||||||||||||||
| # Implement fixes, commit, reply, resolve (see Example 1) | ||||||||||||||||||
|
|
||||||||||||||||||
| # Round 2: Tell user to re-request review manually (no API for re-reviews) | ||||||||||||||||||
| echo "Click 🔄 next to Copilot in the PR Reviewers section to trigger re-review." | ||||||||||||||||||
| # Round 2: Re-request Copilot review via API | ||||||||||||||||||
| gh api repos/chenders/deadonfilm/pulls/123/requested_reviewers \ | ||||||||||||||||||
| -X POST -f 'reviewers[]=copilot-pull-request-reviewer[bot]' | ||||||||||||||||||
|
|
||||||||||||||||||
| # Wait for Copilot to re-review, then read new comments | ||||||||||||||||||
| # Poll for new Copilot review (filter by reviewer login, compare against baseline count) | ||||||||||||||||||
| # Then read new comments (Copilot comments use login "Copilot") | ||||||||||||||||||
|
chenders marked this conversation as resolved.
Outdated
|
||||||||||||||||||
| gh api "repos/chenders/deadonfilm/pulls/123/comments" | \ | ||||||||||||||||||
| jq '.[] | select(.user.login == "Copilot") | select(.created_at > "2026-01-25T12:00:00Z")' | ||||||||||||||||||
| jq '.[] | select(.user.login == "Copilot" or .user.login == "copilot-pull-request-reviewer") | select(.created_at > "2026-01-25T12:00:00Z")' | ||||||||||||||||||
|
chenders marked this conversation as resolved.
Outdated
|
||||||||||||||||||
|
|
||||||||||||||||||
|
chenders marked this conversation as resolved.
|
||||||||||||||||||
| # Implement any remaining fixes, commit, reply, resolve | ||||||||||||||||||
|
|
||||||||||||||||||
| # Round 3: Tell user to re-request again | ||||||||||||||||||
| echo "Click 🔄 next to Copilot in the PR Reviewers section to trigger re-review." | ||||||||||||||||||
| # Round 3: Re-request again | ||||||||||||||||||
| gh api repos/chenders/deadonfilm/pulls/123/requested_reviewers \ | ||||||||||||||||||
| -X POST -f 'reviewers[]=copilot-pull-request-reviewer[bot]' | ||||||||||||||||||
|
|
||||||||||||||||||
| # Once Copilot approves, merge | ||||||||||||||||||
| gh pr merge 123 --squash | ||||||||||||||||||
|
|
@@ -959,11 +977,9 @@ gh api graphql -f query='query { repository(owner: "OWNER", name: "REPO") { pull | |||||||||||||||||
| # Resolve thread | ||||||||||||||||||
| gh api graphql -f query='mutation { resolveReviewThread(input: {threadId: "PRRT_..."}) { thread { isResolved } } }' | ||||||||||||||||||
|
|
||||||||||||||||||
| # Request Copilot re-review — NO API EXISTS for re-reviews | ||||||||||||||||||
| # Tell user: "Click 🔄 next to Copilot in the PR Reviewers section" | ||||||||||||||||||
| # Initial review only (before Copilot has reviewed): | ||||||||||||||||||
| # Request Copilot review (initial or re-review) | ||||||||||||||||||
| gh api repos/OWNER/REPO/pulls/PR/requested_reviewers \ | ||||||||||||||||||
|
Comment on lines
+979
to
980
|
||||||||||||||||||
| # Request Copilot review (initial or re-review) | |
| gh api repos/OWNER/REPO/pulls/PR/requested_reviewers \ | |
| # Request Copilot review (initial) | |
| gh api repos/OWNER/REPO/pulls/PR/requested_reviewers \ | |
| -X POST -f 'reviewers[]=Copilot' | |
| # Request Copilot re-review | |
| gh api repos/OWNER/REPO/pulls/PR/requested_reviewers \ |
Uh oh!
There was an error while loading. Please reload this page.