-
Notifications
You must be signed in to change notification settings - Fork 1
Automate Copilot re-review requests and add review loop #590
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 1 commit
dea92b3
79e312e
4de031a
641edd3
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,144 +1,95 @@ | ||
| # Respond to Copilot | ||
|
|
||
| Review and respond to GitHub Copilot review comments on a pull request. | ||
| Review and respond to GitHub Copilot review comments on a pull request. Loops until Copilot has no new comments. | ||
|
|
||
| ## Arguments | ||
|
|
||
| - `$ARGUMENTS` - The PR number or branch name (optional, defaults to current branch) | ||
| - `$ARGUMENTS` - PR number or branch name (optional, defaults to current branch) | ||
|
|
||
| ## Instructions | ||
|
|
||
| ### Per-round steps | ||
|
|
||
| 1. **Identify the PR** | ||
| - If a PR number is provided, use it directly | ||
| - If a branch name is provided, find the PR for that branch | ||
| - If no argument provided, use the current branch to find its PR | ||
| - Run `gh pr view [PR] --json number,headRefName,comments,reviews` to get PR details | ||
| - If PR number provided, use directly | ||
| - Otherwise find PR for current branch via `gh pr view` | ||
|
|
||
| 2. **Fetch all review comments** | ||
| - Run `gh api repos/{owner}/{repo}/pulls/{pr_number}/comments` to get all review comments | ||
| - Filter for comments from `github-actions[bot]` or `copilot` that represent Copilot suggestions | ||
| - Also check `gh pr view [PR] --json reviews` for review-level comments | ||
|
|
||
| 3. **Analyze each comment** | ||
| For each Copilot comment, evaluate: | ||
| - **Validity**: Is the suggestion technically correct? | ||
| - **Relevance**: Does it apply to the actual code context? | ||
| - **Value**: Would implementing it improve code quality, security, performance, or maintainability? | ||
| - **Scope**: Is it within the scope of this PR, or is it unrelated cleanup? | ||
| - **Trade-offs**: Are there downsides to the suggestion (complexity, over-engineering, etc.)? | ||
|
|
||
| 4. **Categorize suggestions** | ||
| - **Will implement**: Valid, valuable, and within scope | ||
| - **Won't implement**: Invalid, not valuable, or has significant trade-offs | ||
| - **Needs discussion**: Unclear or requires user input | ||
|
|
||
| **IMPORTANT: Never defer work or create issues without explicit user approval.** If a suggestion is valid but you believe it's out of scope: | ||
| - First, attempt to implement it if it's reasonably small | ||
| - If it's too large, ask the user: "This suggestion would require significant work. Should I implement it now, or would you prefer to defer it to a separate PR?" | ||
| - Only create tracking issues if the user explicitly asks for deferral | ||
|
|
||
| 5. **Make changes for accepted suggestions** | ||
| - Implement the changes for suggestions you've decided to accept | ||
| - Run tests to ensure changes don't break anything: `npm test && cd server && npm test` | ||
| - Run quality checks: `npm run lint && npm run type-check` | ||
|
|
||
| 6. **Commit and push changes before responding** | ||
| - Stage and commit with a message like: "Address Copilot review feedback" | ||
| - Push the changes to update the PR | ||
| - Note the commit SHA for use in responses: `git rev-parse --short HEAD` | ||
|
|
||
| 7. **Respond to each comment on GitHub** | ||
| Use `gh api` to reply to each comment: | ||
| ```bash | ||
| gh api repos/{owner}/{repo}/pulls/{pr_number}/comments/{comment_id}/replies \ | ||
| -method POST -f body="Your response" | ||
| gh api repos/chenders/deadonfilm/pulls/{pr_number}/comments | jq '.[] | {id, body, path, line}' | ||
|
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not implementing — the agent processing these comments uses contextual analysis to determine which are Copilot comments vs human comments. Pre-filtering in the jq command would lose context about the full review thread (e.g., human replies to Copilot comments). |
||
| ``` | ||
|
Comment on lines
17
to
21
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not implementing. PRs in this repo rarely exceed 30 review comments. The agent processing these instructions can handle pagination if needed — this is guidance for an AI agent, not a script to be copy-pasted verbatim. |
||
|
|
||
| Response format: | ||
| - **If implemented**: Reference the commit SHA and explain what change was made. Use format: "Fixed in <commit_sha>. <explanation>" | ||
| - **If not implemented**: Explain why (invalid suggestion, out of scope, trade-offs, etc.) | ||
| - **If needs discussion**: Ask clarifying questions | ||
| 3. **Check for new comments** — If there are no new unaddressed comments since the last round, the loop is done. Report the final status and stop. | ||
|
|
||
| 8. **REQUIRED: Resolve implemented comment threads** | ||
| 4. **Analyze each new comment** | ||
| - Validity: Is the suggestion technically correct? | ||
| - Value: Would it improve code quality? | ||
| - Scope: Is it within the scope of this PR? | ||
|
Comment on lines
+23
to
+28
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not implementing. Copilot always posts inline review comments, not just review-level bodies. The comment-based loop detection has worked correctly across multiple rounds on this PR. |
||
|
|
||
| **This step is mandatory for any comments where you implemented fixes.** Do not skip this step. | ||
| 5. **Categorize**: Will implement / Won't implement / Needs discussion | ||
|
|
||
| After responding to comments, resolve the threads using the GraphQL API: | ||
| **IMPORTANT: Never defer work or create issues without explicit user approval.** If a suggestion is valid but out of scope, attempt to implement it if reasonably small. If too large, ask the user. | ||
|
|
||
| **Step 8a:** Query for thread IDs (thread IDs have `PRRT_` prefix, different from comment IDs which have `PRRC_` prefix): | ||
| 6. **Implement accepted suggestions** | ||
| - Make changes | ||
| - Run tests: `npm test && cd server && npm test` | ||
| - Run quality checks: `npm run lint && npm run type-check` | ||
| - Commit: "Address Copilot review feedback" | ||
| - Push changes | ||
|
|
||
| ```bash | ||
| gh api graphql -f query=' | ||
| query { | ||
| repository(owner: "{owner}", name: "{repo}") { | ||
| pullRequest(number: {pr_number}) { | ||
| reviewThreads(first: 50) { | ||
| nodes { | ||
| id | ||
| isResolved | ||
| comments(first: 1) { nodes { body } } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| ' | ||
| ``` | ||
| 7. **Reply to each comment** | ||
|
|
||
| **Step 8b:** For each comment where you implemented the fix, resolve its thread using the `PRRT_` ID: | ||
| ```bash | ||
| gh api graphql -f query=' | ||
| mutation { | ||
| resolveReviewThread(input: {threadId: "PRRT_kwDO..."}) { | ||
| thread { isResolved } | ||
| } | ||
| } | ||
| ' | ||
| gh api -X POST repos/chenders/deadonfilm/pulls/{pr}/comments/{id}/replies -f body="Fixed in $(git rev-parse --short HEAD). Explanation." | ||
|
chenders marked this conversation as resolved.
Outdated
|
||
| ``` | ||
|
|
||
| **Rules:** | ||
| - ✅ Resolve threads where you implemented the suggested fix | ||
| - ❌ Do NOT resolve threads where you declined to make changes | ||
| - **If implemented**: Reference commit SHA and explain | ||
| - **If not implemented**: Explain why | ||
| - **If needs discussion**: Ask clarifying questions | ||
|
|
||
| 8. **Resolve implemented threads** (use PRRT* thread IDs, not PRRC* comment IDs) | ||
|
|
||
| 9. **Notify user to request Copilot re-review** | ||
| If any changes were committed and pushed, tell the user: | ||
| ```bash | ||
| # Get thread IDs | ||
| gh api graphql -f query='query { repository(owner: "chenders", name: "deadonfilm") { pullRequest(number: PR) { reviewThreads(first: 50) { nodes { id isResolved comments(first: 1) { nodes { body } } } } } } }' | ||
|
|
||
| > "Changes pushed. To trigger a Copilot re-review, click the 🔄 re-request button next to Copilot's name in the Reviewers section on the PR page." | ||
| # Resolve | ||
| gh api graphql -f query='mutation { resolveReviewThread(input: {threadId: "PRRT_..."}) { thread { isResolved } } }' | ||
| ``` | ||
|
Comment on lines
+51
to
+59
Owner
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not implementing. The agent matches threads to comments by comparing the body text preview, which works reliably in practice. Adding comment node IDs would add complexity without solving a real problem. |
||
|
|
||
| **Why manual?** GitHub has no API or CLI to re-request a review from a reviewer that has already submitted one. The REST API `POST /pulls/{pr}/requested_reviewers` and `gh pr edit --add-reviewer` only work for initial requests — they return success but silently do nothing for re-reviews. This is a [known GitHub limitation](https://github.com/orgs/community/discussions/186152). | ||
| Rules: | ||
| - Resolve threads where you implemented the fix | ||
| - Do NOT resolve threads where you declined | ||
|
|
||
| ## Example Responses | ||
| 9. **Re-request Copilot review**: | ||
|
|
||
| ```bash | ||
| gh api repos/chenders/deadonfilm/pulls/{PR_NUMBER}/requested_reviewers -X POST -f 'reviewers[]=copilot-pull-request-reviewer[bot]' | ||
| ``` | ||
|
|
||
| **Implemented (simple fix):** | ||
| > Fixed in 2f50cc1. Added null check before accessing the property to prevent potential runtime errors. | ||
| 10. **Wait for the new review** — Poll until a new review appears (review count increases): | ||
|
|
||
| **Implemented (refactoring):** | ||
| > Fixed in 3a8bc12. Extracted the shared logic into a reusable utility function in `src/lib/utils.ts` and updated both call sites to use it. | ||
| ```bash | ||
| gh api repos/chenders/deadonfilm/pulls/{PR_NUMBER}/reviews --jq 'length' | ||
| ``` | ||
|
|
||
| **Implemented (security fix):** | ||
| > Fixed in 5d9ef34. Changed from string interpolation to parameterized query to prevent SQL injection. | ||
| Poll every 15 seconds. Timeout after 10 minutes (assume review is delayed). | ||
|
chenders marked this conversation as resolved.
Outdated
|
||
|
|
||
| **Not implemented (invalid):** | ||
| > This suggestion doesn't apply here - the variable is already guaranteed to be non-null at this point due to the guard clause on line 42. | ||
| 11. **Loop back to step 2** — Fetch comments again and check for new ones. | ||
|
|
||
| **Not implemented (trade-off):** | ||
| > Chose not to implement this. While the suggested abstraction would reduce duplication, it would also add complexity for a pattern that only appears twice in the codebase. Will reconsider if this pattern appears more frequently. | ||
| ### Completion criteria | ||
|
|
||
| ## Completion Checklist | ||
| The loop ends when: | ||
|
|
||
| Before finishing, verify you have completed ALL of these steps: | ||
| - Copilot's latest review has **no new comments** (clean review), OR | ||
| - The poll in step 10 times out (report this and stop) | ||
|
|
||
| - [ ] Analyzed all Copilot comments | ||
| - [ ] Implemented accepted suggestions | ||
| - [ ] Ran tests and quality checks | ||
| - [ ] Committed and pushed changes | ||
| - [ ] Responded to each comment on GitHub | ||
| - [ ] **Resolved implemented comment threads** (step 8 - don't skip!) | ||
| - [ ] Notified user to manually re-request Copilot review (if changes were made) | ||
| When complete, report a summary: total rounds, comments addressed, comments declined. | ||
|
|
||
| ## Notes | ||
|
|
||
| - Be respectful and constructive in responses | ||
| - Don't dismiss suggestions without explanation | ||
| - If you're unsure about a suggestion, ask the user before responding | ||
| - Copilot comments may appear as regular review comments or as part of a review | ||
| - Never dismiss suggestions without explanation | ||
| - Never defer work without explicit user approval | ||
| - Thread IDs (PRRT*) are NOT the same as comment IDs (PRRC*) | ||
| - Track comment IDs across rounds to distinguish new comments from previously addressed ones | ||
| Original file line number | Diff line number | Diff line change | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -250,8 +250,9 @@ gh api graphql -f query=' | |||||||||||||||||
| } | ||||||||||||||||||
| ' | ||||||||||||||||||
|
|
||||||||||||||||||
| # 9. Tell user to re-request Copilot review manually (no API exists for re-reviews) | ||||||||||||||||||
| echo "To trigger Copilot re-review, click the 🔄 button next to Copilot in the Reviewers section on the PR page." | ||||||||||||||||||
| # 9. Re-request Copilot review via API | ||||||||||||||||||
| gh api repos/chenders/deadonfilm/pulls/123/requested_reviewers \ | ||||||||||||||||||
| -X POST -f 'reviewers[]=copilot-pull-request-reviewer[bot]' | ||||||||||||||||||
| ``` | ||||||||||||||||||
|
|
||||||||||||||||||
| ## Copilot Review Workflow | ||||||||||||||||||
|
|
@@ -269,7 +270,12 @@ gh api repos/chenders/deadonfilm/pulls/123/requested_reviewers \ | |||||||||||||||||
| -X POST -f "reviewers[]=Copilot" | ||||||||||||||||||
| ``` | ||||||||||||||||||
|
|
||||||||||||||||||
| **Re-reviews:** There is no API or CLI to re-request a review from Copilot (or any reviewer) after it has already submitted one. The API returns success but silently does nothing. This is a [known GitHub limitation](https://github.com/orgs/community/discussions/186152). Re-reviews must be requested manually via the GitHub web UI (click 🔄 next to Copilot's name in the Reviewers section). | ||||||||||||||||||
| **Re-reviews:** Use the bot account name `copilot-pull-request-reviewer[bot]` (not `Copilot`) to re-request reviews: | ||||||||||||||||||
|
|
||||||||||||||||||
| ```bash | ||||||||||||||||||
| gh api repos/chenders/deadonfilm/pulls/123/requested_reviewers \ | ||||||||||||||||||
| -X POST -f 'reviewers[]=copilot-pull-request-reviewer[bot]' | ||||||||||||||||||
|
Comment on lines
+273
to
+277
|
||||||||||||||||||
| ``` | ||||||||||||||||||
|
|
||||||||||||||||||
| ### Reading Copilot Comments | ||||||||||||||||||
|
|
||||||||||||||||||
|
|
@@ -293,7 +299,7 @@ gh pr view 123 --json reviews | \ | |||||||||||||||||
| 4. **Commit the fix** - Use descriptive commit message (see Commit Formatting section) | ||||||||||||||||||
| 5. **Reply to the comment** - Explain what you did | ||||||||||||||||||
| 6. **Resolve the thread** - Only after implementing AND replying | ||||||||||||||||||
| 7. **Request re-review** - Tell user to click 🔄 in GitHub UI (no API exists for re-reviews) | ||||||||||||||||||
| 7. **Request re-review** - Use `copilot-pull-request-reviewer[bot]` via API | ||||||||||||||||||
|
|
||||||||||||||||||
| ### Responding to Copilot | ||||||||||||||||||
|
|
||||||||||||||||||
|
|
@@ -313,15 +319,22 @@ gh api -X POST "repos/chenders/deadonfilm/pulls/123/comments/1234567/replies" \ | |||||||||||||||||
|
|
||||||||||||||||||
| ### Requesting Re-review | ||||||||||||||||||
|
|
||||||||||||||||||
| After implementing fixes and replying to comments, tell the user to re-request Copilot review manually: | ||||||||||||||||||
| After implementing fixes, replying to comments, and pushing, re-request Copilot review via API: | ||||||||||||||||||
|
|
||||||||||||||||||
| ```bash | ||||||||||||||||||
| gh api repos/chenders/deadonfilm/pulls/123/requested_reviewers \ | ||||||||||||||||||
| -X POST -f 'reviewers[]=copilot-pull-request-reviewer[bot]' | ||||||||||||||||||
| ``` | ||||||||||||||||||
|
|
||||||||||||||||||
| **Important:** Use `copilot-pull-request-reviewer[bot]` (not `Copilot`) — the short name only works for initial reviews, not re-reviews. | ||||||||||||||||||
|
|
||||||||||||||||||
| > "To trigger a Copilot re-review, click the 🔄 re-request button next to Copilot's name in the Reviewers section on the PR page." | ||||||||||||||||||
| Poll for the new review to arrive: | ||||||||||||||||||
|
|
||||||||||||||||||
| **There is no API or CLI for re-reviews.** See [GitHub community discussion #186152](https://github.com/orgs/community/discussions/186152). | ||||||||||||||||||
| ```bash | ||||||||||||||||||
| gh api repos/chenders/deadonfilm/pulls/123/reviews --jq 'length' | ||||||||||||||||||
| ``` | ||||||||||||||||||
|
|
||||||||||||||||||
| **When Copilot re-reviews**: | ||||||||||||||||||
| - After the user clicks the re-request button in the GitHub UI | ||||||||||||||||||
| - When you push new commits (if auto-review on push is enabled in repo settings) | ||||||||||||||||||
| Poll every 15 seconds. Timeout after 10 minutes. | ||||||||||||||||||
|
chenders marked this conversation as resolved.
Outdated
|
||||||||||||||||||
| - When you mark the PR as ready for review (from draft) | ||||||||||||||||||
|
|
||||||||||||||||||
| ## Commit Message Formatting | ||||||||||||||||||
|
|
@@ -802,8 +815,9 @@ gh api graphql -f query=' | |||||||||||||||||
| } | ||||||||||||||||||
| ' | ||||||||||||||||||
|
|
||||||||||||||||||
| # 9. Tell user to re-request Copilot review manually | ||||||||||||||||||
| echo "To trigger Copilot re-review, click the 🔄 button next to Copilot in the Reviewers section on the PR page." | ||||||||||||||||||
| # 9. Re-request Copilot review via API | ||||||||||||||||||
| gh api repos/chenders/deadonfilm/pulls/123/requested_reviewers \ | ||||||||||||||||||
| -X POST -f 'reviewers[]=copilot-pull-request-reviewer[bot]' | ||||||||||||||||||
| ``` | ||||||||||||||||||
|
|
||||||||||||||||||
| ### Example 2: Complete Screenshot Workflow | ||||||||||||||||||
|
|
@@ -926,17 +940,19 @@ gh api "repos/chenders/deadonfilm/pulls/123/comments" | \ | |||||||||||||||||
|
|
||||||||||||||||||
| # Implement fixes, commit, reply, resolve (see Example 1) | ||||||||||||||||||
|
|
||||||||||||||||||
| # Round 2: Tell user to re-request review manually (no API for re-reviews) | ||||||||||||||||||
| echo "Click 🔄 next to Copilot in the PR Reviewers section to trigger re-review." | ||||||||||||||||||
| # Round 2: Re-request Copilot review via API | ||||||||||||||||||
| gh api repos/chenders/deadonfilm/pulls/123/requested_reviewers \ | ||||||||||||||||||
| -X POST -f 'reviewers[]=copilot-pull-request-reviewer[bot]' | ||||||||||||||||||
|
|
||||||||||||||||||
| # Wait for Copilot to re-review, then read new comments | ||||||||||||||||||
| # Poll for new review, then read new comments | ||||||||||||||||||
| gh api "repos/chenders/deadonfilm/pulls/123/comments" | \ | ||||||||||||||||||
| jq '.[] | select(.user.login == "Copilot") | select(.created_at > "2026-01-25T12:00:00Z")' | ||||||||||||||||||
|
|
||||||||||||||||||
|
chenders marked this conversation as resolved.
|
||||||||||||||||||
| # Implement any remaining fixes, commit, reply, resolve | ||||||||||||||||||
|
|
||||||||||||||||||
| # Round 3: Tell user to re-request again | ||||||||||||||||||
| echo "Click 🔄 next to Copilot in the PR Reviewers section to trigger re-review." | ||||||||||||||||||
| # Round 3: Re-request again | ||||||||||||||||||
| gh api repos/chenders/deadonfilm/pulls/123/requested_reviewers \ | ||||||||||||||||||
| -X POST -f 'reviewers[]=copilot-pull-request-reviewer[bot]' | ||||||||||||||||||
|
|
||||||||||||||||||
| # Once Copilot approves, merge | ||||||||||||||||||
| gh pr merge 123 --squash | ||||||||||||||||||
|
|
@@ -959,11 +975,9 @@ gh api graphql -f query='query { repository(owner: "OWNER", name: "REPO") { pull | |||||||||||||||||
| # Resolve thread | ||||||||||||||||||
| gh api graphql -f query='mutation { resolveReviewThread(input: {threadId: "PRRT_..."}) { thread { isResolved } } }' | ||||||||||||||||||
|
|
||||||||||||||||||
| # Request Copilot re-review — NO API EXISTS for re-reviews | ||||||||||||||||||
| # Tell user: "Click 🔄 next to Copilot in the PR Reviewers section" | ||||||||||||||||||
| # Initial review only (before Copilot has reviewed): | ||||||||||||||||||
| # Request Copilot review (initial or re-review) | ||||||||||||||||||
| gh api repos/OWNER/REPO/pulls/PR/requested_reviewers \ | ||||||||||||||||||
|
Comment on lines
+979
to
980
|
||||||||||||||||||
| # Request Copilot review (initial or re-review) | |
| gh api repos/OWNER/REPO/pulls/PR/requested_reviewers \ | |
| # Request Copilot review (initial) | |
| gh api repos/OWNER/REPO/pulls/PR/requested_reviewers \ | |
| -X POST -f 'reviewers[]=Copilot' | |
| # Request Copilot re-review | |
| gh api repos/OWNER/REPO/pulls/PR/requested_reviewers \ |
Uh oh!
There was an error while loading. Please reload this page.