Document live-sweep operational quirks in AGENTS.md#44
Merged
Conversation
…ona tunnel bridging, timeout sizing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documentation only — no code changes. Captures operational learnings from a real 29-task
autobrin-vs-pithos sweep across all four benchmarks, for whoever runs the next one:
git clone --branch <sha>failure mode (silent, ~1s, can coincidentallyscore as "correct" by luck) and the pre-seed workaround.
bridged to the remote sandbox via a tunnel, plus the DNS-propagation wait before it's reachable.
detectOnlydoesn't bound the lead/contributor phase, onlyevaluation depth; a full CyberGym engagement can lose an already-confirmed finding to a
too-tight external timeout since evaluation alone can run 45+ minutes after self-test passes.
GH_TOKEN-scope gotcha (gh auth tokenas a fallback).Test plan
Note
Low Risk
Documentation-only change with no runtime or scoring logic affected.
Overview
Adds a Running live comparison sweeps section to
AGENTS.mdthat records operational lessons from a 29-task autobrin-vs-pithos sweep.It tells operators to drive multi-contender sweeps via
runSingle()in a script rather thanbench run, because the CLI cannot build apithoscontender or setcontributors/maxCycles. The section also documents PITHOS cold-cache SHA clone failures (silent ~1s empty claims), Daytona + host-loopback targets (tunnel + wait for DNS beforecontender.run()), timeout budgeting (detectOnlyvs lead phase; CyberGym evaluation after confirmation; suggest 90+ minutes), andGH_TOKENvsgh auth tokenwhen private clones 403.Reviewed by Cursor Bugbot for commit f278e32. Configure here.