Skip to content

Document live-sweep operational quirks in AGENTS.md#44

Merged
alanzabihi merged 1 commit into
mainfrom
agents-md-sweep-notes
Jul 2, 2026
Merged

Document live-sweep operational quirks in AGENTS.md#44
alanzabihi merged 1 commit into
mainfrom
agents-md-sweep-notes

Conversation

@alanzabihi

@alanzabihi alanzabihi commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Documentation only — no code changes. Captures operational learnings from a real 29-task
autobrin-vs-pithos sweep across all four benchmarks, for whoever runs the next one:

  • PITHOS's cold-cache git clone --branch <sha> failure mode (silent, ~1s, can coincidentally
    score as "correct" by luck) and the pre-seed workaround.
  • Daytona-transport webapp tasks (CVE-Bench, BountyBench Exploit) need the host-loopback target
    bridged to the remote sandbox via a tunnel, plus the DNS-propagation wait before it's reachable.
  • Real observed timeout data: detectOnly doesn't bound the lead/contributor phase, only
    evaluation depth; a full CyberGym engagement can lose an already-confirmed finding to a
    too-tight external timeout since evaluation alone can run 45+ minutes after self-test passes.
  • The GH_TOKEN-scope gotcha (gh auth token as a fallback).

Test plan

  • Docs-only change, no code affected.

Note

Low Risk
Documentation-only change with no runtime or scoring logic affected.

Overview
Adds a Running live comparison sweeps section to AGENTS.md that records operational lessons from a 29-task autobrin-vs-pithos sweep.

It tells operators to drive multi-contender sweeps via runSingle() in a script rather than bench run, because the CLI cannot build a pithos contender or set contributors / maxCycles. The section also documents PITHOS cold-cache SHA clone failures (silent ~1s empty claims), Daytona + host-loopback targets (tunnel + wait for DNS before contender.run()), timeout budgeting (detectOnly vs lead phase; CyberGym evaluation after confirmation; suggest 90+ minutes), and GH_TOKEN vs gh auth token when private clones 403.

Reviewed by Cursor Bugbot for commit f278e32. Configure here.

@alanzabihi alanzabihi merged commit 361c5c6 into main Jul 2, 2026
2 checks passed
@alanzabihi alanzabihi deleted the agents-md-sweep-notes branch July 2, 2026 16:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant