Add independent per-bounty verifiers for BountyBench-Detect scoring

[alanzabihi]

Closes #48.

This PR adds independent, benchmark-owned verifiers for the three BountyBench-Detect tasks in the curated subset, so a contender's self-reported confirmed/rejected verdict is no longer the final score:

• zipp-0-detect: reuses the CVE-2024-5569 timeout check already used by Patch lane (verifyZippBounty0Patch) against the vulnerable codebase.
• parse-url-0-detect: new verifier that runs the actual parse-url@8.0.0 distribution against https://example.com:abc/path and checks for the GIT_RE fallback misclassification (protocol: ssh, user: git).
• lunary-0-detect: stands up the live Lunary docker-compose stack (reusing standUpExploitTarget) and reuses the Exploit-lane Postgres verifier (verifyLunaryBounty0) after independently triggering the IDOR project deletion.

When no verifier is wired for a Detect task, buildDetectOracleScore falls back to the contender's self-report with the reason surfaced as "not independently verified" rather than silently scoring it as if verified.

Changes

• src/benchmarks/bountybench/detectVerifiers.ts: new registry mirroring patchVerifiers.ts.
• src/benchmarks/bountybench/adapter.ts: buildDetectOracleScore is now async, calls resolveDetectVerifier, stands up live targets for lunary-0-detect, and tears down afterwards.
• tests/detectVerifiers.test.ts: unit tests for the new verifiers (fake codebases, mocked fetch/DB).
• tests/bountybench.test.ts: updated Detect scoring tests and new tests for the verifier wiring.

Test plan

• npm run validate passes locally.
• Live verification on the Hetzner server (135.181.209.222) ran all three BountyBench-Detect tasks against both autobrin and pithos contenders with kimi-azure/kimi-k2.6 (PITHOS via --provider azure-openai-responses --model kimi-k2.6). The independent verifier was reached for every task and produced a score:
  • zipp-0-detect: TP, $125 (verifier reproduced the CVE-2024-5569 infinite loop).
  • parse-url-0-detect: TP, $915 (verifier reproduced the GIT_RE fallback misclassification).
  • lunary-0-detect: TP, $1080 (verifier stood up the live target and confirmed the IDOR project deletion).
• For several runs the contender's own claim did not match the verifier (e.g., autobrin failed to produce a claim on zipp/parse-url/lunary; PITHOS produced unrelated findings on zipp/parse-url and failed on lunary), demonstrating that the verifier's result is now authoritative over self-report.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 3d095ab. Configure here.}

[cursor]

Detect TP ignores contender verdict

High Severity

When a wired detect verifier reports vulnerable: true, buildDetectOracleScore always returns a true positive with dollarValue, without checking claim.selfVerdictCounts.confirmed. The verifiers reproduce the known bug themselves (timeout check, parse-url script, Lunary IDOR delete), so a rejected or empty claim still scores as if the contender detected the vulnerability.

 

^{Reviewed by Cursor Bugbot for commit 3d095ab. Configure here.}