Skip to content

Add CVE-Bench benchmark adapter#15

Draft
homanp wants to merge 2 commits into
mainfrom
homanp/cve-bench-adapter
Draft

Add CVE-Bench benchmark adapter#15
homanp wants to merge 2 commits into
mainfrom
homanp/cve-bench-adapter

Conversation

@homanp

@homanp homanp commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds an official-compatible CVE-Bench benchmark path for PITHOS. The new pithos cvebench commands can run doctor, pull, smoke, and eval flows, build a Pi-enabled CVE-Bench Kali agent image, use a Pi-backed Inspect solver, and write normalized benchmark artifacts.

Fixes #

Checklist

  • I tested my changes
  • I reviewed my own code

Test plan

  • uv run ruff check .
  • uv run ruff format --check .
  • uv run pytest
  • uv run python -m compileall pithos tests
  • uv build

@homanp

homanp commented Jun 20, 2026

Copy link
Copy Markdown
Contributor Author

@superagent review

@superagent-security superagent-security Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superagent found 2 security concern(s).

Comment thread pithos/cvebench_runner.py
Comment thread pithos/cvebench_runner.py
@superagent-security superagent-security Bot added pr:flagged Superagent PR scan flagged and removed pr:flagged Superagent PR scan flagged labels Jun 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant