Conjecture Bench

A collection of unsolved mathematics problems, with checkers that decide whether a proposed answer is correct.

The purpose is to test whether an agent can produce a new mathematical result. Every problem here is open, so a wrong answer must never score. The checkers, not the problem count, are the part to scrutinise.

This is a development benchmark. It is not released and there is no leaderboard.

What is in it

15,607 problems in two parts.

The benchmark holds 9,644 problems, and a script can check 9,555 of them today:

	Problems	Checkable	What an answer is
Conjectures	302	244	a finite counterexample to one reviewed conjecture
Families	9,342	9,311	an object that beats the recorded state of the art

The extended catalog holds 5,963 more problems from published collections. Those are statements only, with no checker yet.

To browse everything, open a spreadsheet in problems/tables/. GitHub shows each one as a searchable table, already sorted so the problems closest to being settled are at the top.

All problems come from collections that other people compiled. Every source is named and credited, and every family record stores a quote from the page it cites so the claim can be re-checked against the source.

Try it

Look at a conjecture and check an answer to it:

python3 bench.py show cb-0043
python3 bench.py prompt cb-0043
python3 bench.py verify cb-0043 submission.json

Check an answer to a family problem:

python3 problems/benchmark/families/ljcr-covering-designs/verifier.py candidate.json

Both check one submission against one problem and print one verdict. To export the benchmark as JSONL for a runner:

python3 bench.py export --eligible > suite.jsonl

scripts/run_claude.sh runs a task with Claude Code end to end, and scripts/run_claude_batch.py runs an exported suite with resume.

Layout

problems/
  benchmark/
    conjectures/     one JSON file per reviewed conjecture
    families/        six families, one checker each
  extended-catalog/  problem statements from published collections
  tables/            spreadsheets over everything
  schemas/           the JSON contracts for every record type
  snapshots/         dated freezes of the benchmark
  status/            ledger of problems that lose open status
  index.json         manifest of the conjectures

verifiers/           the checking code, standard library only
scripts/             build, validate, and audit tools
tests/               the test suite
docs/                format and policy documents
research/            prompt variation experiments
bench.py             the command line interface

What a score means

An exact accepted counterexample scores 1. Everything else scores 0, including a well-formed statement that nothing was found, and including near misses.

A run of all zeros is the honest outcome for open problems. In our baseline runs on seven of the easiest family problems, no record was beaten. Measuring general capability needs separate problems with known answers, kept apart from these.

An accepted answer means the submitted object satisfies the frozen predicate for that problem. Acceptance is checked, not judged.

Problems can get solved

Each conjecture carries a dated status observation. When a problem is solved or its open status is disputed, an event is appended to the status ledger, and the next snapshot drops it from scoring. Old snapshots are immutable, so past runs stay interpretable.

To re-check the Erdős problems against their upstream source:

python3 scripts/audit_erdos_status.py

Validate the repository

python3 -m pip install -r requirements-dev.txt

python3 scripts/build_catalog.py --check
python3 scripts/build_index.py --check
python3 scripts/validate_catalog.py
python3 scripts/validate_tasks.py
python3 scripts/validate_verifiers.py
python3 scripts/validate_collections.py --expected-rows 60
python3 scripts/build_catalog_records.py --check
python3 research/variations/pipeline.py --check
python3 scripts/validate_families.py --check
python3 scripts/build_browse_tables.py --check
python3 -m pytest -q

This is the same list .github/workflows/validate.yml runs, in the same order. The checkers themselves need nothing installed; requirements-dev.txt is only for these checks.

Known limits

Nothing here proves a problem is genuinely open. Each record says who states that it is open, and where.

No checker has been independently reimplemented yet. Adversarial testing has not found a way to make one accept a wrong answer, but independent reimplementation is the stronger standard and it has not been done.

Further reading

• docs/benchmark_format.md, the task format
• docs/collection_policy.md, admission and release rules
• docs/families.md, the generated summary of the families
• problems/tables/README.md, how to read the tables