benchmark_format.md

Portable benchmark format

Design goal

The smallest useful public contract is:

1. one JSON task file;
2. one JSON submission;
3. one deterministic verifier result.

Everything else—Claude Code, Harbor, containers, prompt templates, dashboards, and distributed execution—is an adapter around that contract.

Task record

Every file in problems/benchmark/ contains:

• a stable task ID and exact statement;
• the finite object that would refute it;
• a draft-07 JSON Schema for agent output;
• either an executable verifier entry point or an explicit design-only marker;
• frozen resource limits;
• a dated status observation and sources.

The JSON points to verifier code rather than embedding code as a string. Snapshot manifests hash both artifacts.

Submission envelope

Every agent returns the same top-level shape:

{
  "task_id": "cb-0043",
  "outcome": "counterexample",
  "witness": {
    "n": "4",
    "factorization": [
      {"prime": "2", "exponent": 2}
    ]
  },
  "notes": "optional"
}

When an agent found nothing:

{
  "task_id": "cb-0043",
  "outcome": "no-result",
  "witness": null
}

Large mathematical integers are canonical nonzero decimal strings. This avoids the 53-bit exact-integer boundary in common JSON runtimes. Small bounded indices, exponents, lengths, and ±1 sequence entries remain JSON integers.

Verifier result

python3 bench.py verify TASK SUBMISSION emits:

{
  "protocol_version": "1.0",
  "task_id": "cb-0043",
  "verdict": "rejected",
  "score": 0,
  "message": "phi(n) does not divide n−1",
  "details": {"phi": 2},
  "snapshot_id": "development-2026-07-24-r52",
  "status_cutoff": "2026-07-24"
}

Verdicts are accepted, no-result, rejected, not-executable, not-eligible, quarantined, snapshot-mismatch, or error. Only accepted has score 1. Non-scorable states use null.

Snapshot and status model

Task files are immutable within a snapshot, but mathematical status changes. The snapshot stores:

• status cutoff;
• SHA-256 of every task;
• SHA-256 of every referenced verifier source;
• status and execution disposition;
• development and release eligibility.

The append-only status ledger overlays a baseline observation. A quarantine event immediately removes the task from new scoring manifests. A restore event requires reviewed evidence. Historical manifests are never rewritten.

review-pending is distinct from quarantined: the former lacks enough independent status evidence; the latter has an affirmative reason to suspect the task is no longer open.

Variations

A presentation variation is not a new conjecture. It belongs under research/variations/ and must retain:

• the canonical task ID;
• a digest of the unchanged mathematical payload;
• a variation family ID used for grouped train/test splitting.

A changed statement is either a declared shadow task or a new canonical task. Known-counterexample shadows are useful for dense capability measurement, but must be kept out of the frontier-open score.

Executable runtime renderings belong under research/variations/adapters/<runtime>/. A Harbor task can therefore be generated as soon as a portable task becomes runnable without making Harbor the permanent mathematical source format.

Claude Code adapter

Claude Code supports noninteractive print mode and JSON-Schema-constrained structured output. The included adapter reduces a run to:

scripts/run_claude.sh cb-0043 --model opus --max-turns 50

The model name, turn limit, tools, and budget are run metadata, not task metadata. They must be frozen separately for a published comparison.

Portable JSONL suites

bench.py export emits one self-contained JSON object per line. Each job contains the rendered prompt, exact output schema, frozen task metadata, and an allow-listed verifier command:

python3 bench.py export --eligible > suite.jsonl

The row contract is problems/schemas/suite.schema.json. It is runtime-neutral: a harness may send prompt to any model, constrain the response with output_schema, and feed the response to verification.command.

The included Claude Code batch adapter validates every row and rejects arbitrary commands before execution. Its default mode makes no model calls:

python3 scripts/run_claude_batch.py \
  suite.jsonl \
  --output runs/development-r52

Add --execute to make calls, --jobs N for concurrency, and --resume when continuing an existing run. Canonical verifier rejection is a completed benchmark result, not a process failure.

Discovery research contract

Broad source inventories remain under problems/extended-catalog/ until they pass task admission. They use the same top-level ideas—statement, status, provenance, promotion gates, and verifier readiness—but do not have a task-specific witness schema or mathematical checker yet.

Secondary-source rows may also carry statement quality_flags, a pinned raw source URL, and a separate human-readable context URL. A well-formed discovery record with a collision or extraction flag is still only a repair assignment; the flag must be resolved against a primary source before promotion.

The runner can still render a constrained research prompt:

python3 bench.py discovery prompt kourovka-main-01-003

It can also export filtered discovery jobs:

python3 bench.py discovery export \
  --source kourovka-notebook-21 \
  --status listed-unsolved \
  --limit 100 \
  > research-suite.jsonl

Its generic output distinguishes counterexample candidates, proof candidates, status findings, verifier designs, formalization needs, and no-result. Passing that generic JSON Schema—and the corresponding discovery validate command—means only that the research report is well formed; it does not verify a proof or witness and never produces a benchmark score.