Agent Royale

Unit tests for AI agents and retrieval layers that browse the web.

Live site: https://agentroyale.onrender.com/

AI agents now browse, search, scrape, and cite the web. Agent Royale tests whether they return the exact value your workflow asked for, from the source you required.

It is built for questions where there should be one current answer:

• the latest package version on npm
• the release tag on GitHub
• the price on an official pricing page
• the app rating from an app-store listing
• the quote field from a finance source
• the company metric from a required public profile

If the source is real but the returned value is stale, from the wrong field, from the wrong page, or unsupported by the citation, Agent Royale turns that into a repeatable test failure.

Product Loop

task pack -> oracle audit -> target run -> report -> compare before/after

1. Define source-specific questions in YAML.
2. Fetch independent ground truth from a public API, page parser, or Bright Data.
3. Call the target agent or retrieval layer.
4. Grade exact values deterministically.
5. Store provenance, evidence snippets, oracle status, grading trace, citation checks, latency, and cost.
6. Compare runs after prompt, model, parser, or retrieval changes.

No LLM judge is used for the core exact-value grade.

See Reliability model for the diagrams, failure-mode tables, and report-field guide that explain why the scoring is trustworthy.

Reports are built for quick decisions: agent answer vs. ground truth, the tool used, source-supported accuracy, scoreable tasks, oracle skips, latency, reported cost, failure breakdowns, and concrete "what to fix next" guidance. Ground-truth and grading details stay in the reliability docs so the report itself stays clean.

Quickstart

pip install -e .
agent-royale --version

Run the offline smoke test:

python -m agent_royale demo

Or run the steps directly:

python -m agent_royale lint task-packs/static-smoke.yaml
python -m agent_royale audit task-packs/static-smoke.yaml
python -m agent_royale run task-packs/static-smoke.yaml \
  --target examples/echo_agent.py:answer \
  --report reports/smoke.html \
  --summary reports/smoke-summary.json \
  --junit reports/smoke.xml

Watch the runner catch a wrong answer:

python -m agent_royale run task-packs/static-smoke.yaml \
  --target examples/flaky_agent.py:answer \
  --report reports/failure-demo.html

Reproduce one task by ID when debugging:

python -m agent_royale run task-packs/static-smoke.yaml \
  --target examples/flaky_agent.py:answer \
  --only smoke_price \
  --report reports/smoke-price-debug.html

Targets can be:

• a local or staging HTTP endpoint, such as http://localhost:3000/api/agent
• an OpenRouter model adapter, such as openrouter:provider/model
• a local Python function, such as examples/echo_agent.py:answer

Sweep the same task pack across multiple models or targets:

python -m agent_royale sweep task-packs/static-smoke.yaml \
  --targets examples/echo_agent.py:answer,examples/flaky_agent.py:answer

For OpenRouter model sweeps:

python -m agent_royale sweep task-packs/devtools/dependency-research.yaml \
  --models openai/gpt-4o,google/gemini-3-1-flash-lite

Sweep outputs include per-target JSONL runs, HTML reports, JSON summaries, JUnit XML, and a ranked sweep-summary.md with a recommendation and outcome breakdowns.

Golden Path

The dependency-research example is the best first demo because it is useful, public, and reproducible without API keys. It models a developer assistant that answers npm and GitHub metadata questions.

python -m agent_royale lint task-packs/devtools/dependency-research.yaml
python -m agent_royale audit task-packs/devtools/dependency-research.yaml

python -m agent_royale run task-packs/devtools/dependency-research.yaml \
  --target examples/dev_research_agent.py:answer \
  --output runs/dependency-research-before.jsonl \
  --report reports/dependency-research-before.html

python -m agent_royale run task-packs/devtools/dependency-research.yaml \
  --target examples/dev_research_agent_fixed.py:answer \
  --output runs/dependency-research-after.jsonl \
  --report reports/dependency-research-after.html

python -m agent_royale compare \
  runs/dependency-research-before.jsonl \
  runs/dependency-research-after.jsonl \
  --markdown reports/dependency-research-comparison.md

On the current public sources, the intentionally imperfect target scores 57.1% and the fixed target scores 100.0%. The comparison identifies three concrete improvements:

• github_playwright_latest_release: wrong source fixed
• github_vscode_open_issues: wrong field semantics fixed
• npm_next_weekly_downloads: wrong time window fixed

See docs/golden-path.md for the full walkthrough.

What You Get

• Task packs: YAML files for exact, source-specific retrieval tests
• Oracle audit: preflight checks that verify ground truth before testing a target
• Ground-truth snapshots: source URL, fetch time, parser metadata, evidence text, oracle status, and ambiguity flags
• Oracle policies: task-level checks for evidence support, single-candidate extraction, confidence, and source freshness
• Target adapters: HTTP endpoint, Python function, OpenRouter, and examples for web data, search, browser, and scraping stacks
• Deterministic graders: string, number, currency, percentage, date, and enum matching
• Citation policies: source matching by contains, exact URL, same path, same domain, or explicit allowed sources
• Failure labels: wrong value, wrong source, unsupported citation, no answer, tool failure, oracle ambiguity, correct-but-unsupported
• Reports: clean HTML decision dashboard, terminal summary, JSONL run log, JSON summary, and JUnit XML
• Demo and sweeps: one-command offline demo plus multi-target/model sweeps for choosing the right stack
• Run comparison: before/after accuracy, source-supported accuracy, oracle skips, latency, cost, regressions, and Markdown output
• Task-pack linting: static checks for fragile oracles, volatile CI gates, broad regexes, missing provenance, weak search-result ground truth, missing pack versions, and permissive CI source matching
• CI gates: nonzero exit when accuracy drops below a threshold

Why Ground Truth Is Treated Carefully

Ground truth is the product. Agent Royale does not assume a live page is stable just because it was fetched.

flowchart LR
    A["Task pack"] --> B["Oracle audit"]
    B --> C{"Verified ground truth?"}
    C -- "yes" --> D["Run target"]
    C -- "no" --> E["Skip scoring + salvage report"]
    D --> F["Deterministic grade"]
    F --> G["Source/citation checks"]
    G --> H["HTML/JSON/JUnit reports"]

Loading

Every scored run stores an oracle snapshot, task-pack metadata, a stable task hash, grader version, oracle version, normalized values, and citation checks. If the oracle cannot verify a value, returns conflicting candidates, misses required context, violates the task's oracle_policy, or looks ambiguous, the task is skipped instead of counted against the target. Task packs can mark sources as stable, semi_stable, or volatile, and volatile tasks can be excluded from CI with ci_safe: false.

For build-blocking checks, use stable packs backed by public APIs or slow-moving source fields. For fast-changing pages such as ecommerce prices, social counts, and dynamic public profiles, use scheduled or on-demand reports.

Task packs can tighten oracle and source requirements:

name: pricing-pages-v1
version: 1.0.0
tasks:
  - id: example_pro_price
    question: "Using Example Pricing, what is the current Pro plan monthly price in USD?"
    required_source: "example.com/pricing"
    answer_type: currency
    tolerance: 0
    stability: stable
    ci_safe: true
    source_policy:
      match: same_path
      require_quote: true
    oracle_policy:
      require_single_candidate: true
      require_evidence_contains_value: true
      require_confidence: verified
    ground_truth:
      method: http_regex
      url: "https://example.com/pricing"
      source_url: "example.com/pricing"
      regex: "Pro[\\s\\S]{0,400}?\\$\\s*([0-9]+(?:\\.[0-9]{2})?)"
      require_near_text:
        - "Pro"
        - "monthly"

Source-supported accuracy is stricter than exact-value accuracy: the answer must match the oracle, cite the required source under the task's source_policy, and include a quote that supports the extracted claim.

Task Packs

The repo includes 79 reusable tasks:

task-packs/static-smoke.yaml                     offline smoke tests
task-packs/devtools/dependency-research.yaml     reproducible golden-path demo
task-packs/devtools/dependency-research-v1.yaml  flagship dependency metadata eval
task-packs/devtools/docs-freshness-v1.yaml       docs and release freshness eval
task-packs/business/saas-pricing-v1.yaml         SaaS pricing accuracy eval
task-packs/github/example.yaml                   GitHub counts, releases, files, branches, licenses
task-packs/npm/example.yaml                      npm versions, licenses, downloads, repository URLs, package size, engines
task-packs/finance/yahoo-quotes.yaml             Yahoo Finance quote fields
task-packs/mobile-apps/apple-app-store.yaml      Apple App Store rating and version fields
task-packs/subscription-pricing/example.yaml     official pricing-page examples
task-packs/bright-data/rapid-web.yaml            Bright Data Rapid-mode docs/search checks
task-packs/bright-data/ecommerce-accuracy-v1.yaml Samsung ecommerce variant and title accuracy
task-packs/bright-data/linkedin-company.yaml     LinkedIn company metrics
task-packs/bright-data/ecommerce-pricing.yaml    ecommerce product pricing

Create a starter pack:

python -m agent_royale init task-pack cloud-pricing

Create a Bright Data Rapid-mode starter pack:

python -m agent_royale init task-pack product-pricing --ground-truth bright-data-rapid

Validate and lint all public packs:

python -m agent_royale validate task-packs
python -m agent_royale lint task-packs

The best first contribution is a task pack for a source your own agent depends on. See TASK_PACK_IDEAS.md and docs/task-spec.md.

For the V3 external-user focus, see V3 task banks: developer dependency/docs research, ecommerce product/pricing accuracy, and company intelligence.

Bright Data Ground Truth

Bright Data is useful when the required source is a messy public web page instead of a clean API. Agent Royale uses Bright Data live scraping paths for source-specific public web ground truth rather than relying on stale cached datasets. It supports Bright Data as:

• an independent ground-truth oracle for task packs
• a target retrieval layer through the local Bright Data adapter example

Rapid-mode setup:

export BRIGHT_DATA_API_KEY=...
export BRIGHT_DATA_MCP_URL=https://mcp.brightdata.com/mcp
python -m agent_royale doctor task-packs/bright-data/rapid-web.yaml --check-ground-truth

Export an oracle error audit for tasks that need parser/tool cleanup:

python -m agent_royale audit-errors task-packs/bright-data \
  --output reports/bright-data-error-audit.md \
  --json reports/bright-data-error-audit.json \
  --csv reports/bright-data-error-audit.csv

The audit lists failed or ambiguous tasks with the Bright Data tool, source URL, parser, validation checks, error text, and a suggested salvage action. Use it to debug empty results, selector drift, fallback issues, or tasks that need a rendered/browser workflow.

Run the ecommerce accuracy pack against the Bright Data target adapter:

cd examples/bright-data-agent
pip install -r requirements.txt
uvicorn app:app --host 127.0.0.1 --port 3001

cd ../..
python -m agent_royale run task-packs/bright-data/ecommerce-accuracy-v1.yaml \
  --target http://127.0.0.1:3001/api/agent \
  --report reports/bright-data-ecommerce-agent.html

See docs/bright-data.md for tool selection, Rapid mode, structured tools, and oracle design.

Original Experiment

Agent Royale started as an experiment with 32 live-web tasks and 12 model/retrieval stacks. The tested stacks averaged 54% exact accuracy, and the best stack reached 78%.

Treat that result as motivation, not a universal benchmark. The important finding was qualitative: agents often cited plausible sources while returning stale values, wrong fields, or values from nearby pages.

See docs/experiments/dev-web-retrieval-v1.md for a larger reproducible eval and methodology notes.

Stack Fit Eval

Agent Royale can also be used as a fit test for web retrieval infrastructure. The Stack Fit eval groups tasks by workflow so teams can test URL readers, scrape/extract APIs, search APIs, dynamic-web extractors, and model-search stacks without treating unlike tools as a single leaderboard.

The current public evidence includes:

• Jina Reader on known-source reading: 5/5 exact
• Firecrawl on known-source reading: 5/5 exact
• Tavily extract on known-source extraction: 4/5 exact
• Tavily search on the same known-source pack: 1/5 exact, showing why endpoint choice should be evaluated
• Bright Data on focused dynamic ecommerce extraction: 3/3 exact
• OpenRouter GPT-4o Mini on search/discovery: 2/3 exact
• OpenRouter GPT-4o Mini on Dev Web Retrieval Eval v1: 20/28 exact

See docs/experiments/web-retrieval-stack-fit-v1.md for the lane design, reports, and reproduction commands.

The focused Search Is Not Reading v1 experiment shows why this matters: on the same known-source task pack, Tavily extract scored 4/5 exact while Tavily search scored 1/5 exact.

OpenRouter Model Comparison v1 compares OpenAI GPT-4o, Gemini 3.1 Flash-Lite, and Perplexity Sonar Pro Search on the same known-source pack.

CI

Use CI for stable task packs:

name: Agent Royale

on:
  pull_request:
  push:
    branches: [main]

jobs:
  retrieval-eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: python -m agent_royale validate task-packs
      - run: python -m agent_royale lint task-packs/static-smoke.yaml
      - run: |
          python -m agent_royale run task-packs/static-smoke.yaml \
            --target examples/echo_agent.py:answer \
            --ci \
            --report reports/agent-royale.html \
            --summary reports/agent-royale-summary.json \
            --junit reports/agent-royale.xml \
            --fail-under-exact 1.0
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: agent-royale-report
          path: |
            reports/agent-royale.html
            reports/agent-royale-summary.json
            reports/agent-royale.xml

See docs/github-actions.md for endpoint, OpenRouter, Bright Data, and comparison examples.

Docs

• Golden path
• Quickstart
• Task spec
• Reliability model
• V3 task banks
• Adapter contract
• GitHub Actions
• Integrations
• Bright Data ground truth
• OpenRouter eval
• Realistic dev-agent eval
• Web Retrieval Stack Fit Eval v1
• Search Is Not Reading v1
• OpenRouter Model Comparison v1

Repo Structure

agent_royale/         core runner, grader, lint, compare, report generation
backend/              FastAPI app and live-site API
frontend/             static frontend
task-packs/           public task packs
examples/             local target adapters
docs/                 usage and methodology docs
data/                 original task bank data
storage/              local JSONL storage

Contributing

The best first PR is a task pack for a source your own agent depends on.

• CONTRIBUTING.md
• TASK_PACK_IDEAS.md
• ROADMAP.md

Contact

vinayasharma00@gmail.com