discover-verified-data

An evidence-grounded Codex Skill for turning a natural-language data request into a reviewable specification, finding public records through stable protocols, reconciling field-level evidence, and exporting versioned CSV/XLSX snapshots.

The repository contains:

• skills/discover-verified-data/SKILL.md — the Codex Skill.
• skills/discover-verified-data/scripts/discover_verified_data.py — deterministic validation, collection, verification, export, and refresh CLI.
• references/ and assets/ — data contracts, source policy, adapter guidance, and templates.
• examples/ — an offline protocol fixture, explicitly not a real-world dataset catalogue.
• tests/ — fixed-response and boundary tests.

Safety and evidence boundary

The Skill does not bypass authentication, paywalls, CAPTCHA, robots rules, rate limits, firewalls, port controls, or access controls. Blocked sources are recorded and replaced only with lawful alternatives such as official APIs, public bulk downloads, institutional repositories, or user-provided exports.

The formal dataset contains observed or reproducible derived values with usable evidence. Model estimates and synthetic fixtures are isolated from the formal table. A plausible value is not treated as a verified fact, and a strong downstream metric is not treated as field validity.

Install

python -m pip install -e ".[dev]"

The Skill folder can then be copied to the Codex skills directory, or invoked explicitly from its repository path.

Quick start

$script = "skills/discover-verified-data/scripts/discover_verified_data.py"
python $script validate-spec --spec skills/discover-verified-data/assets/dataset_spec.example.yaml
python $script collect --spec skills/discover-verified-data/assets/dataset_spec.example.yaml --source-config examples/predictive-maintenance/sources-fixture.yaml --output-dir runs/example
python $script verify --spec skills/discover-verified-data/assets/dataset_spec.example.yaml --evidence runs/example/candidate_evidence.jsonl --source-ledger runs/example/source_ledger.csv --output-dir runs/example
python $script export --spec skills/discover-verified-data/assets/dataset_spec.example.yaml --verified-json runs/example/verified_rows.json --output-dir runs/example

For a versioned refresh:

python $script refresh --spec skills/discover-verified-data/assets/dataset_spec.example.yaml --source-config examples/predictive-maintenance/sources-fixture.yaml --output-dir snapshots

Outputs

dataset.csv is the formal table. evidence_ledger.csv, source_ledger.csv, run_manifest.json, and quality_report.* make every non-empty formal field auditable. dataset.xlsx contains Data, Estimates, Evidence, Sources, Changes, and Run Summary worksheets. Refreshes create a new snapshot, changes.csv, latest.json, and a local SQLite state file.

Supported first routes

The generic collector supports local or public HTTP(S) CSV, JSON, XLSX, and HTML tables. The adapter catalog documents how to extend this with CKAN/DCAT, SDMX, Crossref, OpenAlex, DataCite, Zenodo, GitHub, and official government portals without turning the Skill into a collection of fragile site-specific scrapers.

Development

python scripts/validate_skill.py
python -m unittest discover -s tests -v

The GitHub Actions workflow repeats structure validation, tests, and the offline example. CI never depends on live websites.