In plain English: This experiment tests whether representing known legal-form relationships explicitly helps a retrieval system find the correct facts more reliably than searching ordinary text chunks.
Court-form preparation systems must map a person's facts into a fixed, typed set of form fields. When the form schema already exists in PDF fields, data mappers, validation rules, and decoder grammars, rebuilding that structure from unstructured text may discard useful constraints.
This repository tests a narrow engineering question: does compiling an existing court-form schema into retrieval improve field completion compared with retrieving unstructured chunks or inducing a new key-value graph?
flowchart LR
S["Known form schema"] --> G["Compiled field-and-rule graph"]
F["Synthetic case facts"] --> R["Relational retrieval"]
G --> R
R --> O["Typed form-field output"]
O --> V["Exact-match analysis and verification"]
The evaluation compiles the prescribed structure of California Judicial Council form FW-001 into a typed retrieval graph, then holds the decoder, grammar, fixture pool, and evaluation procedure constant while changing the retrieval system.
The four conditions are:
- no retrieval;
- chunked retrieval over the same fixture corpus;
- an induced key-value graph baseline; and
- fixed-ontology graph retrieval compiled from the existing form artifacts.
The canonical Phase 1.5 run covers five decoders, three pool sizes, and three fixture seeds: 180 experimental cells and 178,500 scored field rows.
| Item | Inventory |
|---|---|
| Court form | 1: California FW-001, Request to Waive Court Fees |
| Fixture pools | 3 |
| Synthetic cases per pool | 200 |
| Compiled fields | 189 |
| Validator constraints | 2 |
| Real litigant records or PII | 0 |
All case facts are procedurally generated. TST-prefixed case identifiers, names, addresses, household facts, and financial facts are synthetic. The public package does not redistribute real filings.
The primary comparison is fixed-ontology retrieval versus matched chunked retrieval. The induced key-value graph is a same-budget local baseline, not a full reimplementation of HippoRAG, Microsoft GraphRAG, OG-RAG, or every published graph-retrieval system.
The analyzer reports field-level exact match, paired per-case accuracy deltas, bootstrap 95% confidence intervals, depth-specific deltas, and consistency across four public decoder families.
| Manifest-backed result | Value |
|---|---|
| Scored rows | 178,500 |
| Fixed ontology vs. chunked, public-decoder mean | +10.12 percentage points |
| Cross-decoder coefficient of variation | 0.347 |
| Cross-decoder analyzer verdict | TRANSFERS |
| Fixed ontology vs. induced baseline | PASS for all five decoders |
| Overall analyzer verdict | PASS |
PASS and TRANSFERS are pre-registered analyzer labels for this experiment.
They are not product certification, legal validation, or proof of production
readiness.
results/run-1777944639/poc.jsonl— field-level scored rows.results/run-1777944639/verdict.json— paired deltas, confidence intervals, and verdicts.data/schemas/fw-001.kg.json— compiled FW-001 schema.data/fixtures/— three synthetic fixture pools.scoping/PHASE1_5_SCOPING.md— pre-registered Phase 1.5 comparison and thresholds.paper/MANIFEST.source-of-truth.md— canonical claim and provenance manifest.ARTIFACT_SHA256SUMS— portable integrity index.
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
./scripts/verify_artifact.shThe verifier checks artifact hashes, scans for private-environment markers,
reruns the analyzer, and requires 178,500 rows, an overall PASS, a
TRANSFERS public-decoder result, and passing induced-baseline comparisons.
For a fixed three-minute walkthrough, use
INTERVIEW_DEMO.md.
To rerun only the analyzer:
PYTHONPATH=code python3 -m sweep.analyze results/run-1777944639- Legal AI Defensibility Packages evaluates whether an output's evidence package remains reconstructable after a defect is reported.
- Assistant Memory Risk in Legal Self-Help evaluates the tradeoff between repeated-question reduction and memory safety under three controlled policies.
- Public GitHub profile indexes the related reproducibility artifacts.
- The evaluation covers one California fee-waiver form and synthetic cases, not real litigant filings or multi-form packets.
- Exact field match is not legal correctness, legal sufficiency, usability, or a court outcome.
- The induced baseline is deliberately bounded; full published graph-retrieval implementations remain future comparison work.
- Phase 2—additional forms, full published baselines, and held-out jurisdictions—has not started.
- The system does not file, serve, pay, contact a court, obtain court acceptance, or provide legal advice.
- Neither the benchmark nor its implementation has been used in production.
- Designed by Justin Monk: the legal workflow question, fixed-ontology framing, pre-registration, experiment matrix, comparison controls, metrics, and decision thresholds.
- Implemented and integrated by Justin Monk: the schema compiler, retrieval/evaluation pipeline, canonical sweep, analyzer, integrity checks, and public reproducibility package.
- AI-tool assistance: OpenAI Codex and other coding assistants helped draft and refactor portions of the code and documentation and supported artifact review. Justin directed the work, selected the methods, ran the experiment, inspected the outputs, and accepted the released changes.
- Experimental: this is a research benchmark on controlled synthetic fixtures.
- Incomplete: multi-form, real-world, external-expert, and production validation remain open.
- Not legal advice: the repository evaluates retrieval behavior; it does not advise anyone how to complete or file a form.
- Not production use: no part of this repository has been used to submit a court filing or make a legal-validity determination.