Enrich annotated output with a preset-driven check_info column

[Jamestth]

What

Two related changes to how vowl reports failures:

1. Annotated output now carries a richer, preset-driven check_info column (a JSON array of objects) in place of the old comma-joined check_ids string — so downstream users can read each failing check's quality dimension, tags, and target per data row.
2. Fix: percent-unit library checks (unit: "percent") were generating invalid SQL and coming back as ERROR instead of a real pass/fail verdict. Now fixed and guarded with tests.

---

Part 1 — Annotated check_info column

Replaces the annotated table's check_ids column (comma-joined check names) with a single preset-driven check_info column that emits a JSON array of objects, one per failing check. Passing rows stay null.

Presets

All three presets emit the same type — a JSON array of objects — so consumers parse uniformly via item["check_name"]; they differ only in how many keys each object carries:

• "names" (default) — [{"check_name": ...}]
• "summary" — [{check_name, dimension, tags, target}]
• "full" — [<full check_definition> + check_name + target]

Pick the preset on the config (ValidationConfig.annotated_check_info, default "names") or per call: get_annotated_output(check_info=...) / save(check_info=...).

Residues are now per-check

Checks that can't be folded onto a single table (cross-table, aggregation, column-subset) surface under "residues". These are now one entry per non-mergeable check, keyed "<schema>::<check_name>" (previously several checks could be grouped under one composite-table key like "table_a, table_b"). Residues carry the same check_info column as the annotated tables (a single-element JSON array, shaped by the same preset) plus tables_in_query — so everything get_annotated_output() returns is read uniformly. The legacy consolidated/failed-rows path (output_mode="failed_rows"/"both") is deliberately untouched and keeps its comma-joined check_ids column.

Changes

• config.py — CheckInfoPreset literal + ValidationConfig.annotated_check_info = "names".
• validation/result.py — _check_info_item_json / _join_check_info_items helpers; check_info threaded through get_annotated_output() and save(), resolved against config (mirrors output_mode precedence). _group_check_ids_by_row / _with_null_marker emit check_info. Per-check residue logic via _build_residue_with_check_info (residues emit check_info, not the legacy check_ids). The include_target/targets surface is removed (target now lives inside the objects).
• save() writes per-check residues as <prefix>_<schema>_<check>_residue.csv in "annotated" mode; in "both" mode residues are not re-emitted (those rows already live in the grouped failed-rows CSVs).
• Tests extended (preset shapes, multi-check dedup/order, passing-row nulls, residue/CSV regression guards); README, docs/known-issues.md, and the usage-patterns notebook updated; demo artifacts regenerated.

Breaking change (pre-1.0)

Annotated column renamed check_ids → check_info, contents changed from a plain string to JSON, and include_target/targets removed (target now lives inside each object). Acceptable for v0.0.3.

---

Part 2 — Fix: percent-unit library checks emitting invalid SQL

Library checks written with unit: "percent" (e.g. nullValues, missingValues, invalidValues, duplicateValues) build a query of the form (count * 100.0) / NULLIF(total, 0), using the two counts as scalar subqueries inside an arithmetic expression.

_wrap_percent was attaching table aliases ((SELECT ...) AS _cnt, ... AS _tot) to those subqueries. That's invalid SQL in expression position — DuckDB raises syntax error at or near "AS" and sqlglot can't even re-parse its own output. The result: every percent check came back as ERROR instead of a real verdict, and the existing string-only assertions ("100" in query) didn't catch it.

Fix: emit the subqueries without aliases (.subquery() instead of .subquery("_cnt") / .subquery("_tot")), with a docstring explaining why aliasing is wrong here.

Regression guards:

• TestPercentMetricsValidSql re-parses the rendered SQL through sqlglot for every percent metric — dialect-independent, no live backend needed.
• TestPercentChecksExecuteEndToEnd runs two percent checks through the full validate_data pipeline against a real DuckDB and asserts a real FAILED verdict with the correct ratio (25% null, 50% duplicate) rather than ERROR. Two golden fixtures capture the expected check-results CSVs.

Percent checks are scalar verdicts (a single ratio, not per-row), so they remain non-mergeable and produce no annotated rows — the point of the fix is purely that they now run and return a real verdict.

---

Verification

• uv run pytest tests/test_annotated_output.py tests/test_check_reference_variations.py tests/test_pooled_adapter_integration.py -q → all green
• make test → all green
• Manual save(output_mode="annotated", check_info="summary") produces the expected JSON-array column, and per-check residue CSVs carry the same check_info column.

🤖 Generated with Claude Code