fix(faithfulness): thread COMP composition into report_2x2; correct 2x2 to 31.8%

report_2x2.py recomputed faithfulness on the fly and called the COMP checker
with composition=None -- the same over-reporting bug rescore_faithfulness.py
fixes elsewhere. Since COMP is ~54% of the dataset, this inflated the headline
faithful rate to 45.6%. Thread the real composition (lazy variant_id->composition
map from the dataset) for COMP rows only; non-COMP scoring is unchanged.

Corrected overall faithful rate: 45.6% -> 31.8% (COMP 58.2% -> 32.7%, parse%
0.0% -> 77.4%). README 2x2 table, per-strategy rates, and COMP/held-out scoring
caveats updated to match the regenerated report.

run_faith_rescore.sh: switch phase 2 (6 venv-scored models) from the 9-config
recompute equivalence path to reuse -- the recompute path spawned parallel
compilers + spinning candidate binaries that OOM-crashed the machine and would
take weeks; reuse keeps the (correctness-based) equivalence already in the CSV,
identical to recompute's fallback for non-overfit candidates, while the COMP
structural fix still applies in both modes. Sequential, no compilation.

Author: wlu03

README.md

@@ -56,9 +56,15 @@ The patterns are organized by *why* the compiler fails to fix them:
 
 | | Faithful | Unfaithful | Row |
 |---|---|---|---|
-| **Fast** | 23.6% | 6.7% | 30.2% |
-| **Slow** | 22.0% | 47.7% | 69.8% |
-| **Col** | 45.6% | 54.4% | 100% |
+| **Fast** | 18.7% | 11.5% | 30.2% |
+| **Slow** | 13.0% | 56.7% | 69.8% |
+| **Col** | 31.8% | 68.2% | 100% |
+
+Per-strategy faithful rates are close: generic 30.0%, pattern-aware 33.8%, taxonomy-guided 31.5%.
+
+**Faithfulness-scoring caveats.** Two structural factors shape this aggregate, and the headline rate is sensitive to both:
+- **COMP composition (≈54% of rows).** COMP variants are scored against their constituent-pattern list (`composition` from `metadata.json`); the COMP checker *requires* it, and without it falls back to a generic regex battery that massively over-reports `FAITHFUL`. Earlier runs omitted it and reported an inflated ~45.6% overall (COMP alone read 58% faithful); both `faithfulness/report_2x2.py` and `scripts/rescore_faithfulness.py` now thread `composition`, which drops COMP to 33% faithful and the overall rate to 31.8%.
+- **Held-out patterns (`HO-*`, ≈14% of rows).** These post-cutoff patterns have no dedicated AST checker and fall through to a coarse structural fallback, so they essentially cannot earn a `FAITHFUL` verdict and weigh toward the unfaithful column. Authoring per-pattern held-out checkers is the remaining faithfulness-coverage gap.
 
 ### Findings from the sweep
 

faithfulness/report_2x2.py

@@ -60,6 +60,29 @@
 except Exception:
     _PATTERNS_BY_ID = {}
 
+# COMP variants carry their constituent pattern list in dataset metadata. The
+# COMP checker NEEDS that list: without it, it falls back to a generic regex
+# battery that massively over-reports `FAITHFUL` (COMP is ~54% of the dataset,
+# so this dominates the aggregate). Build variant_id -> composition lazily so
+# non-COMP runs pay nothing, and resolve the dataset relative to the repo root
+# so the lookup works regardless of the caller's cwd.
+_COMP_COMPOSITION: Optional[dict] = None
+
+
+def _composition_for(variant_id: str):
+    global _COMP_COMPOSITION
+    if _COMP_COMPOSITION is None:
+        _COMP_COMPOSITION = {}
+        try:
+            from pdob_core.dataset_evaluator import discover_variants  # type: ignore
+            for v in discover_variants(str(_HERE.parent / "dataset")):
+                comp = v.metadata.get("composition") if getattr(v, "metadata", None) else None
+                if comp:
+                    _COMP_COMPOSITION[v.variant_id] = comp
+        except Exception:
+            pass
+    return _COMP_COMPOSITION.get(variant_id)
+
 
 # --------------------------------------------------------------------------
 # Instrumentation: wrap PatternChecker.check + _parse so we can report
@@ -229,7 +252,16 @@ def _compute_faithfulness_for_row(
 
     _INSTR.reset()
     try:
-        result = check_faithfulness(pid, slow_code, llm_code)
+        if pid == "COMP":
+            # Thread the composition so COMPChecker scores the actual
+            # constituent transforms instead of the over-reporting regex
+            # fallback. Mirrors scripts/rescore_faithfulness.py.
+            result = check_faithfulness(
+                pid, slow_code, llm_code,
+                composition=_composition_for(row.get("variant_id", "")),
+            )
+        else:
+            result = check_faithfulness(pid, slow_code, llm_code)
     except Exception:
         return (Verdict.UNKNOWN, _INSTR.last_path)
     return (result.verdict, _INSTR.last_path)

scripts/run_faith_rescore.sh

@@ -20,11 +20,21 @@ for m in $OLD; do for s in generic pattern-aware taxonomy-guided; do
   $PY scripts/rescore_faithfulness.py "$f" --equivalence reuse
 done; done
 
-echo "=== [2/4] recompute faithfulness: 18 venv-scored cells (recompute 9-config equiv, slow) ==="
+# [2/4] The 6 venv-scored models had their equivalence axis degraded to
+# correctness-based single-config (faithfulness_equivalent == correct on every
+# row). Full 9-config recompute would re-run differential equivalence on ~22k
+# rows; ~39% are timeout-prone broken candidates costing up to 30s x 9 configs
+# each -> weeks single-threaded on a laptop (and the compile+spin load OOM-crashed
+# the machine last attempt). We therefore reuse the stored (correctness-based)
+# equivalence -- identical to what recompute falls back to for any non-overfit
+# candidate. The composition-aware COMP structural fix (the actual point of this
+# rescore) is recomputed in BOTH modes. Bounded limitation: no 9-config overfit
+# detection on the equivalence axis for these 6 models. Sequential, no compilation.
+echo "=== [2/4] recompute faithfulness: 18 venv-scored cells (reuse equiv, fast) ==="
 for m in $NEW; do for s in generic pattern-aware taxonomy-guided; do
   f="results/pareto/${m}_${s}_scored.csv"
-  echo "  recompute  ${m}_${s}"
-  $PY scripts/rescore_faithfulness.py "$f" --equivalence recompute
+  echo "  reuse  ${m}_${s}"
+  $PY scripts/rescore_faithfulness.py "$f" --equivalence reuse
 done; done
 
 echo "=== [3/4] rebuild combined + lean CSVs ==="