Merge pull request #13 from wlu03/fix/coder32b-fence-extraction

Fix/coder32b fence extraction

Author: wlu03

.gitignore

@@ -25,3 +25,8 @@ results/pareto/*.submit.log
 results/pareto/*.score.log
 results/pareto/*.errors.csv
 results/pareto/*.failed_variants.txt
+
+# Local analysis venv + transient re-score artifacts / scored-CSV backups
+.venv-analysis/
+*.prebugfix
+results/pareto/rescore/

README.md

@@ -26,20 +26,54 @@ The patterns are organized by *why* the compiler fails to fix them:
 
 ## LLM Evaluation Status
 
-**No scored model-results CSV is currently committed.** The benchmark infrastructure (prompt strategies, evaluator, faithfulness layer, Modal fan-out, score-from-CSV scoring) is complete and smoke-tested, but the per-(model, strategy) sweep across the Pareto-frontier model shortlist has not yet been run end-to-end. Earlier drafts of this README presented findings from a preliminary single-machine Qwen2.5-Coder-7B-Ollama run (e.g. "taxonomy 16/16, pattern-aware 12.8× on IS-4, SR-2 backfire effect") whose raw outputs were not preserved as a committed artifact. Those numbers should be treated as *exploratory pre-registration hypotheses* rather than results, and have been removed from this README to avoid presenting them as supported claims.
-
-**Once a scored CSV is committed**, this section will report:
-- Per-(model, strategy) correctness, speedup-vs-slow, speedup-vs-hand-optimized
-- Faithfulness 2×2 (Faithful / Faithful-Alternative / Structural-Only / Failed) per pattern
-- Cross-pattern transfer Spearman correlations across models
-- Fine-tune-vs-baseline paired-Wilcoxon p-value on held-out
-
-**Hypotheses to test (from the preliminary Qwen-7B-Ollama exploration, NOT verified):**
-1. Taxonomy-guided prompting achieves highest *correctness*; pattern-aware sometimes degrades it on patterns where the category label triggers a wrong but plausible transformation (the "backfire effect").
-2. Input-sensitive patterns (IS-1/2/3/4/5) are hardest; the spread between best and worst prompt strategy is largest on this category.
-3. Algorithmic and semantic-redundancy patterns are easiest; the model's marginal contribution over `-O3` is dominated by these two.
-
-These will be confirmed or refuted by the scored sweep.
+**The open-weight Modal sweep is complete: 15 models × 3 prompting strategies × all 1,244 variants = 55,980 scored attempts.** Inference runs on Modal (vLLM, one model at a time with per-model decode configs in `modal_app/inference.py`); scoring — compile + correctness + speedup + two-axis faithfulness — runs locally from the saved completions via `scripts/score_completions.py`, which decouples expensive GPU inference from cheap CPU scoring so the extractor/scorer can be iterated without re-running the models. Per-cell `results/pareto/<model>_<strategy>_scored.csv` files concatenate into `results/pareto_combined_scored.csv`; the 2×2 and scaling artifacts are regenerated by `faithfulness/report_2x2.py` and `scripts/scaling_analysis.py`. Per `.gitignore`, the `*_scored.csv` files are the committable artifacts (transient Modal logs are ignored).
+
+> Scope: this sweep covers **open-weight** models only (Qwen2.5 / Qwen3 / QwQ, DeepSeek-R1-distill & Coder-V2-Lite, Codestral, OpenCoder, Yi-Coder). The frontier API models in *Supported Models* below are wired into the harness but were not part of this run.
+
+### Results — geomean speedup and pass@1 per strategy
+
+`geomean×` is the geometric mean of `speedup_vs_slow` over correct **and** reliable variants; `pass@1` is that same mask's rate. Triplets are **generic / pattern-aware / taxonomy-guided**. Reproduce with `python3 scripts/rank_models.py results/pareto_combined_scored.csv --sort size`.
+
+| Model | B | type | pass@1 (g/p/t) | geomean× (g/p/t) |
+|---|---|---|---|---|
+| deepseek-r1-distill-qwen-1.5b | 1.5 | reasoning | 4 / 2 / 2 | 1.6 / 2.4 / 1.1 |
+| qwen2.5-coder-1.5b | 1.5 | coder | 55 / 51 / 57 | 1.3 / 1.3 / 1.5 |
+| deepseek-r1-distill-qwen-7b | 7 | reasoning | 21 / 20 / 24 | 15.5 / 9.6 / 10.1 |
+| qwen2.5-coder-7b | 7 | coder | 72 / 70 / 75 | 2.1 / 3.0 / 3.4 |
+| opencoder-8b | 8 | coder | 75 / 71 / 58 | 5.2 / 5.0 / 7.7 |
+| yi-coder-9b | 9 | coder | 51 / 54 / 61 | 7.6 / 11.2 / 9.7 |
+| qwen2.5-coder-14b | 14 | coder | 61 / 68 / 73 | 2.7 / 7.8 / 6.7 |
+| deepseek-coder-v2-lite | 16 | coder | 54 / 71 / 69 | 3.4 / 4.0 / 4.9 |
+| codestral-22b | 22 | coder | 50 / 45 / 52 | 9.9 / 14.6 / 9.5 |
+| deepseek-r1-distill-qwen-32b | 32 | reasoning | 52 / 61 / 58 | 11.7 / 15.1 / 14.0 |
+| qwen2.5-coder-32b | 32 | coder | 59 / 66 / 69 | 8.2 / 10.4 / 10.0 |
+| qwq-32b | 32.5 | reasoning | 51 / 44 / 46 | 10.2 / 11.8 / 10.1 |
+| qwen3-32b | 32.8 | reasoning | 64 / 52 / 64 | 13.3 / 11.0 / 12.6 |
+| deepseek-r1-distill-llama-70b | 70 | reasoning | 46 / 42 / 46 | 13.4 / 12.3 / 12.2 |
+| qwen2.5-72b | 72 | general | 57 / 54 / 62 | 6.8 / 14.1 / 10.3 |
+
+**Faithful × fast 2×2** (overall, n=55,980; "fast" = `speedup_vs_slow` > 1.5; full per-model/per-pattern breakdown in `results/aggregate_2x2/report.txt`):
+
+| | Faithful | Unfaithful | Row |
+|---|---|---|---|
+| **Fast** | 18.7% | 11.5% | 30.2% |
+| **Slow** | 13.0% | 56.7% | 69.8% |
+| **Col** | 31.8% | 68.2% | 100% |
+
+Per-strategy faithful rates are close: generic 30.0%, pattern-aware 33.8%, taxonomy-guided 31.5%.
+
+**Faithfulness-scoring caveats.** Two structural factors shape this aggregate, and the headline rate is sensitive to both:
+- **COMP composition (≈54% of rows).** COMP variants are scored against their constituent-pattern list (`composition` from `metadata.json`); the COMP checker *requires* it, and without it falls back to a generic regex battery that massively over-reports `FAITHFUL`. Earlier runs omitted it and reported an inflated ~45.6% overall (COMP alone read 58% faithful); both `faithfulness/report_2x2.py` and `scripts/rescore_faithfulness.py` now thread `composition`, which drops COMP to 33% faithful and the overall rate to 31.8%.
+- **Held-out patterns (`HO-*`, ≈14% of rows).** These post-cutoff patterns have no dedicated AST checker and fall through to a coarse structural fallback, so they essentially cannot earn a `FAITHFUL` verdict and weigh toward the unfaithful column. Authoring per-pattern held-out checkers is the remaining faithfulness-coverage gap.
+
+### Findings from the sweep
+
+1. **A label helps speed but often hurts correctness.** A labeled strategy (pattern-aware *or* taxonomy-guided) beats generic on geomean speedup for **12 of 15** models — the 3 exceptions are all reasoning models, where the extra context slightly lowers speedup. Yet pattern-aware *reduces* pass@1 vs generic on **9 of 15** models. The category label pushes models toward a faster transformation at some cost to correctness.
+2. **The pattern-aware backfire effect reproduces — on correctness.** The clearest case is Qwen3-32B: pass@1 falls 64% → 52% (−12pp) from generic to pattern-aware, then recovers to 64% under taxonomy-guided. Taxonomy-guided generally recovers correctness that pattern-aware sheds (coder-7b 70→75, coder-14b 68→73, coder-32b 66→69).
+3. **Reasoning ≠ uniformly better.** Three ~32B reasoning recipes — DeepSeek-R1-distill (distilled CoT), QwQ (RL), Qwen3 (thinking) — span 44–64% pass@1 at the same size, with Qwen3-32B strongest on correctness and the R1-distill strongest on peak speedup.
+4. **Peak speedup is a fragile ranker.** DeepSeek-R1-Distill-Qwen-7B posts the single highest geomean (15.5×) but on only **21%** pass@1 — that mean is taken over a thin correct set. Rank by geomean *among models with healthy pass@1*, not by raw peak.
+
+Still to analyze from the committed scored CSVs: per-category difficulty (the IS-hardest / AL-SR-easiest hypotheses), cross-pattern transfer Spearman correlations, and the fine-tune-vs-baseline paired-Wilcoxon test on the held-out set.
 
 ---
 

faithfulness/report_2x2.py

@@ -60,6 +60,29 @@
 except Exception:
     _PATTERNS_BY_ID = {}
 
+# COMP variants carry their constituent pattern list in dataset metadata. The
+# COMP checker NEEDS that list: without it, it falls back to a generic regex
+# battery that massively over-reports `FAITHFUL` (COMP is ~54% of the dataset,
+# so this dominates the aggregate). Build variant_id -> composition lazily so
+# non-COMP runs pay nothing, and resolve the dataset relative to the repo root
+# so the lookup works regardless of the caller's cwd.
+_COMP_COMPOSITION: Optional[dict] = None
+
+
+def _composition_for(variant_id: str):
+    global _COMP_COMPOSITION
+    if _COMP_COMPOSITION is None:
+        _COMP_COMPOSITION = {}
+        try:
+            from pdob_core.dataset_evaluator import discover_variants  # type: ignore
+            for v in discover_variants(str(_HERE.parent / "dataset")):
+                comp = v.metadata.get("composition") if getattr(v, "metadata", None) else None
+                if comp:
+                    _COMP_COMPOSITION[v.variant_id] = comp
+        except Exception:
+            pass
+    return _COMP_COMPOSITION.get(variant_id)
+
 
 # --------------------------------------------------------------------------
 # Instrumentation: wrap PatternChecker.check + _parse so we can report
@@ -229,7 +252,16 @@ def _compute_faithfulness_for_row(
 
     _INSTR.reset()
     try:
-        result = check_faithfulness(pid, slow_code, llm_code)
+        if pid == "COMP":
+            # Thread the composition so COMPChecker scores the actual
+            # constituent transforms instead of the over-reporting regex
+            # fallback. Mirrors scripts/rescore_faithfulness.py.
+            result = check_faithfulness(
+                pid, slow_code, llm_code,
+                composition=_composition_for(row.get("variant_id", "")),
+            )
+        else:
+            result = check_faithfulness(pid, slow_code, llm_code)
     except Exception:
         return (Verdict.UNKNOWN, _INSTR.last_path)
     return (result.verdict, _INSTR.last_path)

modal_app/inference.py

@@ -97,27 +97,89 @@
         "max_model_len": 8192,
         "reasoning": False,
     },
-    "llama-3.3-70b": {
-        "hf_id": "meta-llama/Llama-3.3-70B-Instruct",
-        # 70B params in bf16 = ~140 GB; doesn't fit on single H100 (80 GB).
-        # H200 has 141 GB — fits with headroom for KV cache.
-        "gpu":   "H200",
+    # DeepSeek-R1-Distill-Llama-70B — non-gated (MIT) reasoning distill; the
+    # 70B member of the DeepSeek-R1-Distill series (base: Llama-3.3-70B). Used
+    # in place of the gated meta-llama/Llama-3.3-70B-Instruct so no HF token is
+    # needed. 70B bf16 checkpoint ≈ 131 GiB — a single H200 (140 GiB) loads the
+    # weights but has no room left for the KV cache ("Available KV cache: -16
+    # GiB"), so shard across 2× H200 with tensor parallelism.
+    "deepseek-r1-distill-llama-70b": {
+        "hf_id": "deepseek-ai/DeepSeek-R1-Distill-Llama-70B",
+        "gpu":   "H200:2",
+        "tensor_parallel_size": 2,
         "max_model_len": 8192,
-        "reasoning": False,
-        "needs_hf_token": True,
+        "reasoning": True,
+        "reasoning_parser": "deepseek_r1",
     },
-    # Non-gated 72B-class general-purpose model. 72B in bf16 = ~144 GB;
-    # H200's 141 GB is just tight enough at gpu_memory_utilization=0.85
-    # (~120 GB allocation, ~24 GB headroom for KV cache).
+    # Non-gated 72B-class general-purpose model. 72B bf16 checkpoint ≈ 135 GiB —
+    # does NOT fit one H200 (140 GiB) once memory profiling + KV cache are added
+    # (OOMs with ~131 MiB free). Shard across 2× H200 with tensor parallelism.
     "qwen2.5-72b": {
         "hf_id": "Qwen/Qwen2.5-72B-Instruct",
-        "gpu":   "H200",
+        "gpu":   "H200:2",
+        "tensor_parallel_size": 2,
         "max_model_len": 8192,
         "reasoning": False,
     },
     # Codestral may also be gated depending on HF account access policy.
     # If you see "Cannot access gated repo" errors, add "needs_hf_token": True
     # to its entry too.
+
+    # ── Added anchors (non-gated) ─────────────────────────────────────────
+    # Second reasoning family + newer generation, both 32B. Both emit <think>,
+    # so the deepseek_r1 reasoning parser applies.
+    #
+    # These run on a SINGLE H200 (not A100-80GB) at 16384 ctx, unlike the
+    # R1-distill-32b. A limit-4 smoke test showed QwQ on A100-80GB@8192
+    # exhausted its token budget (~5k tokens) mid-reasoning and never emitted
+    # code — A100-80GB only leaves ~18.9k tokens of KV pool for a 32B model,
+    # so it can't hold a longer context. QwQ's reasoning is far more verbose
+    # than the R1 distills, so it needs both more tokens (16384) and the KV
+    # headroom of an H200 (~58 GiB free after weights) to batch at that length.
+    "qwq-32b": {
+        # RL-trained reasoning model (NOT an R1 distill). Apache-2.0.
+        # max_tokens 12288 (vs 4096 default): a smoke test showed QwQ's <think>
+        # alone exceeds 4096 tokens, truncating before any code. 12288 + a
+        # ~2-3k prompt stays within the 16384 max_model_len.
+        "hf_id": "Qwen/QwQ-32B",
+        "gpu":   "H200",
+        "max_model_len": 16384,
+        "max_tokens": 12288,
+        "reasoning": True,
+        "reasoning_parser": "deepseek_r1",
+    },
+    "qwen3-32b": {
+        # Newer generation; thinking mode on by default. Apache-2.0.
+        "hf_id": "Qwen/Qwen3-32B",
+        "gpu":   "H200",
+        "max_model_len": 16384,
+        "max_tokens": 12288,
+        "reasoning": True,
+        "reasoning_parser": "deepseek_r1",
+    },
+    # Second coder family — tests whether the coder edge is Qwen-specific.
+    # Non-reasoning; given comfortable GPU headroom to avoid vLLM-init OOM.
+    "opencoder-8b": {
+        # Open/reproducible code LLM (infly). ~7B-tier peer to coder-7b.
+        "hf_id": "infly/OpenCoder-8B-Instruct",
+        "gpu":   "L40S",
+        "max_model_len": 8192,
+        "reasoning": False,
+    },
+    "deepseek-coder-v2-lite": {
+        # 16B MoE (2.4B active) code model — adds a dense-vs-MoE axis too.
+        "hf_id": "deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct",
+        "gpu":   "A100-80GB",
+        "max_model_len": 8192,
+        "reasoning": False,
+    },
+    "yi-coder-9b": {
+        # 9B code model, Apache-2.0. Optional third coder family.
+        "hf_id": "01-ai/Yi-Coder-9B-Chat",
+        "gpu":   "L40S",
+        "max_model_len": 8192,
+        "reasoning": False,
+    },
 }
 
 # --- Modal app + image ------------------------------------------------------
@@ -187,6 +249,9 @@ def load(self):
             enforce_eager=True,
             # Conservative GPU-mem fraction so KV cache + workspace fits.
             gpu_memory_utilization=0.85,
+            # Shard large models across multiple GPUs (1 = single-GPU default).
+            # 70B+ bf16 weights exceed a single H200, so those cfgs set this to 2.
+            tensor_parallel_size=cfg.get("tensor_parallel_size", 1),
         )
         # Reasoning models: enable vLLM's reasoning-content parser so the
         # <think> trace is exposed separately from the answer text. The
@@ -233,8 +298,13 @@ def generate_batch(self, prompts: list[str],
         not separated by vLLM (fallback for older vLLM versions).
         """
         from vllm import SamplingParams
+        # Per-model override: verbose reasoning models (QwQ, Qwen3-thinking)
+        # need far more than the 4096 default or they exhaust the budget mid-
+        # <think> and never emit code. Set "max_tokens" in their MODELS entry.
+        effective_max_tokens = MODELS[self.model_key].get("max_tokens",
+                                                           max_tokens)
         params = SamplingParams(
-            temperature=0.0, top_p=1.0, max_tokens=max_tokens,
+            temperature=0.0, top_p=1.0, max_tokens=effective_max_tokens,
         )
         # Apply each model's chat template so instruct-tuned models get
         # the role/turn markers they were fine-tuned to receive.