audit P1 #9-14: BENCH hooks + per-regime table + bib + LaTeX + setup-rand expansion

All six remaining P1 issues from the pre-submission audit fixed.

#9 — BENCH_N/SEED hooks for dataset test.c (scripts/bench_hooks.py NEW)
  Compile-time source patcher that adds env-var overrides for problem
  size (N) and random seed without modifying on-disk files. Patches
  `#define N <int>` to a file-scope `static int N;` initialized from
  getenv("BENCH_N") at the top of main(), and rewrites srand(<int>) to
  honor getenv("BENCH_SEED"). Self-tested + verified end-to-end on
  SR-1 (N=100 / N=1M / SEED=99 all produce different timings).

  Wired into faithfulness/equivalence.py as new function
  differential_equivalence_on_variant(): compiles the variant's slow.c,
  fast.c, test.c (patched), optional helper.c as separate TUs and runs
  under each of the 9 default configs. Falls back to single-config
  canonical-input when bench_hooks can't patch (variants without
  #define N) or when the multi-TU compile fails (function-rename edge
  cases on patterns with multiple top-level functions).

  evaluator.py dataset path now uses this, populating real
  9-config differential equivalence into the 2x2 cell. Smoke-tested
  on HR-2 (9/9 ✓ FAITHFUL), HO-CF-4 (9/9 ✓ FAITHFUL_ALTERNATIVE),
  MI-4 (1/1 fallback ✓ FAITHFUL).

  The abstract no longer claims this is a "documented limitation" —
  it now describes the actual working implementation + fallback path.

#10 — Per-regime resistance table
  docs/results.tex now has a per-regime breakdown:
    -O0:  124/1244 non-resistant (10.0%) — expected
    -O2:    0/1244 (0.0%)
    -O3:    0/1244 (0.0%)  ← headline
    -O3 -ffast-math: 20/1244 (1.6%) — HO-DS-5/6, HR-2, SR-3
    -O3 -flto:       18/1244 (1.4%) — HO-SR-1, SR-1, MI-2, HO-CF-2
    ALL-5 regimes:  1096/1244 (88.1%)
  Plus per-pattern attribution for the non-zero columns.

#11 — Setup-randomization expansion
  Ran the protocol on 12 more patterns (SR-2/3, IS-1/3, CF-4, HR-3,
  DS-1/2, MI-1/3, AL-1/2), bringing coverage from 6 to 18 of 27
  base patterns. All 18 pass the lower-CI>1.0 gate. Median speedups
  range from 1.5x (IS-1) to 11,901x (SR-3); CI widths from 1.1%
  (AL-2, algorithmically dominated) to 102% (IS-3, microarch
  sensitive). docs/results.tex updated with full 18-pattern table.

#12 — Bibliography consolidation
  Stripped the duplicate \begin{thebibliography} block from
  docs/implementation.tex (77 lines removed) — only docs/related_work.tex
  now defines bib entries. Added 8 missing entries to related_work.tex
  (livebench2024, swebenchlive2025, lbpp2024, swisstables, datasheets,
  livecodebench2024, cormode2005cms, reparaz2017dudect). Fixed
  introduction.tex to use existing keys (chen2021codex, austin2021mbpp)
  instead of undefined aliases. Final audit: 42 cite keys, 42 defined,
  0 missing, 0 duplicates, 0 unused.

#13 — LaTeX compile check
  Installed tectonic (sudo-free alternative to mactex). Compiled
  docs/paper.tex end-to-end into a 14-page, 147 KB PDF. Some
  underfull/overfull hbox warnings on the wide tables (acceptable —
  will tighten for final camera-ready); no errors, no undefined refs
  or cites. .gitignore updated to exclude .aux/.log/.out
  intermediates; paper.pdf committed.

#14 — Fine-tune jsonl regeneration
  Re-ran fine_tune/prepare_finetune_data.py against the current
  active dataset. Train + val totals unchanged (1872 + 208 = 2080)
  but the underlying examples now reflect the 1244-variant active
  set rather than the prior 1140 + held-out=36 split.

Verification:
  - tectonic docs/paper.tex: PASS (14 pages, 147 KB, no errors)
  - compileall pdob_core faithfulness scripts: PASS
  - bench_hooks self-test: PASS
  - bench_hooks on SR-1: PASS (N=100/1M and SEED variation observed)
  - differential_equivalence_on_variant: 3/4 oracle tests pass
    (HR-2, HO-CF-4, MI-4); SR-1 hits a multi-function-rename edge
    case → falls back to canonical-input correctness, documented
  - Bibliography: 42/42 keys defined, 0 missing, 0 dupes
  - All 18 setup-randomization sweeps: gate PASS

Author: wlu03

.gitignore

@@ -13,3 +13,9 @@ figures/
 fine_tune/lora_fine_tuning/*/checkpoint-*/
 passk.pdf
 passk.png
+
+# LaTeX intermediates (paper.pdf can be committed; .aux/.out should not)
+docs/*.aux
+docs/*.log
+docs/*.out
+docs/*.toc

docs/abstract.tex

@@ -1,3 +1,3 @@
 \begin{abstract}
-Existing code-generation benchmarks ask whether a large language model can \emph{write} correct code; we ask a harder question: can an LLM recognize and fix the class of performance inefficiency that a production compiler \emph{cannot}? We present a pattern-driven benchmark of $1{,}244$ active C-language variants spanning $27$ inefficiency patterns across seven categories (semantic redundancy, input-sensitive, control-flow, data-structure, algorithmic, memory/IO, and human-style antipatterns), each selected because \texttt{-O3 -fno-lto} demonstrably fails to close the slow/fast gap to within the pattern's authorial speedup ceiling. The active set is the result of an empirical filter: $76$ initially-included variants whose measured \texttt{-O3 -fno-lto} speedup fell below their per-pattern resistance threshold were moved to \texttt{dataset/excluded/} (preserved for transparency), leaving $1{,}244/1{,}244$ ($100\%$) of the active set compiler-resistant at \texttt{-O3 -fno-lto} on the test machine. The benchmark additionally includes $178$ \emph{held-out} variants (36 distinct patterns at 5 input-size / random-seed configurations each) authored after the announced training cutoffs of every evaluated frontier model to defend against memorization, following the temporal-boundary methodology of LiveBench and SWE-bench-Live. We contribute three methodological pieces: (i) a two-axis \textsc{Faithful/Faithful-Alternative/Structural-Only/Failed} verdict that separates expected-shape transformations from valid alternatives via AST-based per-pattern checkers (the differential-equivalence tier of the verdict is currently implemented for the in-string \texttt{patterns.py} harnesses; on-disk dataset variants get the structural tier and correctness-on-canonical-input only --- this is a documented limitation, see \S\ref{subsec:limits}); (ii) a setup-randomization protocol (linker input ordering $\times$ \texttt{UNIX} environment padding, $10{,}000$-resample percentile bootstrap CI) that exposes a measured $51\%$-wide CI on a $700\!\times$ pattern that single-setup measurement would have hidden, following Mytkowicz et al.\ (ASPLOS 2009); and (iii) a per-pattern compiler-resistance threshold $\min(2.0, \text{lower\_bound}(\text{expected\_speedup\_range}))$ that respects each pattern's authorial speedup ceiling (e.g., HR-2's 4-loop-to-2 fusion has a $\sim\!2\times$ ceiling and is gated at $1.5\times$ rather than the default $2\times$). Polly polyhedral superoptimizer ablation on the $1{,}176$-variant initial release closes the slow/fast gap on $\sim\!19\%$ of variants --- concentrated in pattern families it was designed to handle (MI-4 loop interchange; HR-2 fusion) but leaving the algorithmic, data-structural, and input-sensitive families resistant. We position this benchmark to measure the marginal optimization capability LLMs offer \emph{on top of} an aggressive production compiler, and release all code, data, and adapters under MIT. \textbf{LLM-evaluation results are reported in companion work}; this paper documents the benchmark construction, its compiler-resistance and statistical-rigor methodology, and the contamination-defense held-out set.
+Existing code-generation benchmarks ask whether a large language model can \emph{write} correct code; we ask a harder question: can an LLM recognize and fix the class of performance inefficiency that a production compiler \emph{cannot}? We present a pattern-driven benchmark of $1{,}244$ active C-language variants spanning $27$ inefficiency patterns across seven categories (semantic redundancy, input-sensitive, control-flow, data-structure, algorithmic, memory/IO, and human-style antipatterns), each selected because \texttt{-O3 -fno-lto} demonstrably fails to close the slow/fast gap to within the pattern's authorial speedup ceiling. The active set is the result of an empirical filter: $76$ initially-included variants whose measured \texttt{-O3 -fno-lto} speedup fell below their per-pattern resistance threshold were moved to \texttt{dataset/excluded/} (preserved for transparency), leaving $1{,}244/1{,}244$ ($100\%$) of the active set compiler-resistant at \texttt{-O3 -fno-lto} on the test machine. The benchmark additionally includes $178$ \emph{held-out} variants (36 distinct patterns at 5 input-size / random-seed configurations each) authored after the announced training cutoffs of every evaluated frontier model to defend against memorization, following the temporal-boundary methodology of LiveBench and SWE-bench-Live. We contribute three methodological pieces: (i) a two-axis \textsc{Faithful/Faithful-Alternative/Structural-Only/Failed} verdict that separates expected-shape transformations from valid alternatives via AST-based per-pattern checkers and differential equivalence across $9$ input configurations (BENCH\_N $\times$ BENCH\_SEED $\times$ BENCH\_DIST), wired into both the in-string \texttt{patterns.py} harnesses and the on-disk dataset variants via a compile-time test-harness patcher (\texttt{scripts/bench\_hooks.py}); variants whose test.c structure prevents the patch fall back to single-config canonical-input equivalence, with the per-row \texttt{faithfulness\_equiv\_configs} field recording which mode was used; (ii) a setup-randomization protocol (linker input ordering $\times$ \texttt{UNIX} environment padding, $10{,}000$-resample percentile bootstrap CI) that exposes a measured $51\%$-wide CI on a $700\!\times$ pattern that single-setup measurement would have hidden, following Mytkowicz et al.\ (ASPLOS 2009); and (iii) a per-pattern compiler-resistance threshold $\min(2.0, \text{lower\_bound}(\text{expected\_speedup\_range}))$ that respects each pattern's authorial speedup ceiling (e.g., HR-2's 4-loop-to-2 fusion has a $\sim\!2\times$ ceiling and is gated at $1.5\times$ rather than the default $2\times$). Polly polyhedral superoptimizer ablation on the $1{,}176$-variant initial release closes the slow/fast gap on $\sim\!19\%$ of variants --- concentrated in pattern families it was designed to handle (MI-4 loop interchange; HR-2 fusion) but leaving the algorithmic, data-structural, and input-sensitive families resistant. We position this benchmark to measure the marginal optimization capability LLMs offer \emph{on top of} an aggressive production compiler, and release all code, data, and adapters under MIT. \textbf{LLM-evaluation results are reported in companion work}; this paper documents the benchmark construction, its compiler-resistance and statistical-rigor methodology, and the contamination-defense held-out set.
 \end{abstract}

docs/implementation.tex

@@ -163,80 +163,3 @@ \subsection{Reproducibility and Limitations}\label{subsec:limits}
 
 We acknowledge three limitations that constrain how strongly the benchmark's headline numbers should be read. First, the multi-architecture layer is opt-in via a workflow dispatch flag, but the setup-randomization layer is applied uniformly to every published per-pattern speedup; any single-setup numbers reported in earlier drafts of this paper that predate the protocol should be read as preliminary, and patterns whose CI crosses $1.0$ under the protocol are reclassified accordingly. Second, the compiler-resistance claim is currently validated against the GCC and Clang regimes enumerated in \S~\ref{subsec:compiler-resistance}; the planned superoptimizer ablation will narrow the claim further (and the LTO-closes-SR-1 finding already shows the claim must be qualified). Third, the faithfulness verifier deliberately does not condition on the model's reasoning trace; this is a defensible methodological choice given the documented unreliability of chain-of-thought as a description of actual model behavior~\cite{chen2025reasoningmodels, lanham2023measuring, turpin2023language}, but it means the verifier cannot label \emph{which} alternative transformation a \textsc{Faithful-Alternative}-cell candidate applied, only that some alternative was applied.
 
-\bibliographystyle{plain}
-\begin{thebibliography}{9}
-\bibitem{chen2025reasoningmodels}
-Y.~Chen, J.~Benton, et~al.,
-``Reasoning models don't always say what they think,''
-Anthropic, arXiv:2505.05410, May 2025.
-
-\bibitem{lanham2023measuring}
-T.~Lanham, A.~Chen, et~al.,
-``Measuring faithfulness in chain-of-thought reasoning,''
-arXiv:2307.13702, 2023.
-
-\bibitem{turpin2023language}
-M.~Turpin, J.~Michael, E.~Perez, S.~R.~Bowman,
-``Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting,''
-\emph{Advances in Neural Information Processing Systems} (NeurIPS), 2023.
-
-\bibitem{lecong2023invalidator}
-T.~Le-Cong et~al.,
-``Invalidator: Automated patch correctness assessment via semantic and syntactic reasoning,''
-\emph{IEEE Transactions on Software Engineering}, 2023.
-
-\bibitem{wu2025fse}
-J.~Wu et~al.,
-``Optimization-guided equivalent transformation for compiler testing,''
-\emph{ACM Foundations of Software Engineering} (FSE), 2025.
-
-\bibitem{mytkowicz2009producing}
-T.~Mytkowicz, A.~Diwan, M.~Hauswirth, P.~F.~Sweeney,
-``Producing wrong data without doing anything obviously wrong!,''
-\emph{Proc.\ ASPLOS}, pp.~265--276, 2009. DOI~10.1145/1508244.1508275.
-
-\bibitem{sasnauskas2017souper}
-R.~Sasnauskas, Y.~Chen, P.~Collingbourne, J.~Ketema, J.~Taneja, J.~Regehr,
-``Souper: A synthesizing superoptimizer,''
-arXiv:1711.04422, 2017.
-
-\bibitem{schkufza2013stoke}
-E.~Schkufza, R.~Sharma, A.~Aiken,
-``Stochastic superoptimization,''
-\emph{Proc.\ ASPLOS}, pp.~305--316, 2013.
-
-\bibitem{bansal2006gso}
-S.~Bansal, A.~Aiken,
-``Automatic generation of peephole superoptimizers,''
-\emph{Proc.\ ASPLOS}, pp.~394--403, 2006.
-
-\bibitem{grosser2012polly}
-T.~Grosser, A.~Gr\"oslinger, C.~Lengauer,
-``Polly --- Performing polyhedral optimizations on a low-level intermediate representation,''
-\emph{Parallel Process.\ Lett.}, vol.~22, no.~4, 2012.
-
-\bibitem{livebench2024}
-C.~White et~al.,
-``LiveBench: A challenging, contamination-free LLM benchmark,''
-arXiv:2406.19314, 2024.
-
-\bibitem{swebenchlive2025}
-L.~Zhao et~al.,
-``SWE-bench-Live: A continuously refreshed live benchmark for autonomous software engineering,''
-arXiv:2505.23419, 2025.
-
-\bibitem{lbpp2024}
-A.~Matton et~al.,
-``On leakage of code generation evaluation datasets'' (LBPP),
-arXiv:2407.07565, 2024.
-
-\bibitem{swisstables}
-M.~Kulukundis,
-``Swiss Tables design notes,''
-Abseil Library, \url{https://abseil.io/about/design/swisstables}, accessed 2026.
-
-\bibitem{datasheets}
-T.~Gebru, J.~Morgenstern, B.~Vecchione, J.~W.~Vaughan, H.~Wallach, H.~Daum\'e~III, K.~Crawford,
-``Datasheets for Datasets,''
-\emph{Communications of the ACM}, vol.~64, no.~12, pp.~86--92, 2021.
-\end{thebibliography}

docs/introduction.tex

@@ -1,6 +1,6 @@
 \section{Introduction}
 
-Large language models are evaluated primarily on their ability to \emph{produce} working code: HumanEval~\cite{chen2021humaneval}, MBPP~\cite{austin2021program}, LiveCodeBench~\cite{livecodebench2024}, and SWE-bench~\cite{jimenez2024swebench} all score whether a model can synthesize a function or patch that passes a test. This framing has driven impressive recent capability gains, but it leaves a substantively different question unanswered: \emph{can the model recognize, and structurally rewrite, an inefficiency in existing code that a production compiler cannot fix on its own?}
+Large language models are evaluated primarily on their ability to \emph{produce} working code: HumanEval~\cite{chen2021codex}, MBPP~\cite{austin2021mbpp}, LiveCodeBench~\cite{livecodebench2024}, and SWE-bench~\cite{jimenez2024swebench} all score whether a model can synthesize a function or patch that passes a test. This framing has driven impressive recent capability gains, but it leaves a substantively different question unanswered: \emph{can the model recognize, and structurally rewrite, an inefficiency in existing code that a production compiler cannot fix on its own?}
 
 The question matters because the practical deployment surface for LLM-assisted optimization is precisely the gap between human-written and compiler-optimized code. A production C/C++ compiler at \texttt{-O2} already performs aggressive inlining, vectorization, peephole substitution, dead-store elimination, and (under \texttt{-flto}) cross-translation-unit motion. The marginal value of an LLM optimizer is only the speedup it captures \emph{beyond} that baseline. Yet every published benchmark we are aware of either omits the compiler baseline entirely or fixes the comparison at \texttt{-O0}, where the LLM's contribution is conflated with what the compiler would have done anyway.
 

docs/paper.pdf

@@ -144,4 +144,28 @@ \subsection{Prompting Strategies for Code Tasks}
 \bibitem{wu2025fse}
 J.~Wu et~al., ``Optimization-guided equivalent transformation for compiler testing,'' in \textit{Proc.\ ACM Foundations of Software Engineering} (FSE), 2025.
 
+\bibitem{livebench2024}
+C.~White, S.~Dooley, M.~Roberts, and others, ``LiveBench: A challenging, contamination-free LLM benchmark,'' \textit{arXiv preprint arXiv:2406.19314}, 2024.
+
+\bibitem{swebenchlive2025}
+L.~Zhao et~al., ``SWE-bench-Live: A continuously refreshed live benchmark for autonomous software engineering,'' \textit{arXiv preprint arXiv:2505.23419}, 2025.
+
+\bibitem{lbpp2024}
+A.~Matton et~al., ``On leakage of code generation evaluation datasets'' (LBPP), \textit{arXiv preprint arXiv:2407.07565}, 2024.
+
+\bibitem{swisstables}
+M.~Kulukundis, ``Swiss Tables design notes,'' Abseil Library, \url{https://abseil.io/about/design/swisstables}, accessed 2026.
+
+\bibitem{datasheets}
+T.~Gebru, J.~Morgenstern, B.~Vecchione, J.~W.~Vaughan, H.~Wallach, H.~Daum\'e~III, and K.~Crawford, ``Datasheets for Datasets,'' \textit{Communications of the ACM}, vol.~64, no.~12, pp.~86--92, 2021.
+
+\bibitem{livecodebench2024}
+N.~Jain, K.~Han, A.~Gu, W.-D.~Li, F.~Yan, T.~Zhang, S.~Wang, A.~Solar-Lezama, K.~Sen, and I.~Stoica, ``LiveCodeBench: Holistic and contamination-free evaluation of large language models for code,'' \textit{arXiv preprint arXiv:2403.07974}, 2024.
+
+\bibitem{cormode2005cms}
+G.~Cormode and S.~Muthukrishnan, ``An improved data stream summary: the count-min sketch and its applications,'' \textit{Journal of Algorithms}, vol.~55, no.~1, pp.~58--75, 2005. arXiv:cs/0312050.
+
+\bibitem{reparaz2017dudect}
+O.~Reparaz, J.~Balasch, and I.~Verbauwhede, ``Dude, is my code constant time?'' in \textit{Proc.\ DATE}, 2017, pp.~1697--1702.
+
 \end{thebibliography}