Rust↔Python drift: perf_interp site-transfer disagrees on off-grid GEMM (fp8_block n=1280 k=5120, h200_sxm/vllm/0.19.0)

[tianhaox]

Found by the PR #1454 forced-rust full support-matrix scan (6460 entries): 2 DRIFT entries, 0 REGRESSION.

Isolated root cause (Qwen3-32B-FP8 | h200_sxm | vllm 0.19.0 | agg)

The compiled engine's GEMM query returns +3.06–3.48% vs Python for exactly one shape class:

query (fp8_block)	python	rust	rel
m=1, n=1280, k=5120	21.8954 µs	22.5661 µs	+3.06%
m=4	20.5204	21.1588	+3.11%
m=32	20.9070	21.5700	+3.17%
m=416	26.3475	27.2643	+3.48%

proj (5120,1024), gate_ffn1 (6400,5120), ffn2 (5120,3200) are bit-identical at every m — including the two off-grid ones, so the transfer machinery itself is not uniformly broken.

(n=1280, k=5120) is off-grid for fp8_block on this table (399 collected (n,k) sites; nearest: (1536,5120), (1024,5120), …) → the nearest-site transfer in util space (log2 IDW, curve-coverage filter, distance gate) disagrees between the Python perf_interp and the Rust port for this neighborhood. Python is the spec (.claude/rules/rust-core/parity.md).

Numerology check: qkv is the only tp8 per-rank shape hitting this query; python gen-phase qkv total 357.33 ms × 3.06% = +10.93 ms, matching the observed engine-level gen delta (+10.946 ms) exactly. tp1/2/4 shapes are on-grid → bit-identical, which is why the smoke matrix (303 parity cases) never fired.

Second DRIFT (same mechanism class, not yet isolated)

Llama-4-Scout-17B-16E | gb200 | vllm 0.24.0 | disagg: same identity-selection drift signature (same config (p)tp1dp4etp4 / (d)tp4etp4 gets tpot 3.998 py vs 3.595 rs ≈ 10%), likely an off-grid transfer on a MoE/GEMM table; isolate while fixing.

Impact / adjudication under PR #1454

• Frontier-amplified effect: envelope max deviation 8.9% (Qwen) / 7.6% (Scout) at the frontier extremes because per-step deltas flip pareto row selection.
• Both entries are on power-carrying systems (h200_sxm, gb200) — exactly the set PR feat(sdk): default the engine step to the compiled Rust engine (Phase 2 PR-1) #1454's default routing keeps on the Python step. No default user sees these values post-flip; they are reachable only via explicit engine_step_backend=rust.
• Adjudication: formally accepted for the PR feat(sdk): default the engine step to the compiled Rust engine (Phase 2 PR-1) #1454 flip; must be fixed before PR-2's energy port removes the power routing carve-out (which would put h200/gb200 defaults on the compiled engine).

Fix checklist (per parity discipline)

• Align the Rust site-transfer with Python for this neighborhood (site set / IDW weights / coverage filter — bisect which stage diverges).
• Isolate + fix the Scout/gb200 entry.
• Add both trigger configs to the parity SMOKE_CASES (the July audit lesson: every parity fix adds its trigger config).
• Re-run the two scan entries → STRICT_PASS.

🤖 Generated with Claude Code