Collect DeepSeek-V4-Pro attention module perf data (sglang 0.5.14 + vllm 0.24.0, 6 systems; Blackwell=native, Hopper=FP8)

[tianhaox]

Why

DeepSeek-V4-Pro (128-head, hidden 7168) attention module perf data was never collected as a campaign anywhere: the dsv4 case plan pinned full/raw collection to one canonical Flash-FP8 profile because the old persisted keys could not distinguish Flash from Pro (see #1429). The only Pro rows in the database are an 8-row smoke probe (bs=1, isl 128/129, tp 1–8) in b200_sxm/sparse_attention/sglang/0.5.10. Everywhere else a Pro query resolves nearest-native into Flash buckets, so AIC systematically underestimates Pro attention latency.

#1431 unblocks this end to end: module tables key [native][local] (Flash 64 and Pro 128 cannot collide), the collector derives rank-local heads from the HF config (the old Flash-only NATIVE_HEADS=64 fallback would have mislabeled every Pro row), and the canonical-model pin is lifted for the geometry-keyed ops.

Prerequisites

• fix: enforce vLLM KV-cache budget in disagg sweep; run module collectors with engine-fused ops (#1396) #1423 merged
• fix: unify DSV4 num_heads to the rank-local convention — data migration, strict loaders, collector guard (#1429) #1431 merged (stacked on fix: enforce vLLM KV-cache budget in disagg sweep; run module collectors with engine-fused ops (#1396) #1423; includes the pin lift)

What to collect

dsv4_csa_context_module, dsv4_hca_context_module, dsv4_csa_generation_module, dsv4_hca_generation_module for Pro, on every (system, backend) cell that already has Flash module tables:

System	SM	Backend(s)	Pro artifact — do not mix
b200_sxm	SM100	sglang 0.5.14, vllm 0.24.0	deepseek-ai/DeepSeek-V4-Pro (native — Blackwell)
gb200	SM100	sglang 0.5.14, vllm 0.24.0	deepseek-ai/DeepSeek-V4-Pro (native — Blackwell)
b300_sxm	SM103	sglang 0.5.14, vllm 0.24.0	deepseek-ai/DeepSeek-V4-Pro (native — Blackwell)
gb300	SM103	sglang 0.5.14, vllm 0.24.0	deepseek-ai/DeepSeek-V4-Pro (native — Blackwell)
h100_sxm	SM90	sglang 0.5.14, vllm 0.24.0	sgl-project/DeepSeek-V4-Pro-FP8 (converted — Hopper)
h200_sxm	SM90	sglang 0.5.14, vllm 0.24.0	sgl-project/DeepSeek-V4-Pro-FP8 (converted — Hopper)

rtx_pro_6000_server (SM120) is out of scope: it has no DSV4 module tables today and the native path is unvalidated on SM120.

Artifact rule (owner directive): converted sgl-project/*-FP8 builds are the Hopper deployment artifacts; native deepseek-ai/* (FP4 experts) are the Blackwell ones. The full/raw plan default stays artifact-uniform (Pro-FP8) because the persisted module key carries no artifact dimension — Blackwell cells therefore use targeted native runs, which override the default cleanly. Measured artifact sensitivity on the attention module is small (mean 2–5%, max 8.7%, b200 native-Flash vs Flash-FP8 same-key rows), so the uniform default is safe where it applies; the campaign still uses the platform-right artifact.

How

Smoke first on one Blackwell + one Hopper node (validates FlashMLA h_q=128 and Pro memory bounds on 0.5.14 — the b200 probe already ran native Pro through this exact single-process TP simulation in the 0.5.10 era):

python3 collector/collect.py --backend sglang \
  --model-path deepseek-ai/DeepSeek-V4-Pro --gpu b200_sxm \
  --ops dsv4_csa_context_module dsv4_hca_context_module \
        dsv4_csa_generation_module dsv4_hca_generation_module \
  --smoke

Then per cell (drop --smoke):

# Blackwell (b200_sxm | b300_sxm | gb200 | gb300) — native artifact
python3 collector/collect.py --backend <sglang|vllm> \
  --model-path deepseek-ai/DeepSeek-V4-Pro --gpu <system> \
  --ops dsv4_csa_context_module dsv4_hca_context_module \
        dsv4_csa_generation_module dsv4_hca_generation_module

# Hopper (h100_sxm | h200_sxm) — converted FP8 artifact
python3 collector/collect.py --backend <sglang|vllm> \
  --model-path sgl-project/DeepSeek-V4-Pro-FP8 --gpu <system> \
  --ops dsv4_csa_context_module dsv4_hca_context_module \
        dsv4_csa_generation_module dsv4_hca_generation_module

Runtime images: pinned lmsysorg/sglang:v0.5.14 / the vllm 0.24.0 collection image from framework_manifest.yaml.

Expected behavior / acceptance

• dsv4_csa_topk_calib is intentionally dropped for Pro with a logged reason (calib keys carry no model geometry; calibration stays on canonical Flash-FP8). Pro CSA correction keeps using Flash calibration constants — acceptable first-order; a Pro-specific calib needs a calib-key contract change (file separately if wanted).
• Merged tables gain a [native=128][local=128/64/32/16] bucket per cell; test_shipped_dsv4_module_tables_are_rank_local must stay green (per-model num_heads * tp_size constant).
• Query sanity: Pro tp 1/2/4/8 context+generation lookups on each system resolve the native-128 bucket (not the Flash-64 fallback).

Cost note

The lifted full/raw plan doubles DSV4 module outer tasks (88 → 176 per op per backend); Pro sweeps are heavier than Flash (7168 hidden). Budget GPU-hours accordingly; targeted --ops-scoped runs above avoid re-collecting Flash.

Refs: #1429, #1431.

[tianhaox]

Phase-1 canary pipelines submitted (collector @ 8622bf9f0, PR #1431 branch):

• 59945077 — sglang 0.5.14 × b200_sxm × deepseek-ai/DeepSeek-V4-Pro (native)
• 59945080 — sglang 0.5.14 × h100_sxm × sgl-project/DeepSeek-V4-Pro-FP8

Phase 2 (remaining 10 cells incl. vllm 0.24.0) fires after phase-1 artifacts pass content acceptance (row counts per op, errors_*.json, rank-local num_heads 128/64/32/16 across tp 1/2/4/8). Trigger script: aic-auto-collector/trigger_dsv4_pro_modules.sh.

[tianhaox]

Collection campaign complete — 48/48 files. Data PR: #1453 (stacked on #1423 → #1431).

All 12 (system, backend) cells collected with the platform-right artifacts (Blackwell = native deepseek-ai/DeepSeek-V4-Pro, Hopper = sgl-project/DeepSeek-V4-Pro-FP8). Quality evidence, per-cell row counts, and the three documented gaps (SM103 vllm cooperative_topk, h100 80GB OOM tail, sglang CSA deep-prefix kernel boundary — all classified with error signatures) are in the PR body.

Measured impact: Pro/Flash matched-shape latency ratio is 1.27–1.42× on Blackwell and 1.52–1.60× on Hopper — the size of the systematic under-prediction this data eliminates.