Skip to content

BENCHMARK_KDA_DECODE_GB200.md

Latest commit

 

History

History
145 lines (117 loc) · 6.35 KB

BENCHMARK_KDA_DECODE_GB200.md

File metadata and controls

145 lines (117 loc) · 6.35 KB
Raw

Benchmark Results - KDA Decode

Auto-generated by benchmarks/bench_kda_decode.py on 2026-04-22 19:27:18.

GPU: NVIDIA GB200 | CUDA: 13.0 | PyTorch: 2.9.1+cu130 | Python: 3.12.3

Decode setting: single-token (T=1), K=128, V=128.

Summary

  • v-last speedup (FLA/cuLA): avg=1.11x, min=1.00x, max=1.25x
  • k-last speedup (FLA/cuLA): avg=1.24x, min=1.14x, max=1.41x
  • Batch sizes: [1, 4, 8, 16, 32, 64, 128, 256]
  • Q/K heads (H): [8, 32, 64]
  • V heads (HV): 128
  • Timing params: warmup=30, rep=200, ncu_mode=False

KDA Decode (H=8, HV=128, K=128, V=128)

Performance

N cuLA v-last (ms) cuLA k-last (ms) FLA v-last (ms) v-last speedup k-last speedup
1 0.0518 0.0510 0.0616 1.19x 1.21x
4 0.0586 0.0544 0.0619 1.06x 1.14x
8 0.0512 0.0494 0.0586 1.14x 1.19x
16 0.0705 0.0611 0.0726 1.03x 1.19x
32 0.1360 0.1191 0.1384 1.02x 1.16x
64 0.2620 0.2301 0.2656 1.01x 1.15x
128 0.5000 0.4243 0.5206 1.04x 1.23x
256 0.9759 0.8331 1.0317 1.06x 1.24x

Accuracy (Output)

N cuLA v out RMSE cuLA v out rel cuLA k out RMSE cuLA k out rel
1 2.980e-08 1.359e-04 2.980e-08 1.359e-04
4 1.490e-08 9.084e-05 1.490e-08 9.084e-05
8 8.942e-08 7.267e-04 8.942e-08 7.267e-04
16 1.350e-07 1.453e-03 1.350e-07 1.453e-03
32 1.315e-07 1.073e-03 1.315e-07 1.073e-03
64 1.212e-07 1.073e-03 1.212e-07 1.073e-03
128 1.035e-07 7.440e-04 1.035e-07 7.440e-04
256 1.143e-07 1.603e-03 1.143e-07 1.603e-03

Accuracy (State)

N cuLA v state RMSE cuLA v state rel cuLA k state RMSE cuLA k state rel
1 4.369e-09 2.442e-07 4.369e-09 2.442e-07
4 3.985e-09 1.784e-07 3.985e-09 1.784e-07
8 4.310e-09 2.378e-07 4.310e-09 2.378e-07
16 4.269e-09 2.378e-07 4.269e-09 2.378e-07
32 4.244e-09 2.378e-07 4.244e-09 2.378e-07
64 4.254e-09 2.071e-07 4.254e-09 2.071e-07
128 4.349e-09 2.966e-07 4.349e-09 2.966e-07
256 4.390e-09 2.382e-07 4.390e-09 2.382e-07

KDA Decode (H=32, HV=128, K=128, V=128)

Performance

N cuLA v-last (ms) cuLA k-last (ms) FLA v-last (ms) v-last speedup k-last speedup
1 0.0522 0.0502 0.0605 1.16x 1.20x
4 0.0585 0.0546 0.0647 1.11x 1.19x
8 0.0541 0.0509 0.0667 1.23x 1.31x
16 0.0736 0.0646 0.0739 1.00x 1.14x
32 0.1308 0.1123 0.1398 1.07x 1.24x
64 0.2528 0.2207 0.2662 1.05x 1.21x
128 0.4669 0.3911 0.5248 1.12x 1.34x
256 0.9017 0.7582 1.0352 1.15x 1.37x

Accuracy (Output)

N cuLA v out RMSE cuLA v out rel cuLA k out RMSE cuLA k out rel
1 3.255e-11 9.719e-08 3.255e-11 9.719e-08
4 1.195e-07 7.962e-04 1.195e-07 7.962e-04
8 8.492e-08 6.793e-04 8.492e-08 6.793e-04
16 6.297e-08 6.614e-04 6.297e-08 6.614e-04
32 1.012e-07 5.274e-04 1.012e-07 5.274e-04
64 1.039e-07 1.055e-03 1.039e-07 1.055e-03
128 1.409e-07 1.374e-03 1.409e-07 1.374e-03
256 1.113e-07 1.603e-03 1.113e-07 1.603e-03

Accuracy (State)

N cuLA v state RMSE cuLA v state rel cuLA k state RMSE cuLA k state rel
1 3.774e-09 1.763e-07 3.774e-09 1.763e-07
4 4.163e-09 1.661e-07 4.163e-09 1.661e-07
8 4.248e-09 1.993e-07 4.248e-09 1.993e-07
16 4.296e-09 2.451e-07 4.296e-09 2.451e-07
32 4.345e-09 2.178e-07 4.345e-09 2.178e-07
64 4.384e-09 2.178e-07 4.384e-09 2.178e-07
128 4.400e-09 2.194e-07 4.400e-09 2.194e-07
256 4.430e-09 2.091e-07 4.430e-09 2.091e-07

KDA Decode (H=64, HV=128, K=128, V=128)

Performance

N cuLA v-last (ms) cuLA k-last (ms) FLA v-last (ms) v-last speedup k-last speedup
1 0.0521 0.0524 0.0651 1.25x 1.24x
4 0.0568 0.0526 0.0620 1.09x 1.18x
8 0.0469 0.0457 0.0548 1.17x 1.20x
16 0.0679 0.0592 0.0736 1.08x 1.24x
32 0.1309 0.1138 0.1400 1.07x 1.23x
64 0.2401 0.2011 0.2678 1.12x 1.33x
128 0.4587 0.3828 0.5244 1.14x 1.37x
256 0.8923 0.7435 1.0453 1.17x 1.41x

Accuracy (Output)

N cuLA v out RMSE cuLA v out rel cuLA k out RMSE cuLA k out rel
1 1.490e-08 5.830e-05 1.490e-08 5.830e-05
4 7.465e-09 3.427e-05 7.465e-09 3.427e-05
8 5.990e-08 2.741e-04 5.990e-08 2.741e-04
16 7.778e-08 5.482e-04 7.778e-08 5.482e-04
32 8.604e-08 5.482e-04 8.604e-08 5.482e-04
64 1.411e-07 1.953e-03 1.411e-07 1.953e-03
128 1.015e-07 8.741e-04 1.015e-07 8.741e-04
256 1.038e-07 6.510e-04 1.038e-07 6.510e-04

Accuracy (State)

N cuLA v state RMSE cuLA v state rel cuLA k state RMSE cuLA k state rel
1 3.888e-09 2.085e-07 3.888e-09 2.085e-07
4 4.209e-09 1.955e-07 4.209e-09 1.955e-07
8 4.347e-09 1.858e-07 4.347e-09 1.858e-07
16 4.411e-09 1.858e-07 4.411e-09 1.858e-07
32 4.382e-09 1.717e-07 4.382e-09 1.717e-07
64 4.389e-09 1.625e-07 4.389e-09 1.625e-07
128 4.400e-09 2.728e-07 4.400e-09 2.728e-07
256 4.430e-09 2.334e-07 4.430e-09 2.334e-07

Reproduce

python benchmarks/bench_kda_decode.py