Benchmark Results

Auto-generated by benchmarks/generate_benchmark_md.py on 2026-04-23.

GPU: NVIDIA GB200 | CUDA: 13.0 | PyTorch: 2.9.1+cu130

FLA baseline: flash-linear-attention v0.4.2

KDA (Kimi Delta Attention)

B	T	FLA Triton (ms)	cuLA (ms)	Speedup
1	512	0.602	0.492	1.22x
1	1024	0.633	0.521	1.22x
1	4096	0.750	0.539	1.39x
1	8192	1.393	1.002	1.39x
1	16384	2.707	1.916	1.41x
2	512	0.610	0.523	1.17x
2	1024	0.644	0.524	1.23x
2	4096	1.388	1.005	1.38x
2	8192	2.704	1.933	1.40x
2	16384	5.303	3.821	1.39x

Summary (10 configs): avg=1.32x, min=1.17x, max=1.41x.

Config	FLA Triton (ms)	cuLA (ms)	Speedup
uniform 10seqs T=4096 [409..415] avg=409	0.787	0.582	1.35x
random 10seqs T=4096 [24..1201] avg=409	0.782	0.576	1.36x
skewed 10seqs T=4096 [227..2053] avg=409	0.777	0.575	1.35x
uniform 20seqs T=4096 [204..220] avg=204	0.858	0.633	1.36x
random 20seqs T=4096 [5..787] avg=204	0.831	0.616	1.35x
skewed 20seqs T=4096 [107..2063] avg=204	0.813	0.596	1.36x
uniform 10seqs T=8192 [819..821] avg=819	1.389	1.022	1.36x
random 10seqs T=8192 [48..2401] avg=819	1.413	1.041	1.36x
skewed 10seqs T=8192 [455..4097] avg=819	1.440	1.045	1.38x
uniform 20seqs T=8192 [409..421] avg=409	1.476	1.069	1.38x
random 20seqs T=8192 [9..1574] avg=409	1.476	1.073	1.38x
skewed 20seqs T=8192 [215..4107] avg=409	1.484	1.077	1.38x
uniform 10seqs T=16384 [1638..1642] avg=1638	2.671	1.946	1.37x
random 10seqs T=16384 [95..4802] avg=1638	2.680	1.946	1.38x
skewed 10seqs T=16384 [910..8194] avg=1638	2.684	1.950	1.38x
uniform 20seqs T=16384 [819..823] avg=819	2.677	1.947	1.38x
random 20seqs T=16384 [19..3147] avg=819	2.713	1.971	1.38x
skewed 20seqs T=16384 [431..8195] avg=819	2.689	1.950	1.38x

Summary (18 configs): avg=1.37x, min=1.35x, max=1.38x.

To reproduce:

python benchmarks/bench_kda.py --mode both

B	T	FLA Triton (ms)	cuLA (ms)	Speedup
1	1024	0.108	0.069	1.57x
1	4096	0.174	0.157	1.11x
1	8192	0.331	0.293	1.13x
1	16384	0.638	0.563	1.13x
2	1024	0.094	0.063	1.49x
2	4096	0.305	0.176	1.73x
2	8192	0.585	0.328	1.78x
2	16384	1.139	0.632	1.80x

Persistent CuTe DSL kernel vs FLA Triton varlen.

N (seqs)	T	cuLA (ms)	FLA Triton (ms)	Speedup
5	1020	0.090	0.187	2.09x
5	2045	0.113	0.211	1.87x
5	4095	0.164	0.258	1.58x
5	8190	0.265	0.412	1.56x
5	16380	0.465	0.705	1.52x
5	32765	0.859	1.284	1.49x
8	1024	0.090	0.172	1.92x
8	2048	0.114	0.198	1.74x
8	4096	0.158	0.252	1.60x
8	8192	0.243	0.399	1.64x
8	16384	0.413	0.689	1.67x
8	32768	0.758	1.259	1.66x
10	1020	0.107	0.171	1.60x
10	2040	0.135	0.200	1.48x
10	4090	0.182	0.268	1.47x
10	8190	0.266	0.407	1.53x
10	16380	0.440	0.694	1.58x
10	32760	0.791	1.275	1.61x
12	1020	0.120	0.176	1.47x
12	2040	0.145	0.194	1.34x
12	4092	0.192	0.265	1.38x
12	8184	0.279	0.404	1.45x
12	16380	0.455	0.700	1.54x
12	32760	0.795	1.267	1.59x
16	1024	0.124	0.166	1.34x
16	2048	0.150	0.187	1.25x
16	4096	0.189	0.258	1.37x
16	8192	0.268	0.401	1.49x
16	16384	0.426	0.688	1.61x
16	32768	0.742	1.251	1.68x
20	1020	0.163	0.170	1.04x
20	2040	0.192	0.202	1.05x
20	4080	0.237	0.287	1.21x
20	8180	0.321	0.431	1.34x
20	16380	0.482	0.701	1.45x
20	32760	0.806	1.267	1.57x
25	1000	0.195	0.182	0.93x
25	2025	0.223	0.224	1.01x
25	4075	0.263	0.277	1.05x
25	8175	0.348	0.444	1.27x
25	16375	0.522	0.717	1.37x
25	32750	0.835	1.275	1.53x

Summary (126 configs across uniform/skewed/random): avg=1.48x, min=0.93x, max=2.16x.

To reproduce:

python benchmarks/bench_lightning_attn.py --modes no_state varlen

Single-token decode: la_decode (CuTe DSL) vs fla fused_recurrent (Triton).

B	FLA Triton (ms)	cuLA (ms)	Speedup
1	0.0734	0.0125	5.89x
4	0.0706	0.0132	5.37x
16	0.0750	0.0209	3.59x
64	0.0996	0.0843	1.18x
256	0.3497	0.3121	1.12x

B	FLA Triton (ms)	cuLA (ms)	Speedup
1	0.0988	0.0189	5.23x
4	0.0933	0.0182	5.12x
16	0.0990	0.0209	4.74x
64	0.1040	0.0844	1.23x
256	0.3500	0.3134	1.12x

To reproduce:

python benchmarks/bench_la_decode_vs_fla.py --heads 64 --head-dim 128