Because there is a fixed set of inputs that are reused by flashinfer-bench, a kernel can just remember what it previously output for each input and return that without needing to do any computation, resulting in much lower measured latency than is realistic.
I made a minimal reproducible example based on flashinfer-bench-starter-kit: https://github.com/tripplyons/flashinfer-bench-bug
Because there is a fixed set of inputs that are reused by
flashinfer-bench, a kernel can just remember what it previously output for each input and return that without needing to do any computation, resulting in much lower measured latency than is realistic.I made a minimal reproducible example based on
flashinfer-bench-starter-kit: https://github.com/tripplyons/flashinfer-bench-bug