From-scratch C++/CUDA LLM inference engine for the NVIDIA RTX 5090 (sm_120a). The fastest single-user inference on the 5090: faster decode than llama.cpp, at-or-ahead of vLLM on NVFP4, and the only engine running native NVFP4 on consumer Blackwell. 100% written by Claude Code.
cpp cuda quantization inference-engine mixture-of-experts blackwell llm safetensors llama-cpp vllm local-llm llm-inference qwen gguf fp4 rtx-5090 sm120 nvfp4 rtx-pro-6000 gated-deltanet
-
Updated
Jun 24, 2026 - Cuda