A number on Reddit this week made us stop and look: a single 2020-vintage RTX 3090 running Qwen3.8-27B hit 140 token/s. That's 1.9x llama.cpp — and the developer who wrote that CUDA kernel had help from another large model, Claude.

What this is

A megakernel collapses what would normally be multiple GPU launches into a single call. The Qwen3.8 series supports speculative decoding — a smaller model drafts several tokens first, and the larger model verifies them in one pass. Conventional implementations dispatch to the GPU multiple times; this kernel fits the entire verification loop into a single launch, so verifying 4-5 draft tokens costs almost as much as verifying one.

Actual numbers: 140 vs 73 token/s in code-writing (1.9x); 95 vs 56 on a 4K-token context (1.7x); 1,600 vs 1,100 token/s in input processing (1.4x). The project ships as an OpenAI-compatible server — a drop-in replacement for llama-server.

Industry view

Worth underlining the constraints: the kernel is tuned only for a single 3090 with the unsloth Q4_K_M quantization. Swap in a 4090 or A100 and there's no guarantee it'll work. This is hardware-model-kernel lock-in taken to its extreme — not a general-purpose solution.

Other developers are pushing back: single-point benchmark reproducibility is in doubt; stability across varying context lengths needs more samples; KVCache drops to q8 quantization past 80K tokens — accuracy risk in long-context scenarios warrants watching.

But the upside is equally clear: frontier LLMs writing low-level CUDA code means the capability frontier in system programming keeps expanding. "AI optimizing AI" is no longer a slide-deck slogan — it's showing up in GitHub repos.

Impact on regular people

  • For enterprise IT: Don't change anything yet. The megakernel is a single-point extreme optimization; enterprise inference still leans on mature frameworks like vLLM and SGLang. But the result signals there's still meaningful headroom for running LLMs on consumer hardware — the hardware costs of private deployment may come in lower than expected.
  • For individual professionals: Open-source tinkerers willing to run local models now have another option. For data-not-leaving-the-premises scenarios (contracts, internal code), 27B-class models can already carry real workloads — but you need to grasp the basics of CUDA and quantization.
  • For the consumer market: The RTX 3090 has been out of production for three years, yet with optimization it can deliver cloud-API-class speeds. The "must have the latest card" narrative is loosening — the secondhand GPU and GPU-rental markets could get more active.