On an RTX 5090, a developer branch of llama.cpp increased Qwen3-30B-A3B’s generation speed from 46.1 to 79.2 tokens per second when processing a 128,000-token context—a 1.72x improvement. This is not a new model, but a change to the inference mechanism: instead of reading the entire cache for every generated token, it prioritizes the portions most likely to affect the result.

What deserves our attention is that the bottleneck for local large models is shifting from whether they can run at all to whether they run fast enough without losing information. This approach requires no model retraining or separate service stack, offering a more practical path for enterprise long-document analysis and personal local use.

What this is

The core technique is page-sparse decoding: it divides the KV cache—the intermediate information models retain while processing previous text—into pages of 64 tokens, then directs the model to read the first page, the latest content, and the pages most relevant to the current generation first. In the 128,000-token test, each generation step read only about 21% of the cache on average.

The results vary by model: Qwen3-30B-A3B achieved a 1.72x speedup on an RTX 5090, while its average score fell by 1 point in the RULER long-text benchmark. Qwen3-8B was 1.68x faster on an RTX 3090 and actually improved its score by 0.4 points. Llama-3.2-3B, however, fell 4.5 points on average on multi-information retrieval tasks after a 1.38x speedup and failed the test.

Industry view

On the positive side, this change brings the optimization into a mainstream open-source inference framework rather than requiring enterprises to buy specialized hardware or adopt a new deployment architecture. For workloads involving contracts, codebases, and long meeting transcripts that must run on a single high-end GPU, the speed gain offers more commercial value than simply adding parameters.

The risks are equally clear. The developers themselves acknowledge that prefill speed does not improve, nor does VRAM usage fall; below 32,000 tokens, the approach is actually slower. The full cache remains in VRAM, while the additional page statistics consume roughly 3% to 6% of the cache. Furthermore, each configuration was run only once, and the interval estimates do not account for variation across repeated runs. Current support is limited to CUDA, a single batch, and models remaining entirely on the GPU, while the results for a 256k context have not passed an equivalent quality threshold. The 1.72x gain is therefore an engineering result under specific conditions, not a universal conclusion for every long-text workload.

Impact on regular people

For enterprise IT: the cost of processing long documents locally may fall, particularly for organizations with high-end NVIDIA GPUs and existing llama.cpp deployments. Before rollout, however, teams still need to test recall accuracy against their own document types rather than look only at tokens per second.

For individual professionals: analyzing large document collections, code repositories, or meeting records on a personal computer should feel smoother, although ordinary consumer GPUs and shorter contexts may not deliver noticeable benefits.

For the consumer market: local AI tools may support longer usable contexts and faster responses, but in the short term this is more a developer capability upgrade than a product feature ordinary users can consistently enjoy.