What this is
1.04 million token context, a 27B-parameter model, 119 tokens per second—all three numbers appeared together in a single Reddit post this week, on a pair of consumer RTX 5090 GPUs. What makes this worth paying attention to: the hardware barrier to self-hosted large models is being punched through by amateur projects.
The developer modified an open-source inference engine called NInfer (a Qwen-specific tool written from scratch in C++20 and CUDA) to do two things: split model weights across the two GPUs for parallel computation, then stack the YaRN context extension algorithm on top to stretch the native 260,000-token window to 1.04 million. On a 650,000-token long-document workload, it hits 119 tok/s—2.8x faster than vLLM, the most widely used open-source inference framework.
Key detail: once vLLM exceeds the native context window, its speculative decoding hit rate (a trick where a small model guesses first and a large model verifies) drops straight to zero. NInfer still maintains 55–60%—that's where the gap comes from.
Industry view
Supporters see this as a win for the open-source ecosystem—diversification of inference engines means enterprises aren't locked to a single framework, and long context is no longer a luxury that requires an H100.
But we've noticed three overlooked limitations:
First, these are single-stream (one user at a time) lab numbers, not the multi-user concurrent scenarios of enterprise production. NInfer hasn't claimed how many simultaneous users it can handle.
Second, in the prefill phase (when the model first reads the entire prompt), vLLM is actually 20–30% faster. A 1 million-token prompt takes NInfer roughly 18 minutes—unusable for any real-time business.
Third, "consumer-grade" is relative: two RTX 5090s run close to RMB 50,000—for enterprises that's just the cost of one H100 saved, far from "dirt cheap."
The deeper risk: long context does not equal usable context. No one has verified the model's information recall ability at the 1 million-token position in this test.
Impact on regular people
For enterprise IT: The minimum budget for self-hosting large models has dropped from an 8-card H100 server (around RMB 1.5 million) to two 5090s (around RMB 50,000). But that doesn't mean you can jump in immediately—inference engine selection, operations, and stability all still lack mature solutions.
For individual professionals: In long-document work (contracts, codebases, research reports), "feeding the entire project to AI" is becoming viable—but it will take another 1–2 years to become a daily tool.
For consumer markets: No direct impact for now. This is more of a wake-up call to hardware vendors and cloud providers—the "AI premium" baked into your pricing may be more inflated than you think.