A post on Reddit's LocalLLaMA community went viral this week: a user paired a single NVIDIA RTX 5090 consumer GPU with Ninfer, a new inference engine (the software layer that makes trained models actually "run" and produce output), plus a specially compressed Qwen3 8B model, and hit an average of 170 tokens/sec with peaks of 220. Tokens/sec is the unit for measuring how fast large models spit out text; 220 is roughly 1.5x the speed of smooth conversation. The author reports it's more than double the speed of llama.cpp (the most widely used open-source inference engine), which they had been running before.
What This Is
The inference engine — the "scheduling software" that runs the model — has long been the bottleneck for local large-model deployment. Over the past two years, tools like llama.cpp, vLLM, and Ollama have let ordinary developers run models on their own machines, but the speed rarely matches the smoothness of cloud APIs. Ninfer's breakthrough this time: it simultaneously absorbs the hardware dividend from this generation of 5090 cards, Qwen3's built-in MTP (Multi-Token Prediction, which lets the model guess multiple tokens at once and then verify), and the NVFP4 4-bit floating-point format (which compresses model size to roughly a quarter of the original — and counterintuitively runs faster). All three stacked together, and the single-card experience doubles outright.
Industry View
Community members replicated and confirmed the numbers, but skeptics poured cold water: the 220 tokens/sec is a peak-scenario result, and it requires cranking the context window (how much text the model can "see" at once) to an extreme 240,000 tokens. Under typical usage, you might only see 70% of that. More importantly, this combination is heavily dependent on this generation of 5090 hardware and a specific model compression format — it's not a "one-click and it works for anyone" solution. But we noticed something more worth paying attention to: three forces — hardware upgrades, inference optimization, and model compression — are accelerating simultaneously, and the speed advantage of cloud APIs is being eaten away. The premium that OpenAI, Anthropic, and others earned through "speed" in the past will need to be discounted going forward. Domestic model companies, GPU makers, and all-in-one machine vendors should all recalculate the books.
Impact on Regular People
For enterprise IT: models that previously had to be paid for by the token to cloud providers can now run on a single 5090 card costing a little over ten thousand RMB, with sensitive corporate data no longer needing to be uploaded to third parties.
For individual professionals: white-collar workers willing to tinker will soon be able to build private AI assistants on their own computers, with conversation content no longer scraped by the cloud for training.
For the consumer market: AI hardware and "one-click deployment" tools will keep getting cheaper — this is the next growth story for NVIDIA, Lenovo, Dell, and the rest.