The Qwen 27B model (B for billion parameters — roughly "brain size," so 27B means 27 billion) hit 138 tokens/sec (tokens are the smallest text units the model processes) this week on a single consumer-grade RTX 3090 — up from just 82 three weeks ago. The cost curve for local large models is being quietly flattened by the open-source community. The same developer reported a 64-way concurrency peak of 942 (handling multiple user requests simultaneously), with second-turn latency in long conversations cut from 23 seconds to 1 second.

What this is

More important than raw speed is the second-turn latency. Going from 23 seconds to 1 second turns a local AI assistant from "tolerable" into "actually usable." The gain came from four engineering optimizations: speculative decoding (predicting multiple tokens at once), prefix caching (reusing the conversation's opening), KV cache reuse (reusing segments already generated in the conversation history), and caching tuned for the hybrid architecture that combines Transformer and Mamba neural networks. None of these touch model capability — they only touch "how to run the model faster."

Industry view

The bullish camp says: this confirms a judgment we already held — the cost curve for local large models is being quietly flattened by the open-source community. A single consumer card with 24GB of VRAM (market price roughly 1,500-2,000 RMB, or ~$200-280) can already power a mid-sized team's internal AI assistant, and the electricity bill may be an order of magnitude cheaper than a monthly cloud API subscription.

But we should also stay cool-headed: this is one developer's tuning result on his own machine. Production stability, long-context reliability, and the model's own capability ceiling all remain unverified at scale. The RTX 3090 is capped at 250W, and thermals and noise are real problems. Alibaba invests far less in inference than in training — almost every optimization here comes from the community, not the vendor. A Reddit post is not an industry inflection point, but the direction is clear.

Impact on regular people

For enterprise IT: the hardware threshold for self-hosted AI assistants is dropping from "datacenter-grade GPU" to "desktop-grade graphics card." It's worth running the total-cost-of-ownership numbers (electricity, depreciation, labor) again this year.

For working professionals: once long-conversation latency reaches 1-second class, the experience of using local AI to write code, edit documents, and clean up meeting notes approaches ChatGPT — with data that never leaves the machine. For lawyers, doctors, and R&D roles, this is the first time local AI is "actually usable."

For the consumer market: GPUs that can run 27B-class models are shifting from "gaming card" to "small-business server." Watch the secondhand RTX 3090 market.