What this is
GitHub user syv-ai yesterday got Alibaba's Qwen 27B-parameter model running at 99 tokens/second (single request) and 1,150 tps (batch) on an RTX 3090 (24GB VRAM, used price 1,500–2,000 RMB / ~$200–270). This means consumer-grade hardware at the sub-$300 price point can now run mid-size LLMs smoothly.
99 tps far exceeds average adult reading speed (4–5 tps), so users won't feel any "waiting." The point isn't that the model runs — it's that it runs smoothly.
Industry view
The open-source community's excitement centers on how fast the hardware bar has dropped — faster than most of us expected. But we owe this a reality check: 99 tps is the result of deep optimization — fp8 KV cache (storing the model's "short-term memory" in low precision), int8 quantization (computing in low precision), FlashAttention (a faster attention algorithm), and probabilistic sampling. Each item is technical work. There's a non-trivial chance ordinary users won't replicate this by copy-pasting the setup.
A more restrained take we think is worth hearing: the RTX 3090 has been out of production for four years. Today's consumer flagships are the 4090 and 5090. The 3090's value here is mostly in proving the approach, not in telling people what to buy today. The 99 tps figure came from a mid-size model — calling this "ChatGPT replacement at home" is premature. This is a community win, not the finish line.
Impact on regular people
For enterprise IT: on-prem deployment hardware costs keep falling, but "tuning labor cost" is the new bottleneck. Don't be fooled by the hardware sticker price.
For individual professionals: data-sensitive roles (legal, finance, R&D) can now seriously evaluate local AI setups. "Data must go to the cloud" is no longer the only option.
For the consumer market: the pattern of consumer GPUs being snapped up by AI builders and gamers isn't changing anytime soon. Used RTX 3090 prices may get propped up by this wave.