What this is
The number worth remembering this week: 165. A Reddit user ran a 27B-parameter Qwen model (INT4 quantized — compressing model precision from 16-bit to 4-bit to save VRAM) on a home PC (two RTX 3090s + a standard NVMe SSD), hitting 165 tokens/second on Agent-class workloads. That speed means consumer hardware can finally "handle Agent tasks" — not merely "runs, which is nice."
Three technologies made it possible: INT4 quantization lets the 27B model fit into the 48GB VRAM of two 3090s; speculative decoding lets a small model draft tokens while the large model verifies them in batches, yielding a 2.4x speedup; LMCache (an inference caching system) offloads conversation history into 54GB of RAM and 1.5TB of SSD, forming an 18M-token "memory pool" that's reused rather than recomputed.
The full rig costs under ¥20,000 used.
Industry view
The open-source camp (DeepSeek, Qwen, Llama) is pursuing a "small models + inference optimization" path, and this post is a milestone-level proof point. But a word of caution: this is a one-off engineering-optimization demo with an extremely high reproduction barrier — it requires two PR patches against vLLM (open-source inference engine) 0.28.0, a specific LMCache version, and specific SSD models. Average users copying the setup will most likely fail to get it running.
A cooler take: the intelligence gap between 27B local models and cloud-side 200B+ flagships hasn't narrowed — "runs" doesn't equal "works well." Enterprise production environments still need stability, concurrency, and compliance; consumer hardware's power draw and failure rates can't sustain 24/7 operation.
Impact on regular people
For enterprise IT: The hardware barrier to self-built AI is loosening, but there's still a 1-2 year gap between "demo runs" and "production-ready."
For individual professionals: Tech enthusiasts and small teams can now run large models locally for trials, saving thousands per month in API costs — but solid engineering chops are required.
For the consumer market: This is an early signal of "large models on personal devices." Truly consumer-ready products (on-device LLMs on laptops, phones, and car systems) are still 12-24 months from maturity.