This week, a developer on r/LocalLLaMA ran 159 experiments with a clear conclusion: Alibaba's Qwen3.8-27B (a 27-billion-parameter coding model) running on AMD Strix Halo + RTX 3090 Ti consumer hardware hit 153 tok/s (tokens generated per second) at 32K context, and scored 159 out of 164 on HumanEval (the standard code generation benchmark) — faster and more accurate than a dual-3090 cloud server. What we care about: local AI has crossed the usable threshold, no longer just a hobbyist toy.
What this is
The author didn't use a professional server. Instead, they split the 27B model across the AMD processor's unified memory and a single consumer GPU. Baseline speed was just 9.5 tok/s; after several key optimizations, it hit 153 tok/s. But the most valuable finding isn't at the hardware layer — simply switching the chat template to a more compact version (Qwen-Sharp) cut wall-clock time by 44% and halved output tokens.
In other words: often when "AI feels slow," the problem isn't weak hardware — it's that default prompts make the model verbose.
Industry view
Supporters argue the cost curve for local deployment is accelerating downward. A home workstation costing tens of thousands of yuan, paired with an open-source 27B model, can already outperform a dual-3090 cloud configuration on real coding tasks. For finance and healthcare clients with strict data compliance requirements, this is a real alternative.
But we also need to acknowledge the caveats. First, this is a single developer, single benchmark (HumanEval leans toward short code completion, far from real-world workloads). Second, cloud vendors have already handled the stability, thermal management, and long-term operations of heterogeneous setups like eGPU (external GPU enclosures) + unified memory for you. Third, the 153 tok/s figure heavily depends on speculative decoding (where a small model guesses first and skips when correct) and n-gram (consecutive-word-pattern) modes; on the same setup running long-form generation, speeds drop to 35.8 tok/s — a point the author himself concedes.
Impact on regular people
For enterprise IT: building a local AI inference node for tens of thousands of yuan is becoming viable, especially for clients whose data cannot leave the internal network.
For individual professionals: engineers willing to tinker with the command line can now run a genuinely usable coding assistant on a laptop with an NPU (neural processing unit, an AI-dedicated chip) at home, with near-zero monthly cost.
For the consumer market: future "AI laptops" will face a clearer yardstick — can they run a 20B+ model locally at speeds usable for daily work?