What This Is
This started on Reddit's r/LocalLLaMA, where a user shared results from running a 4-bit quantized version of Qwen3.8-Flash-Next (Alibaba's Tongyi Qianwen) — a 93.7 GB model file — locally. The setup: two RTX 3060s (gaming-grade second-hand cards, going for roughly $70–$100 each on the used market) plus an AMD 7800X3D CPU.
A few terms first. "125B parameters + 6B activated" refers to a MoE (Mixture of Experts) architecture — the model has a huge total capacity, but each query only activates a small portion, which is theoretically why consumer-grade GPUs can run it at all. "Prefill speed" measures how fast the model "reads" your input, in tokens/second.
The user initially hit 36 tokens/second — meaning a prompt of roughly 8,000 Chinese characters would take nearly 4 minutes just to "read." After tweaking parameters, he traced the root cause to a llama.cpp (the dominant open-source inference tool) default for "-sm tensor." Switching to "-sm layer" immediately jumped to 135 tokens/sec; bumping ubatch to 2048 pushed it to 400 t/s — 11× faster than the starting point.
Industry View
A deserved note of recognition: Alibaba's Tongyi Qianwen (Qwen) line is genuinely advancing fast. A year ago, running a 125B model on two consumer-grade GPUs was basically unthinkable; now it not only runs but tunes up to near-industrial speeds. This shows Chinese model companies are holding their own on "capability at equivalent parameter counts."
But we think three things warrant cool-headedness. First, 400 t/s is the input "reading" speed — the decode speed for generating replies is only 12–15 t/s, and that's what actually shapes real conversational UX. Second, this setup relied on a community fork, ik_llama.cpp; the main llama.cpp branch ships with a default config that delivers a 7× performance hit — meaning open-source inference software is far from "install and go" maturity. Third, a 90+ GB model plus 100+ GB of memory is still fundamentally a server-class load, not truly "consumer-grade."
Impact on Regular People
For Enterprise IT: Local deployment's appeal to SMBs is mainly data sovereignty and long-term cost control. This case shows that "can it run" is no longer the question; "can it run well" still depends on whether the team is willing to wrestle with community forks like ik_llama.cpp — many companies will fall back to cloud APIs for that reason, and that trade-off should be on the table during selection.
For Individual Careers: "Making the model run, and run fast" is shifting from a research-PhD specialty into a hard skill for AI engineers. People who can talk fluently about tokens/sec, batch size, and VRAM occupancy are gaining bargaining power in enterprise AI deployment projects.
For the Consumer Market: This case is good news for hobbyists who "build PCs to play with AI"; for the vast majority of regular users, hitting the web version or mobile app to access cloud LLMs remains far more convenient.