A developer this week successfully ran a Qwen 35B-class MoE (Mixture of Experts) model locally on an 8GB RTX 4060 GPU—which is itself a milestone. But he also reported that the latest VRAM caching optimization actually slowed generation by 20-50%, exposing the true state of local AI deployment: "hardware ready, software not yet mature."
What this is
A technical post on Reddit's r/LocalLLaMA community caught attention this week: a developer used the latest PR (pull request) #29887 of llama.cpp (a leading open-source LLM inference tool) to run a Qwen 35B-class MoE model on a home machine with an 8GB VRAM GPU and 32GB of system RAM.
MoE models split a large model into dozens of "small experts," activating only a subset per inference—so in theory they can run larger models on less VRAM. This PR's design: keep frequently-called expert weights resident in GPU VRAM, place the rest in regular memory, claiming speed improvements.
Benchmarks flipped the script. With caching enabled, generation speed dropped from a default of 23 characters/second to 12-15 characters/second—the larger the cache, the slower it got. The original poster asked directly: "Did I misconfigure something?"
Industry view
The local large-model track has visibly heated up over the past 12 months. Qwen, DeepSeek, and Llama are all releasing large MoE models, while RTX 4090 and Apple M-series users are trying to cut their cloud API subscription costs.
Dissent deserves airtime. So-called "consumer GPU runs 35B" is currently more "can run" than "runs well." This developer's data isn't isolated—other community members running the same model reported similar results. When VRAM is tight, the overhead of shuffling weights between system memory and the GPU may exceed the cost of direct memory access. The PR author himself acknowledged further optimization is needed.
From another angle, though, this track's long-term value is real: once optimization is in place, the hardware bar for enterprises deploying private large models locally will drop from "several A100s" to "a single consumer card"—a genuine win for data-sensitive businesses (finance, healthcare, legal).
Impact on regular people
For enterprise IT: The "hardware budget sheet" for local large-model deployment can't be written just yet. Procuring based on cloud API pricing remains the safer call—we recommend tracking this PR's follow-ups closely.
For working professionals: Running 35B models on personal laptops remains a hobbyist toy for now. A throughput of 10-something characters per second is unusable in office scenarios—we suggest waiting 1-2 more version iterations.
For the consumer market: The next generation of 16GB/24GB VRAM consumer GPUs (expected in a dense 12-18 month release window) will likely become the new baseline for "home AI hosts"—but software optimization hasn't kept pace with hardware progress.