Qwen 35B Now Runs on RTX 3090 — Local AI Hardware Barrier Drops, Speed Still Lags

What this is

This week, a specific question on Reddit caught our attention: a user wanted to run Alibaba's Qwen 35B parameter MoE model (Mixture of Experts — the model internally splits into multiple sub-networks, activating only a portion at a time; A3B means roughly 3 billion parameters are activated per inference) on a 2020-era RTX 3090 with 24GB of VRAM. He tried tools like Strata and llama.cpp (open-source inference engine). The result: it loads, but the speed is too slow to be usable.

Our takeaway: the 3090 is a four-year-old flagship GPU, yet it can still fit a 35B model. That means the hardware barrier for local AI has been pulled down significantly — only the experience bar remains stuck.

Industry view

The bullish camp: Chinese teams like Qwen and DeepSeek are betting on MoE architecture by design — large total parameters, small per-call activation — to cut inference costs. Open-source inference engines (llama.cpp, vLLM) shipped dense updates this week, all playing catch-up on MoE optimization.

But the skeptical view is just as real: the user's complaint proves that "runs" doesn't mean "runs fast." Current generation speed on a 3090 is only a few tokens per second (the smallest unit of text the model processes, roughly one Chinese character), nowhere near usable. In other words, the hardware bar is falling, but the experience bar is still stuck.

Impact on regular people

For enterprise IT: Mid-sized companies looking to self-host local AI as a partial cloud replacement can use a secondhand 3090 (around 1,500 RMB) for a minimum viable validation. Going into production still requires multi-GPU setups or a 4090/5090.

For individual professionals: Regular office workers don't need to tinker for now — local LLMs still rely on the command line and are far less convenient than simply using the web versions of ChatGPT, Tongyi, or DeepSeek.

For the consumer market: If inference speed gets solved within 1-2 years, it could spawn a new "home AI host" category — similar to the NAS (home network storage) or mini-PC waves of years past.