Back to home

Compare

Comparing: Open Source Runs MoE on a 16GB GPU at 2,000+ Tokens per Second & 开源社区让 16GB 显卡跑混合专家模型,速度突破每秒两千 token

AEN
ExLlamaV3Qwen 3.6 35B A3BRTX 9070 XT·

Open Source Runs MoE on a 16GB GPU at 2,000+ Tokens per Second

A Reddit developer has disclosed that, on an RTX 9070 XT with 16GB of VRAM, Qwen 3.6 35B A3B reached a prefill speed of up to 2,606 tokens per second by offloading some expert modules to memory through ExLlamaV3. At a 32K context length, the speed was roughly 2,000 tokens per second. What deserves our attention is not the isolated benchmark, but the shift in local large-model requirements: from “needing an expensive GPU” to “squeezing a limited device through coordinated hardware and software optimization.”

What this is

Mixture-of-experts models (MoE, an architecture that activates only some parameter modules for each input) can generally reduce inference computation, but the complete model still requires substantial VRAM. This developer placed some expert parameters in DDR4 memory and had the CPU and GPU compute together. At the same time, they tuned the batch size, CPU thread count, and context length so the GPU could process more requests at once.

The results showed a clear benefit once the batch size was increased to 5,120, with no further gains from raising it further. Adjusting the CPU thread setting produced a modest improvement, while several experimental switches caused performance regressions. The author also encountered a core dump. The full tests and charts have been published on GitHub, but the current evidence comes primarily from a single device and a single configuration.

Industry view

We believe the significance of this kind of optimization lies in expanding the range of devices capable of running local models. When VRAM is insufficient, keeping less frequently used parameters in memory amounts to trading speed for capacity, potentially allowing individual developers and small teams to deploy larger models at a lower hardware cost. For cloud models, it also suggests another approach: not every model parameter has to reside on fast compute hardware at all times.

The risks are equally clear. Moving data between the CPU and memory introduces latency, and the test results depend heavily on the GPU, memory bandwidth, context length, and batching method. A one-off benchmark from the open-source community cannot be equated directly with the capabilities of a stable product. Some parameter switches can also crash the program, requiring developers to troubleshoot the issue themselves. In other words, this is a viable engineering path, not a mature consumer-grade solution.

Impact on regular people

For enterprise IT: The hardware threshold for local deployment of mixture-of-experts models may fall, making it worth testing for teams with limited budgets that cannot send data off-site. Stability, operational complexity, and total cost of ownership, however, still need to be evaluated separately.

For individual professionals: Developers can build a more capable private-model environment with a single consumer GPU. The major speed gains, however, occur during high-concurrency batch processing; whether everyday interaction becomes faster still depends on the model size and context length.

For the consumer market: Improvements in local-model capabilities could bring features such as “offline document processing and code assistance” to personal computers. At this stage, however, the approach is better suited to technical users, and community benchmark results should not be treated as promises of a mass-market product.

BZH
ExLlamaV3Qwen 3.6 35B A3BRTX 9070 XT·

开源社区让 16GB 显卡跑混合专家模型,速度突破每秒两千 token

一名 Reddit 开发者披露:在 16GB 显存的 RTX 9070 XT 上,通过 ExLlamaV3 卸载混合专家模型的部分专家模块,Qwen 3.6 35B A3B 的预填充速度最高达到每秒 2606 个 token,32K 上下文时约为每秒 2000 个 token。我们注意到,真正值得关心的不是单项跑分,而是本地大模型的门槛正从“需要昂贵显卡”转向“通过软硬件配合压榨有限设备”。

这是什么

混合专家模型(MoE,指每次处理输入时只调用部分参数模块的模型架构)通常可以降低推理计算量,但完整模型仍需要较多显存。这名开发者把部分专家参数放入 DDR4 内存,由 CPU 与 GPU 协同计算;同时调整批处理大小、CPU 线程和上下文长度,让显卡一次处理更多请求。

结果显示,批处理大小提高到 5120 后收益明显,继续增加没有进一步提升。CPU 线程设置带来小幅改善,几项实验性开关则出现性能倒退;作者还遇到了核心转储问题。完整测试和图表已发布在 GitHub,但目前主要来自单台设备和单次配置实验。

行业怎么看

我们认为,这类优化的意义在于扩大本地模型的可运行设备范围。显存不够时,把冷门参数留在内存,相当于用速度换容量,可能让个人开发者和小团队以较低硬件成本部署较大模型。对云端模型而言,这类方案也提供了另一条思路:不一定每次都把所有模型参数放在高速计算设备上。

风险也很清楚。CPU 与内存之间的数据搬运会带来延迟,测试结果高度依赖显卡、内存带宽、上下文长度和批处理方式;开源社区中的单次跑分不能直接等同于稳定产品能力。部分参数开关还会导致程序崩溃,开发者需要自行排查。换言之,它更像一条可用的工程路径,而不是已经成熟的消费级方案。

对普通人的影响

对企业 IT:混合专家模型的本地部署门槛可能下降,预算有限但数据不能外流的团队值得测试;不过,稳定性、运维复杂度和总持有成本仍需单独核算。

对个人职场:开发者可以用单张消费级显卡搭出更强的私有模型环境,但速度提升主要出现在高并发批量处理,普通交互是否更快仍要看模型大小与上下文。

对消费市场:本地模型能力提升可能推动“离线文档处理、代码辅助”等功能进入个人电脑,但现阶段更适合技术用户,不宜把社区跑分视为大众产品承诺。