A Reddit developer has disclosed that, on an RTX 9070 XT with 16GB of VRAM, Qwen 3.6 35B A3B reached a prefill speed of up to 2,606 tokens per second by offloading some expert modules to memory through ExLlamaV3. At a 32K context length, the speed was roughly 2,000 tokens per second. What deserves our attention is not the isolated benchmark, but the shift in local large-model requirements: from “needing an expensive GPU” to “squeezing a limited device through coordinated hardware and software optimization.”
What this is
Mixture-of-experts models (MoE, an architecture that activates only some parameter modules for each input) can generally reduce inference computation, but the complete model still requires substantial VRAM. This developer placed some expert parameters in DDR4 memory and had the CPU and GPU compute together. At the same time, they tuned the batch size, CPU thread count, and context length so the GPU could process more requests at once.
The results showed a clear benefit once the batch size was increased to 5,120, with no further gains from raising it further. Adjusting the CPU thread setting produced a modest improvement, while several experimental switches caused performance regressions. The author also encountered a core dump. The full tests and charts have been published on GitHub, but the current evidence comes primarily from a single device and a single configuration.
Industry view
We believe the significance of this kind of optimization lies in expanding the range of devices capable of running local models. When VRAM is insufficient, keeping less frequently used parameters in memory amounts to trading speed for capacity, potentially allowing individual developers and small teams to deploy larger models at a lower hardware cost. For cloud models, it also suggests another approach: not every model parameter has to reside on fast compute hardware at all times.
The risks are equally clear. Moving data between the CPU and memory introduces latency, and the test results depend heavily on the GPU, memory bandwidth, context length, and batching method. A one-off benchmark from the open-source community cannot be equated directly with the capabilities of a stable product. Some parameter switches can also crash the program, requiring developers to troubleshoot the issue themselves. In other words, this is a viable engineering path, not a mature consumer-grade solution.
Impact on regular people
For enterprise IT: The hardware threshold for local deployment of mixture-of-experts models may fall, making it worth testing for teams with limited budgets that cannot send data off-site. Stability, operational complexity, and total cost of ownership, however, still need to be evaluated separately.
For individual professionals: Developers can build a more capable private-model environment with a single consumer GPU. The major speed gains, however, occur during high-concurrency batch processing; whether everyday interaction becomes faster still depends on the model size and context length.
For the consumer market: Improvements in local-model capabilities could bring features such as “offline document processing and code assistance” to personal computers. At this stage, however, the approach is better suited to technical users, and community benchmark results should not be treated as promises of a mass-market product.