With an RTX 3090, CPU offload (placing some layers in memory for computation), and the Q8-quantized version of Qwen3.6-35B-A3B, prefill throughput rises from 345 to 466 tokens per second, an improvement of roughly 35%. The result comes from GitHub user makneeee, whose cudamemcpy-tuner automatically optimizes how weights are copied between the GPU and CPU. It does not modify the model itself.
What this is
MoE (Mixture of Experts) refers to a sparse architecture that activates only part of its parameters for each inference. With 35B total parameters but only 3B active per run, its VRAM footprint remains manageable. The problem is that when such a model does not fit on a single consumer GPU, some weights must remain in system memory and move across the PCIe bus. How those weights are transferred therefore becomes a bottleneck.
This project targets that transfer path: it lets the GPU begin computing before all the data has arrived, using execution overlap to mask latency. The result is 30% faster prefill—the phase that processes a prompt by converting the input context into vectors, as opposed to token-by-token generation—on the same hardware and with the same model.
Industry view
The positive side is clear: inference performance is increasingly improving not by “buying a bigger GPU,” but by squeezing more work out of existing hardware. Optimizations like this can take only a few hundred lines of code and a few days of work while saving money that would otherwise go toward new hardware, delivering a tangible cost reduction for researchers and small teams with limited budgets.
Three risks deserve attention. First, these are benchmark results for a single user and a single use case, submitted by the developer alone and not yet independently reproduced. Second, the approach depends on the layer structure of this particular version of Qwen3.6-35B-A3B; changing the model, quantization format, or context length could produce entirely different results. Third, engineering a workaround for moving weights between the GPU and PCIe is, by definition, a “patch.” If upstream inference frameworks such as llama.cpp implement better scheduling natively, users will bear the maintenance burden of this tuning layer.
Impact on regular people
For enterprise IT: the unit cost of deploying private models locally is falling. A single RTX 3090 can now support more concurrency and longer contexts, making more edge-deployment scenarios viable.
For professionals: more open-source models can run on personal machines, meaning analyses involving internal documents do not always need to be sent to an external API. That gives organizations greater control over how sensitive data is handled.
For the consumer market: speed gains from this type of optimization will spread downward to each provider’s local-running editions, ultimately producing faster responses in offline assistants and tools for translation and summarization. For cloud services, however, the impact will come much later; for now, this remains an edge optimization driven by a small number of enthusiasts.