A public test has pushed local inference for a 40B-parameter model to 64.6 tokens per second: RustingStrata runs on an RTX 3060 with 12GB of VRAM, about 33% faster than llama.cpp, although the model still requires 30GB of system memory.
This result matters because the bottleneck in deploying large models locally is often not GPU compute. The real challenge is efficiently moving weights between the GPU and system memory once the model no longer fits in VRAM. It affects whether enterprises can run larger models on existing hardware and the cost of building private AI tools at home.
What this is
Recursive Flint is a 40B-parameter mixture-of-experts model. A mixture-of-experts model activates only a subset of its parameters for each question rather than using the full model every time.
The UD-Q5_K_XL quantized version used in the test is approximately 27.9GB, so it cannot fit entirely in 12GB of VRAM. RustingStrata makes two main improvements: dynamically placing expert parameters according to the current response, and releasing memory pages already copied to VRAM to reduce redundant transfers. After warmup, it generates 64.6 tokens per second, with 4K prompt prefill reaching approximately 1.14 to 1.30K tokens per second.
Industry view
We believe this test shows that inference engines still have substantial room for optimization and that upgrading hardware is not the only option. With the same GPU and model, adjusting how memory and VRAM are scheduled can deliver roughly a one-third performance improvement.
The risks are equally clear: this was an individual test on one device, with one model and a specific version. Because the model exceeds the GPU's memory capacity, initial loading and cache conditions affect the results. The author says the speed gains did not sacrifice output quality, but the results have not yet been repeatedly validated across more hardware, prompts, and models. Nor can Flint's superiority over the Qwen3.6:35-a3B model mentioned by the tester be established from personal experience alone.
Moreover, 64.6 tokens per second does not mean every task will run faster. The model requires 30GB of memory, and its first run may be slower because of insufficient caching. Before deployment, most enterprises still need to calculate memory requirements, concurrency, response stability, and total cost.
Impact on regular people
For enterprise IT: Scheduling optimizations in inference engines may extend the useful life of existing GPUs, but capacity, stability, and total cost must still be tested before adoption in production.
For individual professionals: Larger models can process documents and code more privately, but “it runs” does not mean “it works well.” Ordinary computers still require trade-offs among quantization, speed, and quality.
For the consumer market: Local models could reduce reliance on expensive cloud services. However, a setup combining 12GB of VRAM with 30GB of system memory also shows that low-cost personal deployments have not yet become widely accessible.