An RTX 3060 12GB GPU ran an 80B model at roughly 64 tokens/s under RustingStrata optimization. The results send a clear signal: competition in local large language models is shifting from parameter count to memory and scheduling efficiency.
What this is
The test covered several Qwen models ranging from small parameter counts to 177B, comparing RustingStrata with llama.cpp, an open-source inference tool for running large language models locally, back to back on the same computer.
The results split by model size. At smaller sizes, llama.cpp delivered faster prefill. Once the model exceeded the GPU's 12GB of VRAM, RustingStrata's advantage emerged. Prefill speeds for the 35B, 80B, and 177B models improved by about 1.5x, while decode speeds reached nearly 2x on some models. Prefill is the speed at which a model reads and processes input context; decode is the speed at which it generates a response token by token.
Industry view
We believe these results are worth watching, but they are not yet a comprehensive performance verdict. The test involved only two runs per configuration, with different models, quantization formats, and parameter settings. The device was also operating under ordinary desktop conditions, with a browser, Discord, and other programs running simultaneously, making the data closer to real-world use than to a laboratory benchmark.
There is another risk. With MTP enabled—speculative parallel decoding, which accelerates generation by using multiple candidate results—RustingStrata reached roughly 160 tokens/s in decoding for the 35B model. The benchmark disabled this feature, however. Actual gains will therefore depend on software configuration, context length, and hardware conditions. Even after manual tuning, llama.cpp reached approximately 2,236 tokens/s in prefill and 68 tokens/s in decoding with the 35B model, showing that the competition remains intense.
Impact on regular people
For enterprise IT: The deployment cost of local models may fall, particularly for organizations with limited VRAM that need to run larger models. But evaluations should not rely on single-GPU benchmarks alone; stability, operational complexity, and data security also matter.
For individual professionals: Running larger models on consumer GPUs means individuals can process longer internal documents, code, and business materials, reducing the need to send sensitive data to cloud services. Faster performance does not, however, mean the user experience automatically reaches cloud-service levels.
For the consumer market: Model quantization and memory scheduling are becoming new sources of product differentiation, further blurring the boundaries among GPU makers, system vendors, and local AI software providers. What ordinary users ultimately buy may be a more complete local-model experience rather than a single hardware specification.