Open-source developers have increased the decoding speed of large models running on AMD GPUs by roughly 10%–20%, with some code tasks improving by as much as 23.9%. We believe this progress marks local models’ transition from merely running to becoming serious contenders on both performance and cost.
What this is
This update targets Strata EXL3, an open-source quantization branch designed to compress and accelerate local large models. Quantization reduces the space occupied by model parameters, allowing models to run with lower memory and compute requirements. The developer optimized AMD HIP (the GPU compute interface) kernels and memory management, and added CPU-assisted processing of expert weights.
On a Ryzen 9 7950X with 96GB of memory and two Radeon RX 7900 XTX GPUs, code testing using 4.05-bit weights increased from 61.88 tokens per second to 76.67, a 23.9% gain. The developer also added experimental support for MiMo V2.6 Flash, reaching approximately 25–30 tokens per second on the dual-GPU setup.
Industry view
What matters here is not simply faster output, but bringing lower-bit quantized models closer to the quality of higher-bit models. EXL3’s 3.05-bit version is said to approach the quality of common 4-bit quantization while retaining runtime efficiency close to 3-bit quantization.
The risks are equally clear: the results come from a single developer, fixed hardware, and a specific workload, so they cannot be directly extrapolated to every AMD device or model. MiMo support remains experimental, and CPU assistance may actually reduce efficiency in a dual-GPU configuration. This is better suited to technical teams evaluating particular use cases than to serving as a universal performance promise.
We view it more as progress in open-source infrastructure than as evidence that the AMD ecosystem has already caught up with proprietary inference solutions. Software optimization can improve costs, but hardware compatibility, deployment complexity, and reproducibility remain barriers to enterprise adoption.
Impact on regular people
For enterprise IT: Teams handling sensitive data or requiring stable offline services now have another route for evaluating local models. Before moving into production, however, they still need to validate latency, power consumption, model quality, and maintenance costs.
For individual professionals: Faster local inference means that code, documents, and internal materials can be processed with both privacy and responsive performance. The barrier to installation and tuning nevertheless remains higher than with cloud-based chat products.
For the consumer market: GPU and system manufacturers may gradually offer more complete local large-model experiences. A device’s value will no longer be determined solely by compute specifications, but also by whether its software can fully unlock performance.