On an AMD RX 9070 XT with only 16GB of VRAM, MoE models with a 128K context still achieved prefill speeds of 1,168 to 1,246 tokens per second and generation speeds of 57.3 to 71.6 tokens per second. This result shows that the barrier to running local large models is falling, but for now it is more an engineering breakthrough under specific configurations than a guarantee of general performance.
What this is
The core of this test was MoE expert offloading. MoE (mixture-of-experts) models activate only a subset of their parameters for each token, reducing computation. Offloading schedules those parameters between VRAM and system memory, while quantization, expert caching, and speculative decoding (using a smaller model to generate candidates in advance and a main model to verify them) further improve speed.
The developer used engines including ExLlamaV3 and Polystrata to test Ornith 1.5 35B-A3B, Qwen 3.6-35B-A3B, and Qwen 3.8-Flash-Next Coder at a 128K context. The tests used roughly 4-bit quantization, a Q8 KV cache, and speculative decoding methods such as DFlash2. Results varied significantly across models and engines, showing that actual speed cannot be judged solely by model size.
Industry view
We believe the most important takeaway is that the bottleneck for local models is shifting from “whether they can run at all” to “how to allocate memory and optimize scheduling.” Previously, obtaining enough context capacity on a 16GB GPU usually required trade-offs between VRAM capacity and quantization precision. Expert offloading and hot-cache techniques offer another path to preserving larger contexts.
The risks are equally clear. First, this was an individual test, and the results depend on the benchmark scripts, quantized builds, and cache settings; the author also acknowledged inconsistent performance across some models and engines. Second, the test covered only a small number of models and a single GPU, which is not enough to prove that consumer hardware can broadly handle complex Agent workloads. Here, Agents (AI systems that autonomously plan and call tools to complete tasks) still require long contexts and reliable tool invocation, so generation speeds of dozens of tokens per second cannot be equated directly with full production readiness.
We therefore view this more as a sign of continued maturation in local inference engineering than as evidence that consumer hardware has comprehensively overtaken professional cloud clusters.
Impact on regular people
For enterprise IT: For enterprise IT teams handling internal code, knowledge bases, and documents, local deployment can further reduce dependence on cloud compute. However, 16GB-VRAM solutions still require careful model selection, quantization, and service operations. In the short term, they are better suited to technical-team pilots than to replacing existing data centers outright.
For individual professionals: For individual professionals, local models are more attractive in terms of privacy, offline use, and fixed costs, particularly for code assistance and private-document analysis. But they impose substantial demands on hardware, model configuration, and troubleshooting, and everyday users may not need to take on that complexity for routine writing and question-answering tasks.
For the consumer market: For the consumer market, if this approach becomes more widely adopted, GPU and system manufacturers may turn “large-context local inference” into a new selling point. At the same time, the interaction among model files, memory capacity, and inference software will shape the actual user experience rather than nominal GPU compute power alone.