A developer pruned Google’s Gemma 4 26B-A4B to 15% of its original size while retaining 92% agreement with the original model on a single classification task, and reducing memory use to roughly 3.8GB. The result matters: model deployment is beginning to shift from trying to do everything toward concentrating limited capabilities on a few high-value tasks.
What this is
Gemma 4 26B-A4B uses a Mixture-of-Experts (MoE) architecture, in which the model dynamically selects a subset of parameters for each input. The original model can call on 32 experts per layer. The developer ultimately fixed the model to eight experts per layer, reducing its parameter count from about 26 billion to 3.81 billion and removing the routing module responsible for selecting experts.
The pruned model performed close to the original on a classification task designed by the author, achieving 90% agreement with the smaller 12B model. On a Mac, it generated 79 tokens per second—about twice the speed of the 12B version—while using roughly half the memory. Knowledge-intensive tasks, however, degraded markedly because the relevant parameters had been removed. This is not a general-purpose model.
The author also proposed “block-local training”: rather than training the final answer directly, it teaches the pruned model to preserve the original model’s processing process. The method is said to enable training on personal devices with about 5GB of memory, but it has not yet been validated on actual user devices.
Industry view
We believe this experiment sends a clear signal: the commercial value of on-device models is more likely to come from specialization than from a general-capability arms race. Fixing a model to a limited set of tasks such as content moderation, intent recognition or on-device command processing can reduce memory use, energy consumption and response time. It can also lower the cost of enterprises buying large-scale computing resources for a handful of tasks.
The risks are equally clear. The 92% agreement rate comes from the author’s single task and self-selected data, so it cannot be generalized to other scenarios. Pruning is time-consuming and requires task data and test-set design. Removed knowledge may impair performance on complex questions, while on-device training remains at the developer-validation stage. The author also acknowledges that the method for deciding which experts to retain still has room for improvement.
This is therefore not proof that most models can lose 85% of their weights. It is an engineering sample worth reproducing. It suggests that model compression still has substantial headroom, but remains some distance away from low-cost, standardized productization.
Impact on regular people
For enterprise IT: Teams running internal AI assistants continuously should consider evaluating specialized small models first. Doing so can reduce server, memory and energy costs while limiting the need to upload sensitive data to the cloud.
For working professionals: Local tools for classification, summarization and document processing may become faster and more resource-efficient, but selection should be tested against specific tasks rather than judged by a single accuracy score.
For the consumer market: Phones, computers and wearables will be better able to run AI features around the clock. Device prices and subscription fees may fall, but specialized experiences will mature before all-purpose assistants do.