We noticed a remarkably specific number: by zeroing out less than 0.05% of an LLM’s pretrained weights, the model can match the results of full fine-tuning across reinforcement learning, supervised fine-tuning and policy distillation. In other words, adding new capabilities to a model may not require changing most of its parameters.
What this is
The research introduces Deletion Tuning: a binary mask is trained for the gated projection layers of a Transformer, the model architecture used to process sequential information. Instead of updating all weights, the selected weights are set to zero. Full fine-tuning saves an entirely new set of weights, while the masks produced by this method are hundreds to thousands of times smaller.
It also aims to mitigate “forgetting” in continual learning, where a model’s performance on an old task declines after it learns a new one. In the experiments, the original-task accuracy of Deletion Tuning remained close to that of the base model. Full fine-tuning, LoRA (low-rank adaptation) and random sparse fine-tuning caused declines of as much as 5.4 percentage points.
Industry view
The result is worth praising because it offers a concrete route to reducing training, model storage and update costs. If it can be reproduced in real-world business settings, companies training proprietary models can rely less on expensive full fine-tuning and massive checkpoint files.
The risks are equally clear. The current findings come from a paper and a specific experimental setup; they cannot be assumed to apply to every model, task or deployment environment. Zeroing weights may damage model capabilities that have not been tested, and a modification rate below 0.05% does not mean compute costs fall by the same proportion. The paper was released through a community discussion channel, so independent replication, testing on more models and safety evaluations are still needed.
Impact on regular people
For enterprise IT: Model customization may become lighter, while training and version-management costs could fall. But enterprises still need to validate domain expertise, factual accuracy and safety; they should not assume a model is more reliable simply because fewer parameters were changed.
For individual professionals: Continual learning is pushing companies to turn models into assistants that are increasingly tailored to specific business workflows. Response speed, inference cost and update efficiency may improve, although in the near term, job changes will still depend more on how enterprises use those capabilities.
For the consumer market: Cheaper training and deployment could bring customized models into more software and services. What consumers ultimately experience, however, will still be price, response speed and output quality—not how many weights were deleted.