RL Changes Just 1-3% of Output — Reasoning Model Training May Be 1000x Overpriced

What This Is

This week, a viral paper on Reddit's r/LocalLLaMA dropped a counter-intuitive finding: when training reasoning models with reinforcement learning (RL—a training method where the AI self-iterates by scoring itself for correct and wrong answers), only 1-3% of the final output's tokens actually change. The bolder claim: drop RL entirely, use plain supervised learning, and you can still replicate similar reasoning capabilities—at roughly 1/1000 the compute cost.

A key detail worth flagging: "1-3%" refers to the proportion of changes at the token level (tokens are the smallest units a model uses to read and generate text), not "1-3% capability difference." So what the paper is really challenging is this—the chain-of-thought abilities (where the model explicitly writes out step-by-step reasoning) that RL seems to unlock may already be baked into the base model. RL just flips the trigger switch more visibly.

Industry View

The biggest casualties are the players betting on the RL route: OpenAI's o1/o3, DeepSeek-R1, Qwen, and Zhipu's GLM all market RL as a core selling point. Supporters argue this actually explains why open-source models keep getting stronger—the base model has hit a critical threshold, and RL is just the catalyst.

But the pushback is sharp. The paper only validated on specific benchmarks (curated test sets) and small models. One commenter put it bluntly: "RL changes little on math problems, but on agent tasks (AI that can autonomously operate tools to complete jobs) it's the core." Others push back: if RL doesn't matter, why are all frontier labs still doubling down?

Our take: no one will stop RL training in the short term, but this will become a perpetually cited "cost-saving argument"—especially in enterprise self-built model scenarios. Who wants to pay 1000x more for the same result?

Impact on Regular People

- For enterprise IT: the budget logic for self-built reasoning models needs rewriting. If "ordinary training + a little RL" delivers 90% of the effect, the sales pitch that "you must use the full-RL model" stops holding up.
- For working professionals: the marginal cost of reasoning keeps dropping, which means tasks like data analysis and contract review will show up in tools at lower prices—not stuck at premium pricing forever.
- For consumers: reasoning model API prices will keep falling. DeepSeek already cut prices earlier this year; if more companies adopt this paper's findings, the next wave of price cuts could land around Q3.