This week llama.cpp — the most popular open-source tool for running LLMs locally — merged a PR that lets the model decide how many tokens to "speculate" in a single pass (the industry calls this MTP, or multi-token prediction). The results are polarizing: code-writing tasks can see up to a 2x speedup, but ordinary prose runs about 3% slower. This is an update that matters to developers yet stays invisible to most people.
What this is
MTP, in short, lets a model output multiple tokens per pass rather than generating them one at a time. It has been one of the main directions in inference acceleration over the past year. The trade-off: guess too many and you get them wrong; guess too few and you waste compute. Previously, users had to manually set a depth parameter (commonly 3 or 6). This PR introduces an "adaptive" mode that lets the system pick depth automatically based on context.
Empirical numbers: dense prose is roughly 3% slower; code generation runs 10–15% faster; reusing code already produced during a "thinking" phase can be 50%+ faster, and in extreme cases approaches a 2x speedup. The recommended configuration is a depth ceiling of 12, with the system dynamically adjusting between 3 and 12.
Industry view
Supporters see this as a marker that the local LLM toolchain is moving toward "less hassle": users used to trial-and-error through a dozen parameters; now a critical one can be handed off to the machine. Developers in the community have already chimed in saying they plan to benchmark it.
The counter-argument is equally valid: a 3% regression is not a small number in production — for a service chewing through hundreds of millions of tokens a day, 3% is a real line item on the compute bill. On top of that, MTP mainly benefits models with native support for it (such as the DeepSeek family); ordinary open-source small models can't take advantage. In short, this is a technology that "pays off only when the hardware and the model are both aligned" — broad utility is limited.
Impact on regular people
For enterprise IT: Teams already running LLM inference on-premises or in a private cloud can theoretically save on tuning labor, and throughput on some code workloads will see visible gains.
For working professionals: Non-technical roles will barely notice; front-line developers will get a smoother experience, but the presence of "AI tools" on your machine won't fundamentally change.
For the consumer market: No near-term spillover to consumer products. This is a low-level accelerator, several layers of abstraction away from the "AI feels faster" experience a user can actually perceive.