The per-Token (the smallest unit a model processes) price across major LLMs has fallen over 90% in two years — one of the most underappreciated facts in the AI industry.
In conversations with readers, we've found that most people know AI has gotten cheaper, but not why. To answer that, you have to open up the "inference" pipeline of large models.
What This Is
You ask AI a question, and it walks through this chain: your text is sliced into Tokens (the smallest "bricks" the model can read) → each Token becomes a vector Embedding (turning words into computable arrays) → fed into a Transformer (a "language computing network") → processed through the Attention mechanism (deciding which prior words the current word should "look at") → answers emerge one Token at a time.
Three critical terms: Q/K/V are the three dimensions Attention uses to retrieve information (Query, Key, Value); KV Cache stores already-computed K/V to avoid recomputation.
Why does KV Cache matter? Because LLMs generate token by token. The longer the answer and the more conversation turns, the larger the cached intermediate state. This directly determines how cheap and fast one inference call is.
Industry View
The camp in favor argues KV Cache is the core hero behind the past two years of LLM price drops. Anthropic, OpenAI, and ByteDance's Doubao are all grinding on this at the engineering level — one of the main reasons Token prices fell over 90% in two years.
But the opposing view deserves equal airtime. A former Google engineer posted bluntly on X: squeezing inference cost alone is "squeezing oil from the gas tank" — the real bottleneck is training cost and high-quality data. If we only make existing models cheaper without solving capability, the so-called "inference dividend" will peak within three years. Cheap doesn't mean useful.
Impact on Regular People
For enterprise IT: When selecting models, don't just look at "which is the strongest" — calculate "what each call actually costs." For the same question, using a product with caching can be 5–10× cheaper.
For working professionals: Understand the rule "longer context costs more." Stuffing AI with piles of conversation history may produce better results, but your wallet will protest. Learning structured prompting and pruning unnecessary context is a new core skill.
For consumer market: Free AI is becoming more capable, not because it's gotten smarter, but because vendors have crushed inference costs enough to subsidize it. But there's no free lunch — heavy-use features will continue tilting toward paid tiers.