A developer going by Vasili_Sk posted a pointed question on r/LocalLLaMA: why does every LLM inference framework require the full conversation history to be resent to the server on every turn? It looks like an implementation detail, but we think it's likely the invisible black hole behind today's stubbornly high AI costs.
What this is
The post draws an analogy to WeChat: we wouldn't resend three years of chat history every time we send a new message. But that's exactly how today's LLM APIs work — every new input arrives with the entire prior conversation attached, and the server then digs through its cache (the KV cache — the intermediate computation results produced during LLM inference, think of it as the model's "short-term memory") to figure out what it has already processed and what it can reuse.
The result: every request forces the server to match prefixes, look up cache entries, restore breakpoints, avoid cache invalidation, and decide which conversations to keep and which to evict. Every client framework — LangChain, the various agent frameworks — has to reimplement this logic from scratch.
The author's proposal is intuitive: make the inference server behave like a database — create_slot to open a session, append messages to it, query, compact, delete. State lives on the server; the application just consumes it. We note that the community already has parallel discussions under names like "persistent context," "infinite cache," and "server-side context composition."
Industry view
The supportive view is blunt: this is obvious waste at the infrastructure layer. Whoever restructures this layer first will pull ahead on inference cost and latency. A rough estimate: if cache reuse shifts from "recompute every turn" to "persistently maintained server-side," per-request compute and VRAM consumption could drop by an order of magnitude.
But there are sober counterarguments. A common rebuttal is "the ecosystem has already ossified around this — who pays the migration cost?" Every application-layer framework assumes the conversation gets resent every turn. Switching to server-side state means rewriting APIs, breaking compatibility, and tackling complex engineering problems like branching, rollback, and cache invalidation. Another risk is locking sessions to a single vendor: when users switch devices or providers, how does the conversation history migrate? Who owns the data? These are unresolved hard problems.
Worth noting: OpenAI, Anthropic, Alibaba, and ByteDance — none of the major vendors have publicly moved in this direction. It's probably not that they haven't thought of it; the migration cost is too high — especially when your customers are already running tens of billions of requests a day on the old interface.
Impact on regular people
For enterprise IT: over the next one to two years, we expect internal LLM deployment bills to drop 30%–50% thanks to under-the-hood refactors like this. A big chunk of what makes today's AI projects expensive is engineering waste from "redundant inference."
For working professionals: "why is AI so expensive" will shift from a topic for engineers to something managers also need to understand. Judging whether an AI project is worth it can't stop at the demo — we have to look at the inference cost structure and caching strategy underneath.
For consumers: the AI assistants we use will likely "remember more" — not because model parameters get bigger, but because conversation state can be efficiently retained on the server. The "context length limit" we keep hitting today will gradually become a historical footnote.