With 93.8% of prompts returned directly from disk cache and the GPU recomputing only 49,000 tokens, this result makes one thing clear: the new threshold for local large models is no longer “can it run,” but “can it avoid using expensive compute over and over again.” A developer shared an experiment on r/LocalLLaMA: on a Mac Studio with 96GB of memory, they modified qMLX, introduced a more aggressive cold-cache strategy, and tried to make Qwen3.5 122B support three concurrent sessions.
What this is
This is not a breakthrough in model capability. It is an inference engineering optimization. The author moved more of the KV cache (the intermediate memory a model keeps when processing long context) onto SSD, reducing repeated GPU prefilling work; then paired that with a better eviction policy, so multiple sessions appear to be online at the same time while the underlying system still processes as serially as possible. The author summed it up as “concurrency, not parallelism”: users can open multiple chats at once, but the machine does not actually have to pay a full compute bill for each one.
This matters because it goes straight to the most practical equation in local deployment: if you cannot afford bigger hardware, the only option is to squeeze more out of scheduling, caching, and storage.
Industry view
We should note that this path is not the same as cloud vendors competing by building ever-larger clusters. It is closer to answering the question mid-sized teams actually care about: can their existing machines support more Agents (automated systems that can call tools and break down tasks) or internal assistants without blowing through the budget? For companies doing private deployment and unwilling to send data to the cloud, optimizations like this are more meaningful than “shipping yet another larger model.”
But the objections are equally clear. First, this is still a developer-built setup, not an out-of-the-box product. Second, SSD-based cold caching introduces latency variability, which may suit background tasks but not necessarily strong real-time interaction. Third, the sample so far only covers specific hardware and a 20-minute test, which is still some distance from enterprise-grade reliability. In other words, it proves the direction, not the maturity.
Impact on regular people
For enterprise IT: the procurement logic behind private LLM projects may change. Instead of chasing more expensive GPUs from day one, it may make more sense to first evaluate caching, scheduling, and multi-session utilization.
For individual work: the feasibility of local Agents is rising. Tools like knowledge bases, document assistants, and coding copilots are more likely to run directly beside office PCs in the future, rather than relying entirely on the cloud.
For the consumer market: high-spec Macs, workstations, and large-memory endpoints will gradually expand their pitch from “creative performance” to “local AI hosting capacity.” Both hardware vendors and software vendors will repackage this story.