While 90% of inference costs are often attributed to GPUs, a project to build a lightweight Kimi K3 clone has revealed a thornier problem: fixed-state attention such as KDA breaks traditional caching mechanisms, forcing inference frameworks to redesign how states are saved, restored, and reused.
What this is
This is a record of a CPU-based, lightweight replication of Kimi K3. KDA (Kimi Delta Attention), an attention mechanism that preserves historical information through a recurrent state, does not retain the full KV data for every historical token. Instead, it continuously updates a fixed-size state.
This means that the conventional approach in inference systems—caching historical results by prefix—cannot be used unchanged. After processing text of different lengths, a model leaves behind different states. If a request is split, paused, resumed, or transferred to another server, the corresponding state must be recovered exactly; otherwise, the output may become distorted.
The proposed approaches include SGLang's tree-structured cache with time-based checkpoints, sparse and compressed checkpoints, copy-on-write, and unified memory management. vLLM, meanwhile, separates the logical prefix from the physical state and explores partial hits and segmented retention. The core objective is not to keep states permanently in GPU memory, but to balance speed, memory usage, and correctness.
Industry view
We believe this is worth watching not because running large models on CPUs has enormous commercial potential in itself, but because it makes the infrastructure impact of architectural changes concrete. Once a model adopts hybrid attention and recurrent states, its inference engine can no longer treat every model as the same caching problem.
The risks, however, are equally clear. Source-code adaptation does not mean a solution has been validated at production scale. Too many checkpoints increase overhead, while too few may raise recomputation costs. Sharing states across instances also introduces consistency, scheduling, and fault-recovery challenges. More importantly, the replication project uses scaled-down or approximate components. It demonstrates engineering problems, but its findings cannot be treated directly as performance conclusions for the official Kimi K3.
Another objection is that complex caching may only benefit particular models and workloads. With short requests and conventional concurrency patterns, the additional management overhead may not be offset by memory savings. The final verdict therefore depends on throughput, latency, and memory usage measured under real traffic.
Impact on regular people
For enterprise IT: When replacing models in the future, procurement lists may no longer cover only model capability and API pricing. Teams will also need to assess state caching, elastic scaling, fault recovery, and cross-hardware migration capabilities, all of which affect total cost of ownership.
For individual careers: It is becoming more important to understand the difference between “model quality” and “whether a model can be served reliably.” People with knowledge of model fundamentals, inference optimization, and system troubleshooting will be closer to the core of AI projects than those who know only how to use an off-the-shelf interface.
For the consumer market: If lightweight models can reduce local inference costs, long-document assistants, privacy-sensitive tools, and services on lower-spec devices may become more common. But the real determinants of the experience will still be answer accuracy, response speed, and task completion rates—not whether the model has been successfully replicated.