In testing on an RTX 5070 Ti 16GB with 64GB of RAM, tandem.cpp v0.3.0 cut Qwen3.8 prompt processing for a 5.5K-token prompt from 12.4 seconds to 4.1–7.0 seconds, while reducing DeepSeek-V4.1's 15.7K-token prompt processing time from 885 seconds to 57 seconds. We believe this materially improves the practical usability of local large language models for processing long documents, but it has not yet reached the stage of broad, plug-and-play adoption.
What this is
tandem.cpp is an inference engine that coordinates model execution between a GPU and system memory, supporting many of the models covered by llama.cpp. The new version introduces “expert caching,” temporarily placing active experts in VRAM and processing prompts in 4,096-token batches. VRAM is borrowed only during long-prompt processing and released once processing ends.
In this test, prompt processing speeds for Qwen3.8 and DeepSeek-V4.1 improved by roughly 2–6x. The developers also added support for converting DeepSeek-V4.1 weights into the GGUF format for Project Maya, then compared performance with the Maya engine using the same files.
Industry view
Long-prompt processing is becoming a clear competitive benchmark for inference engines, shifting from a phase dominated by waiting for model weights to load. Enterprise knowledge-base retrieval, contract review and codebase analysis all require long context, so time to first token directly affects the experience and cost of local deployments.
The risks are equally clear: these results come from specific hardware, quantization versions and test sets, while generation-speed figures are also affected by strategies for producing short answers. The findings therefore cannot be extrapolated directly to every device. tandem.cpp currently supports only Linux, and AMD VRAM borrowing is not yet available. DeepSeek-V4.1 still reads from SSD when memory is insufficient, while Maya remains faster in the longest-prompt test. In other words, v0.3.0 has improved a major weakness, but it has not eliminated the hardware barriers to local inference.
Impact on regular people
For enterprise IT: Teams that cannot send sensitive data to the cloud but need to process long documents can reassess local model solutions. However, they must still account for the costs of memory, SSD storage and operating-system requirements before deployment.
For individual professionals: Developers and technology enthusiasts can process codebases and long reference materials with less waiting, while nontechnical employees are unlikely to see any obvious product changes for now.
For the consumer market: This progress may encourage local AI applications to reduce cloud calls, but the absence of a Windows release and relatively demanding hardware requirements make mass-market adoption unlikely in the short term.