A OneThingCloud OEC device with 3.9 GB of memory and no swap can enable only about 58,000 tokens of context in practice—roughly half of LiquidAI's advertised 128K. The result shows that whether a large model's advertised capabilities can actually be realized depends not on its specification sheet, but on whether the deployment device has enough memory to run it.
The test subject was LiquidAI/LFM2.5-1.2B-Instruct, running locally through llama.cpp. The model supports a 128K context, but long contexts require it to continuously retain intermediate representations of earlier content, namely the KV cache. It does not load the entire conversation into memory at once, but this additional memory requirement grows as the conversation expands.
What This Is
The article first calculates memory requirements from the model configuration: six of its 16 layers use full attention and therefore need to store a KV cache, while the remaining 10 use a convolutional architecture and do not incur that cost. This yields a requirement of 12,288 bytes per token. Tests across six context sizes measured 12,339.5 bytes per token, a variance of about 0.4%.
The real constraint comes from total runtime overhead. The Q4-quantized model file is 697 MiB, but the process's resident memory reaches 1,432.6 MiB. With 2,121 MB of memory available on the device at the time, approximately 688 MiB remains for the KV cache after allowing for the base runtime footprint—enough for only about 58,000 tokens.
Another easily overlooked conclusion is that increasing model weights from 4-bit to 8-bit does not reduce KV cache usage: the cache remains stored in FP16, leaving the per-token cost virtually unchanged. A more effective way to save memory is to reduce the cache requirements of the relevant layers or quantize the KV cache specifically, rather than repeatedly adjusting weight precision.
Industry View
The value of this test is that it turns “supported context length” back into the capacity actually available on a device. Model architecture, quantization format, OS caching, and memory headroom jointly determine the final limit. Looking only at the 128K figure on a model card makes it easy to overestimate what a small edge device can deliver.
But we also need to recognize the risks. First, the findings are based on one 1.2-billion-parameter model, one quantization format, and one device, so they cannot be extrapolated directly to every small model: KV cache requirements can vary widely across architectures. Second, the article uses process resident memory for its estimates. This includes memory-mapped file pages and temporary working copies, so it may not equal the context length the device can reliably sustain. Third, the more memory-constrained the advertised context is, the more deployers need to plan capacity in advance rather than leaving the limit to chance at runtime.
Impact on Regular People
For enterprise IT: When procuring compact AI devices, teams should reserve memory for runtime overhead and the KV cache in addition to computing capacity and model file size. They should also establish fallback plans so that long-running tasks do not trigger an abrupt OS termination of the process.
For individual professionals: Running large models locally is better suited to controlled tasks such as document analysis and text processing. But an advertised long context does not mean the device will always be able to handle an extremely long document, so critical work should not rely on a single local device alone.
For the consumer market: “Running large models on a small box” is becoming increasingly realistic. However, product promotions that show only model parameters and maximum context—without available memory or actual usable length—cannot serve as a reliable basis for purchasing decisions.