What this is
An inclusionAI engineer this week tested the company's Ling-3.0-flash model — 124 billion parameters, Q5 quantization (a compression format that roughly halves model size) — on an NVIDIA DGX Spark desktop. After typing a 33-character prompt, he walked away from the machine; the model then continuously emitted 15,128 tokens over roughly 7 minutes.
The speed is the headline: at token 2,793 it decoded at 35.62 tok/s; at token 15,062 it was still 35.68 tok/s. The KV cache (the intermediate state a model keeps to remember what it has already generated) accumulated across those 12,000 in-between tokens — and the speed didn't budge.
Industry view
The optimistic read: stable long-output throughput means local inference (running the model on your own hardware instead of calling a cloud API) is starting to hold up under real workloads. KV cache not degrading decode speed is a hard engineering signal.
But we'd flag the catch: the dashboard frontend the model produced looks polished — and is full of Math.random() faked data. The prompt explicitly asked for "dummy data," and it dutifully complied, but it never actually read the host machine's real GPU state. There's still a gap between "it runs" and "it's usable." On top of that, the DGX Spark starts in the low thousands of dollars, and producing one frontend in 7 minutes is still too slow for most enterprise scenarios.
Impact on regular people
For enterprise IT: deploying large models on-prem is moving from the "demo" phase into "evaluation." Industries with strict data-residency requirements — finance, healthcare — can now start testing in earnest.
For working professionals: cloud APIs remain the price-performance choice today for AI-assisted coding and long-document work. But within the next 12–18 months, having an AI workstation in the office may stop being science fiction.
For the consumer market: desktop AI devices like the DGX Spark are opening up a new product category — think the "gaming PC" of its era. Not a mass-market product yet, but an early-adopter community is forming.