What this is
This week on Reddit's r/LocalLLaMA, an independent developer published a hands-on note: he ran 25 AI Agents doing "agentic coding" — having AI actually call tools, execute code, and read files — on a single RTX5090 (a consumer GPU, retail price 20,000–30,000 RMB) paired with a top-tier 9800X3D CPU. The result: the GPU sat idle at ~700 tokens/second, while every CPU thread ran at 100%. The conclusion: if you think your Agent is slow because the model is slow, it's actually being strangled by the scheduling system.
He calls this phenomenon the "tool call" bottleneck. When AI stops merely "talking" and has to actually take action — hitting APIs, querying databases, modifying files — the CPU cost of orchestrating those actions far exceeds the cost of generating text. This developer has written his own inference engine and implemented prefill cache reuse; he is someone who has already squeezed a single GPU to its absolute limit.
How the industry sees it
This finding aligns with Andrew Ng's statement last week that "90% of Agent projects stall at deployment": the problem isn't that models aren't good enough — system architecture can't keep up. But the industry isn't unanimous, and we need to lay out the competing views side by side:
- Cloud vendors (Anthropic, OpenAI, etc.) would push back: their solutions already handle tool calls in a dedicated orchestration layer, so users don't feel it
- Traditional IT departments will be alarmed: the Agent bottleneck is in scheduling, not compute, which means the "stack more GPUs" procurement logic may be misaligned
- Developers in the comments point out: this is an inherent limitation of single-machine, single-GPU setups — distributed scheduling can ease it
We think the point worth flagging is this: this developer is no hobbyist. Even at the limit of optimization, a single-machine system still can't handle concurrent multi-Agent workloads — that is the signal worth taking seriously, not "one person who didn't tune well enough."
Impact on regular people
For enterprise IT: when procuring AI compute, the budget ratio for CPU, memory, and orchestration frameworks (how Agents queue up, how they communicate with each other) needs to be rebalanced. GPU count alone can no longer drive the purchase.
For individual careers: don't be fooled by "AI is so fast" demos. Whether Agents can work reliably depends on the engineering behind them, not on how flashy the model is.
For the consumer market: over the next year or two, SaaS vendors will pack Agent features into their products, but stability will likely be uneven — first movers will expose the fragility of their scheduling layer, while fast followers will get a chance to copy what works.