What This Is
OpenRouter's latest "State of AI" report surfaced a number most people have overlooked: across 100 trillion tokens of real call samples, the average prompt token (the volume of text fed into the model) per request has nearly quadrupled (~4x) compared to early benchmarks. This isn't because users are asking longer questions — it's because Agents (AI programs that autonomously plan, call tools, and complete tasks across multiple steps) must feed the entire conversation history, tool outputs, and sub-task context back to the model on every run.
This week, NVIDIA unveiled two GPU lines — Vera Rubin and Blackwell (graphics processing units, the core chips for AI training and inference). The pitch isn't "faster," but "performance-per-watt" (how much compute you get per unit of electricity consumed). In other words, NVIDIA is now treating it as a given: in the Agent era, the compute bottleneck is no longer peak speed — it's the power bill and thermal dissipation.
Industry View
The optimistic read: hardware upgrades will replay the 2023 "compute price collapse," driving Agent deployment costs off a cliff and letting mid-sized companies run multi-step reasoning. NVIDIA's own blog emphasizes that Blackwell delivers an order-of-magnitude efficiency improvement over the prior generation on identical workloads.
But we hear sober pushback from two directions. First, on the power infrastructure side: U.S. outlets like Data Center Dive have repeatedly reported that some data center campuses in Virginia and Oregon have paused new cabinet hookups due to transformer queues — however power-efficient the GPUs are, the rooms are full. Second, on the model architecture side: some researchers argue that what truly contains Agent costs is "context compression" and "phased caching," not stacking more powerful cards. Anthropic's investment in MCP (Model Context Protocol, a unified protocol for letting models call external tools) is, at its core, doing exactly that.
There's another risk that's easy to overlook: the energy efficiency numbers come from NVIDIA's own testing. In real multi-step Agent workflows, memory bandwidth (the speed at which data moves between GPU and VRAM) is often the tighter bottleneck. Performance-per-watt does not necessarily equal performance-per-dollar.
Impact on Regular People
For enterprise IT: if you're evaluating Agent procurement, don't just watch the API (the call interface vendors expose to customers) unit price — push for the total cost of a "complete task." That figure may have quietly doubled in the past six months.
For individual professionals: if you only use AI to draft weekly reports or build spreadsheets, you won't notice. But once you start letting AI "handle multiple things in sequence" — pulling data, reformatting, sending emails — the underlying call volume is 3-5x that of Q&A-style usage. Expect a fatter end-of-month bill.
For consumer markets: "unlimited usage" plans for smart speakers, AI assistants, and other consumer-facing (C-end) products will almost certainly see price hikes or throttling — vendors themselves can't absorb 4x token consumption per user.