In a blog post this week, Google Research raised a question the AI industry would rather not answer: as AI models climb higher on benchmarks (standardized tests measuring model capability), are they actually getting better at doing the work?
What This Is
Google's research team points out a structural bias in how the industry evaluates AI. When a model scores high on an exam, buyers default to assuming it's "capable" — but capability (the ability to answer questions) and utility (the ability to do work well) are two different things. A model that nails 90% of math problems may still fail to fill out an Excel report correctly inside an enterprise.
The blog's core argument: the evaluation system is being "exam-ified." Models are now optimized to chase scores, not solve the concrete problems inside real workflows. This isn't the first time Google has voiced dissatisfaction with evaluation methodology, but the voice comes from the research division, not sales — and that carries different weight.
Industry View
Supporters see Google as puncturing an emperor-with-no-clothes situation. Most AI vendor launches use leaderboard climbing as their sales pitch, but customers consistently find significant gaps when deploying. A product lead at a major Chinese tech firm told us privately: "Internally we barely trust external leaderboards anymore. We run our own benchmarks against actual business scenarios."
The opposing view deserves airtime too. Multiple researchers caution that benchmarks aren't useless — there simply isn't a better alternative yet. Dismissing them wholesale would leave enterprise AI procurement dependent on subjective demos and vendor self-promotion, driving decision costs even higher. One AI investor put it more sharply: "Google itself leans heavily on benchmarks to market Gemini. Take that criticism with a discount."
Impact on Regular People
For enterprise IT: Don't pick AI agents based on vendor leaderboard scores alone. Run small pilots on your own real business data, or you'll spend big money on an "exam-cramming machine."
For individual professionals: AI tools that promise to boost productivity by "X times" almost certainly don't deliver that out of the box. Worth spending a week running them through your core workflow before deciding whether to renew.
For consumer markets: The same logic applies to every AI product. The gap between launch demos and real experience will be the dominant complaint zone for the next year.