This week, a post titled "This benchmark is getting out of hand" on Reddit's LocalLLaMA subreddit surged to the top of the board, with community users collectively expressing the same judgment: the AI benchmark race is spiraling out of control. We've observed a specific trend: since 2024, mainstream evaluation leaderboards have expanded from a dozen—MMLU, HELM, and others—to over a hundred specialized leaderboards covering math, code, reasoning, and multimodal tasks; nearly every major model company now attaches an "we achieved SOTA (state-of-the-art) on XX benchmark" report when releasing a new model.
What This Is
Benchmarks were originally meant to be standardized tools for measuring model capability, serving as a unified yardstick. But over the past year, the explosion in leaderboard numbers and frequent "leaderboard gaming" by vendors has triggered widespread skepticism among community developers. We sorted through discussions in the local deployment circle (the LocalLLaMA user base), and the focus centers on three things: whether benchmarks still reflect real capability, whether enterprises using benchmarks for vendor selection is still reasonable, and whether consumers can still trust the scores advertised in product marketing.
Industry View
Community criticism focuses on three points: first, training data contamination—benchmark questions may have entered the training set directly, meaning the model is essentially "memorizing the answers"; second, overfitting—vendors fine-tune specifically for evaluation items (fine-tuning, i.e., continuing training on specific data), so scores look good but generalization ability (the ability to perform in new scenarios) may not improve; third, metrics disconnected from real-world scenarios—high scores don't mean solving actual user problems.
But some practitioners hold reservations. A model evaluation team lead told us: "Benchmarks aren't useless—they just shouldn't be the only one or two used. Combining multiple leaderboards with manual blind testing (evaluators scoring without knowing the model's identity) and real user feedback gets closer to true capability." We also see a risk: as capability gaps between models narrow, "benchmark gaming" will become the marketing tactic with the highest return on investment, further diluting the meaning of leaderboards themselves.
Impact on Regular People
For enterprise IT: If you only look at vendor-provided "benchmark scores" during selection, you're handing decision-making authority to their marketing department. We recommend requiring vendors to run tests on your own business data.
For individual careers: The increasingly common "AI skills assessment" scores in hiring are similarly at risk of contamination from exam-oriented training; actual project outputs are more worth examining.
For the consumer market: The "benchmark-leading" claims advertised for phone assistants, smart speakers, and similar products have limited short-term reference value. In the long run, this will push the entire industry to pull evaluation standards back to real scenarios—but that will take at least a year or two.