This week, a highly upvoted discussion surfaced on Reddit's LocalLLaMA developer community (the gathering place for hardcore players running large models locally): a developer who has been running models for 2-3 years called mainstream AI evaluation leaderboards "useless." He argued that the hardware benchmarking approach completely breaks down when applied to AI — a 9B (9 billion parameters) older model and a 27B newer model may score similarly on paper, yet the gap in actual business performance is enormous. When the hardcore testing community collectively doubts benchmark numbers, enterprise readers should pay attention.
What this is
The core of the discussion is the benchmark mechanism itself (using a standardized question set to score models). In traditional hardware, it's trustworthy — the same chip runs the same test suite, and results are stable and consistent with daily experience. AI models behave differently: tweak the temperature parameter (the setting controlling answer randomness) or rephrase a prompt, and scores swing 10-20 points. AI benchmarks don't measure the model — they measure "the model's specific answer at a particular moment."
For model companies, benchmarks are a marketing tool; for investors, benchmarks are a leaderboard; for enterprise IT, benchmarks are supposed to be a procurement basis. When three parties each take what they need, credibility is the casualty.
Industry view
The original poster's stance is blunt: only the most basic retrieval tests like "needle in a haystack" have any meaning; everything else is barely better than nothing. He runs a Qwen (Alibaba Tongyi Qianwen) 35B plus 27B combo as his daily workhorse and has never switched.
Counterarguments exist. Completely rejecting benchmarks would collapse the evaluation system — enterprise procurement can't compare options, and compliance-heavy environments like hospitals and banks need some form of standardized comparison. The real question isn't whether benchmarks should exist, but who designs the question sets, how to avoid data contamination (test questions leaking into training sets), and how to make scores reflect real production workloads. Many question sets that major model companies submit for evaluation have already been reverse-engineered. Without that context, reading leaderboards will lead you to overestimate certain vendors.
A more subtle risk: once benchmarks devolve into marketing props, enterprises must build their own metrics for real productivity — preparing business samples, running them themselves, comparing them themselves. This pushes costs that evaluation agencies should bear onto procurement teams.
Impact on regular people
For enterprise IT: In the past, you'd glance at a few public leaderboards and place orders. We now recommend preparing at least 20-50 real business questions for blind testing (comparative testing where testers don't know which model they're evaluating).
For individual professionals: When using AI tools daily for emails and reports, don't fully buy the "latest and greatest" rhetoric — lightweight models with reasonable pricing and stable speed are often more practical.
For consumer markets: "AI phone benchmark scores" and "AI PC IQ scores" will proliferate in the short term. Remember this post's warning: good-looking numbers don't equal good experience.