The two most-cited AI model evaluation platforms — Artificial Analysis and LMArena (formerly Chatbot Arena) — delivered opposite verdicts this week on the same pair of open-source models. AA ranks Alibaba's Qwen3 27B ahead of Google's Gemma4 31B; LMArena places Gemma4 nearly 20 spots above Qwen3. For developers planning to run open-source models on local machines or servers, this means "pick a model by leaderboard" is effectively broken.
What this is
The story began when a developer on Reddit's r/LocalLLaMA (a community for AI model enthusiasts running models locally) noticed that two similarly-sized open-source models — Google's Gemma4 31B and Alibaba's Qwen3 27B — landed at nearly reversed positions on the two leading benchmarks. AA's "Intelligence Index" shows Qwen winning across the board; LMArena's human blind-vote rankings place Gemma near the top, about 20 positions higher. Two methodologies — one running academic test sets automatically, the other relying on real human blind evaluation — produced entirely opposite answers.
Industry view
The local community leans toward Qwen, viewing it as more "persistent" on reasoning tasks, though it pays for this by overthinking simple problems. Gemma isn't dismissed either — it's still considered a competent model. The mainstream explanation: the Qwen team has likely optimized specifically for AA-style academic benchmarks, while the Gemma team prioritizes Arena-style "real user voting." The two optimization targets are inherently incompatible.
But there's pushback: some argue both leaderboards are vulnerable to "score gaming" — vendors can run their own Arena votes or train on data matching AA's test style. So "leaderboard divergence" may not mean "two truths" — more likely "two exam-cheating strategies." Our judgment: when the gap between two benchmarks is this large, developers should downgrade both leaderboards by one tier and run actual business-data tests rather than trusting any single number.
Impact on regular people
For enterprise IT: don't make open-source model selection decisions from a single leaderboard. Review both human evaluation and automated scores side by side, and prepare a small "business question set" to test for a week before committing.
For working professionals: if you're using closed-source products like ChatGPT, ERNIE, or Doubao, what leaderboards their vendors optimize for barely matters to you — each vendor charts its own path, and what you care about is "which product actually works."
For the consumer market: this incident will reinforce the perception that "AI models can't be directly compared." For the coming year, treat any ad claiming "Model X beats Model Y" with a heavy discount.