In Agent Index v1.2’s preview of 86 models, only 26 met the scoring threshold. The top-ranked models were separated by just 1.8 to 2.8 percentage points; based on 500 tasks, that gap is not statistically significant. Model leaderboards are rapidly losing their inherent credibility.
What this is
Agent leaderboards test models against a set of programming or terminal-operation tasks, then package the pass rate into a score that looks precise. But a score is only a point estimate, not a definitive answer. A confidence interval shows the range in which plausible results may fall when the same test is repeated under identical conditions. It adds a necessary layer of uncertainty that “90.4 points” alone cannot provide.
This article provides an approximately 200-line Python script that examines leaderboards in three ways: calculating Wilson score intervals, testing whether the difference between two models is statistically significant, and estimating how many tasks are needed to identify a target performance gap. Wilson intervals are well suited to evaluations with high pass rates and small samples; unlike the common normal approximation, they do not produce an erroneous upper bound above 100%.
More importantly, statistical checks only work if the underlying assumption holds: both the tasks and the scores are genuine. In Terminal-Bench 4, roughly 60% of tasks were identified as flawed. The top three models in Terminal-Bench 2.0 were removed after answer leaks or instances of directly reading test files. Contaminated data and defective tasks cannot be fixed by adding a few more lines of formula.
Industry view
We believe Agent evaluation is shifting from “who has the highest score” to “whether the results are reproducible.” A leaderboard comes closer to usable evidence for procurement decisions only if it publishes the number of tasks, sources of randomness, confidence intervals, code versions, and failure samples. For model vendors, this raises evaluation costs and reduces the commercial value of a one-time champion title.
But this approach has limits. Wilson intervals and significance tests depend on reasonable sampling and task distribution. The same model may perform very differently across tools, prompts, or execution environments, with the resulting variance potentially far exceeding the interval suggested by the number of tasks alone. Statistical significance also does not equal business utility: a model may lead statistically while frequently getting stuck in real enterprise workflows. Conversely, adding more tasks cannot repair a biased task set.
Impact on regular people
For enterprise IT:When purchasing an Agent, do not treat a leaderboard champion as the final selection decision. Require vendors to run blind tests using your own enterprise tasks, and record the success rate, human-intervention rate, and cost of failure.
For individual professionals:“Model X ranks first” is better treated as a prompt to try it than as a reason to restructure your work. First observe its stability with your actual materials, permissions, and processes, then decide whether to incorporate it into your workflow.
For the consumer market:Model providers may continue citing eye-catching scores in marketing, but ordinary users should ask how many tasks were tested, whether the differences are statistically significant, and how failure cases were handled.