A Stanford team audited all 89 tasks in Terminal-Bench 2.0 and confirmed that 28 required fixes. After the release of the updated version, the overall accuracy of 19 agents rose by more than 10 percentage points. This shows that supposedly objective agent leaderboards, like software, also deteriorate continuously.
What this is
Terminal-Bench is a benchmark for testing whether agents can complete real-world coding tasks in terminal environments. Agents—AI systems that autonomously invoke tools and complete tasks in stages—can operate command lines, modify code, and run tests. Companies often use their scores to compare models and tools.
Stanford researchers Crystal Zhou, Scott Linderman, and Christopher Ré call this problem “benchmark maintenance debt”: dependency upgrades, interface changes, or websites going offline can make original tasks impossible to reproduce. Inconsistencies between task instructions and scoring rules can also cause correct answers to be marked wrong.
The team had three agents take turns handling evidence collection, auditing, and verification before humans reviewed and applied the fixes. Of the 89 tasks, 28 were confirmed to need work. After Terminal-Bench 2.1 was released, the relevant agents’ average accuracy increased by more than 10 percentage points, although the overall rankings remained relatively stable.
Industry view
We believe this study does not invalidate existing leaderboards. Instead, it fills in one of the most frequently overlooked conditions for using them. The broad hierarchy of model capability still holds, but models with close scores may switch positions depending on the benchmark version, with individual score changes reaching as high as 12 percentage points.
The time budget is even more concerning. Under the default settings, more than half of all failures are timeouts; in some testing frameworks, that figure reaches 68%. Once time limits are relaxed, the roughly 20-percentage-point gap between GPT 5.4 on two testing frameworks almost disappears. This proves that some purported differences between frameworks are actually comparisons of which system runs faster.
The counterargument is equally clear: continuous auditing is costly. Pinning dependency versions, isolating external environments, and conducting manual reviews all require investment. Fixing rules can also change task difficulty, meaning scores from new versions cannot simply be compared horizontally with those from older versions. A more reliable approach is to report the benchmark version, testing framework, and resource budget together, and to run budget-sensitivity tests.
Impact on regular people
For enterprise IT: When purchasing agentic tools, do not rely only on leaderboard scores in vendors’ screenshots. Ask which test version was used, how timeouts were configured, and whether tasks received human review. Internal evaluations should also include dependency locking and regular maintenance, so procurement decisions are not based on distorted tests.
For individual professionals: “Model X outperforms product Y” is still not enough to justify paying for a service or migrating a workflow. Speed, cost, and task-success rates often reflect real-world value more closely than a single ranking, and automated tools still require human review.
For the consumer market: Product claims made after model updates may rely on inflated or incomplete comparisons. As benchmarks gradually disclose more version information, consumers will be better able to determine whether a “score improvement” reflects genuine capability gains or simply a more permissive testing environment.