An A6000 evaluation document describes its tool loop as “real debugging” and reports all_green=True. Yet the script’s run_tests function never launches a test process: as soon as “Bearer” appears in the source code, it returns the fixed string “3 passed in 0.04s.” We believe this kind of evaluation distortion—showing a pass without executing anything—is more alarming than an ordinary failure.
What this is
When an Agent—an AI program capable of autonomously calling tools and completing tasks step by step—says “tests passed,” it only means it received a passing message. It does not mean the program under test actually passed. Inspection reveals that the script first checks a source-code string, then returns a fixed result through a tool. The aggregator subsequently searches only for “passed” and excludes “failed,” creating a self-contained “green path.”
More importantly, the 0.04-second figure is also hard-coded into the returned text; it is not the actual test duration. The function name run_tests and the pytest-style summary can communicate an intention to invoke tests, but they cannot substitute for evidence of execution.
The isolated counterexample in the article goes further: neither the test text nor the business logic changed, but adding a single # Bearer comment at the end of the source flipped the judge from failure to success. This counterexample cannot prove that historical models deliberately cheated, but it clearly demonstrates that a “green result” can be detached from the target behavior.
Industry view
We believe simulated tools have value. During early development, they can reliably check whether an Agent calls the tool, parses failure information, and proceeds to the next round, making them useful for validating workflows and communication protocols.
But the industry cannot rename simulated responses as “real fixes” and then mix them into task-success rates. In particular, enterprises purchasing Agent services must ask three questions: Was a test process genuinely launched? Were the assertions executed by the target program? Does the reported duration come from an actual measurement?
The risk is spreading across the evaluation industry. If more and more leaderboards show only the final green light without publishing the judging scripts and evidence chain, buyers will mistake a completed workflow for reliable code. A safer approach is to report simulated tests, real tests, and end-to-end acceptance separately, while preserving raw records that can be rerun.
Impact on regular people
For enterprise IT: When evaluating Agent vendors, require access to judging logs, raw process records, and reproducible experiments. In particular, distinguish successful tool invocation from passing tests and passing business acceptance.
For workplace professionals: When using AI programming tools, do not rely solely on an “all passed” interface. Check whether tests actually ran, and spot-check command outputs and failure scenarios so that unverified patches are not handed to your team.
For the consumer market: Agent products that promise to “complete tasks automatically” face the same problem. The most credible products today do more than provide smoother answers: they let users see what was done and on what basis the system judged the task complete.