Top Agents often score below 50% on real-world benchmarks. That number shows AI assistants can already demonstrate complex workflows, yet still fail to complete roughly half of real-world tasks reliably.
What this is
Unlike ordinary chatbots, Agents—AI systems capable of autonomously decomposing tasks, invoking tools, and performing multi-step operations—must handle end-to-end work. Booking a ticket, for example, may involve searching, comparing prices, filling out forms, and making payment; a failure at any step can invalidate the result.
Agent evaluation, which tests an AI system’s execution process and outcomes through real tasks, therefore needs to examine at least four things: whether the task was completed, whether tools and parameters were correct, whether key identities and permissions were verified, and whether execution time and invocation costs remained under control.
Automated evaluation is also beginning to use an “LLM as judge,” allowing another large model to review execution logs. But accuracy alone does not establish that a judge is effective. If it passes every sample, then even though its agreement rate with human judgments reaches 81%, it has no meaningful ability to identify failures.
Industry view
We believe Agent evaluation is shifting from comparing model capabilities to assessing reliability and unit-task cost in production environments. Enterprises care about whether failures can be detected, errors can be contained, and each completed task has a clear cost—not whether a demo succeeds once.
The risks are equally clear: an automated judge may be biased or make mistakes, and a correct outcome does not necessarily mean a safe process. For refunds, payments, and sensitive data, relying solely on model self-evaluation may amplify privilege misuse or silent errors. The more prudent approach is to validate the judge against human-labeled samples first, then gradually expand the scope of automation.
Impact on regular people
For enterprise IT: When procuring or building Agents, require vendors to provide success rates in real scenarios, error types, and invocation costs. Make identity verification, access control, and human confirmation prerequisites for deployment.
For individual professionals: Off-the-shelf assistants are better suited initially to low-risk, reviewable work. For expenses, procurement, contracts, and similar operations, retain human confirmation so that errors hidden behind work that merely “looks complete” do not slip through.
For the consumer market: More services will be able to place orders, issue refunds, and make reservations on users’ behalf. Consumers will increasingly care about failure compensation, audit trails, and fee transparency rather than comparing model parameters alone.