ClawBench tested 8 Agents on everyday online tasks. Claude, the strongest performer, completed only 33.3% of tasks, marking a shift from merely “being able to respond” to “having work verified.” The key question is no longer whether an Agent can speak, but whether it can reliably get the job done.
What this is
Agent evaluation—the systematic assessment of whether an AI assistant can reliably complete tasks—typically has three layers: end-to-end evaluation checks whether the task was completed; process evaluation examines whether planning, tool use, and reflection were sound; capability evaluation tests individual abilities such as memory and collaboration in isolation.
Task success rate remains the most important metric, but examining only the final answer is not enough. In a ticket-booking task, a system saying “Order placed” does not count as success unless the order is confirmed to have been created. Process traces help assess efficiency, incorrect tool calls, and the number of human interventions required.
The article proposes an 11-dimension framework covering task success, output quality, planning and reasoning, tool use, autonomy, safety alignment, efficiency, long-term memory, and multi-Agent collaboration, among other areas. Its core value is not to create a longer scorecard, but to prevent “false success.”
Industry view
We believe this framework will shift Agent competition from model demos to engineering delivery. Enterprise buyers will care more about real-world task completion rates, invocation costs, and fallback mechanisms after failures than about a single impressive demo.
That said, 33.3% does not define the industry baseline. ClawBench’s task coverage, difficulty, and operating conditions do not reflect every enterprise environment, while end-to-end tests cannot fully capture safety or long-term consistency. A more realistic risk is that, once evaluation metrics become fixed, vendors may optimize for leaderboards rather than solving problems in actual workflows.
Evaluation itself is also costly. Complex tasks require humans to verify system states, process scoring depends on model- or rule-based judgments, and test environments may differ from real business operations. Public leaderboards are therefore better treated as procurement leads, not substitutes for an organization’s own acceptance testing.
Impact on regular people
For enterprise IT: As companies introduce Agents, budget priorities will shift from model calls toward permissions, auditing, testing, and failure recovery. Systems that handle orders, emails, or files must first define what counts as genuine task completion.
For individual work: Office assistants will increasingly emphasize delivery rather than advice—for example, creating calendar events, organizing files, and following up on tasks directly. However, actions involving sending, deleting, or paying will still require human confirmation at critical steps.
For the consumer market: Consumer-facing Agents are unlikely to take over complex affairs in the near term. More likely to gain adoption are single-purpose assistants with clear boundaries and reversible failures, such as tools for organizing travel itineraries and comparing shopping prices.