Back to home

Compare

Comparing: Agent Bottlenecks: Workflow Design Decides Success, Not Model Speed & Agent 落地瓶颈暴露:模型并非慢点,流程设计更决定成败

AEN
AgentGraphRAGRAG·

Agent Bottlenecks: Workflow Design Decides Success, Not Model Speed

In one case, an Agent took 30 seconds to respond. The model accounted for only 3 seconds; the other 25 seconds went to a search tool waiting for a website to retry. This exposes a common industry misconception: an Agent may look slow because the model is slow, and a correct answer does not mean the workflow is sound.

What This Is

Agent engineering connects models to retrieval, code execution, databases, and external APIs so they can complete multistep tasks. Problems usually hide in four areas: whether tool descriptions are clear, whether call paths are overly complex, whether retrieval finds and ranks the right material, and whether timeouts and retries are controlled.

The article introduces two key metrics: Hit Rate@K measures whether the answer appears among the top K retrieval results, while MRR (mean reciprocal rank) measures how high the correct answer ranks. The former answers, “Did we retrieve it?” The latter answers, “Did the model see it first?” If teams evaluate only the final answer, an Agent that checks a balance in one second and one that takes three detours over 30 seconds both count as successes. By production standards, however, they offer very different levels of service.

Industry View

Our judgment is that the threshold for enterprise adoption of Agents is shifting from “Can the model answer?” to “Can the system complete tasks reliably?” When one large tool is split into small tools with single responsibilities and no more than three parameters, the case study shows call accuracy rising from 60% to 95%. Switching retrieval to a hybrid of keyword and vector search raises the hit rate from 70% to 86%. These improvements are closer to production value than simply replacing the model.

Engineering optimization also carries costs. GraphRAG (an approach that first builds a knowledge graph, then performs hierarchical retrieval and summarization) can improve completeness on global questions. It achieved a win rate of 70% to 80% in relevant evaluations, but requires continuous maintenance of the graph index. Splitting tools too finely, meanwhile, increases the probability that the model selects the wrong one. A larger risk is that the article uses a single case to validate empirical claims such as “roughly 80% of errors can be self-corrected within two rounds.” Without sample size or scenario boundaries, these figures cannot be treated as universal industry conclusions.

This is why Trace—logging the time spent and call path across model and tool stages—matters. But more logs do not equal engineering maturity. The real thresholds remain end-to-end success rate, response time, cost, and recoverability from failures.

Impact on Regular People

For enterprise IT: When introducing AI assistants, procurement teams should not compare model scores alone. They should also ask about tool response times, path selection, and error-retry mechanisms. A system may produce the “right answer” while losing its practical value by adding tens of seconds to every task.

For workplace professionals: AI assistants capable of calling multiple tools and executing tasks in stages will gradually enter document organization, data analysis, and internal workflows. At the same time, individuals need to write clearer task instructions and inspect the evidence behind tool calls rather than judging only whether the final text reads smoothly.

For the consumer market: Consumers will feel the speed and stability of AI products more directly, rather than comparing answer length alone. In customer service, shopping, and booking, frequent waits or detours make it difficult to establish long-term usage habits even when the final result is correct.

Source: juejin.cn
BZH
AgentGraphRAGRAG·

Agent 落地瓶颈暴露:模型并非慢点,流程设计更决定成败

一次 Agent 响应耗时 30 秒,其中模型只用了 3 秒,25 秒耗在搜索工具等待网站重试。这个案例揭示了行业常见误区:Agent 看起来慢,不一定是模型慢,答案正确也不代表流程合格。

这是什么

Agent 工程是把模型接入检索、代码执行、数据库和外部接口,让它完成多步骤任务。问题通常藏在四处:工具描述是否清晰、调用路径是否过度复杂、检索是否找到并排好资料,以及超时与重试是否受控。

文章提出两个关键指标:Hit Rate@K 衡量前 K 个检索结果中有没有答案,MRR(正确答案排名的倒数平均值)衡量答案排得是否够靠前。前者回答“捞到了吗”,后者回答“模型先看到吗”。如果只看最终答案,Agent 查一次余额耗时 1 秒或绕三步耗时 30 秒,都会被判为成功;但从上线标准看,两者完全不是同一类服务。

行业怎么看

我们的判断是,企业采用 Agent 的门槛正在从“模型能否回答”转向“系统能否稳定地完成任务”。把一个大工具拆成职责单一、参数不超过三个的小工具,文中案例显示调用准确率从 60% 升至 95%;检索改为关键词与向量混合后,命中率从 70% 升至 86%。这类改进比单纯换模型更接近生产价值。

但工程优化也有代价。GraphRAG(先建立知识图谱,再分层检索和汇总的方案)能改善全局问题的完整性,其在相关评测中的胜率为 70%至80%,却需要持续维护图索引;工具拆得太细,又会增加模型选错工具的概率。更大的风险是,文章用单一案例验证“约 80% 的错误可在两轮内自修”等经验比例,缺少样本规模与场景边界,不能直接当作行业通用结论。

因此,Trace(记录模型、工具各环节耗时和调用路径的日志)很重要,但也不能把日志变多等同于工程成熟。真正的门槛仍是端到端成功率、响应时间、成本和异常可恢复性。

对普通人的影响

对企业 IT:企业引入 AI 助手时,采购重点不应只比较模型分数,还要追问工具响应、路径选择和错误重试机制。系统可能“答案没错”,却因每项任务多耗几十秒而失去实用性。

对个人职场:能调用多个工具、分步骤执行的 AI 助手会逐步进入资料整理、数据分析和内部流程;与此同时,个人需要把任务说明得更明确,并检查其调用依据,而不能只看最终文字是否顺滑。

对消费市场:消费者会更直观地感受到 AI 产品的快慢与稳定性,而非只比较回答篇幅。客服、购物和预订等场景若频繁等待或绕路,即使最终给出正确结果,也很难形成长期使用习惯。

Source: juejin.cn