Alibaba Cloud’s team ran a hands-on evaluation using 15,788 scored cases, 12 models, and 10 real incidents. Specialized decision models outperformed general-purpose models on most of five types of operational judgment, but their gains varied significantly depending on where they were integrated into the full incident investigation. The next hurdle for Agents—AI systems that autonomously call tools and complete tasks step by step—is not better answer generation, but more accurate decisions about what to do next.
What this is
The research focuses on AgenticOps: bringing AI into incident investigation, inspection, diagnosis, and remediation. Instead of sending every problem to the same large model, this approach uses smaller, specialized “decision models” to select targets, assess evidence, choose tools, organize context, and verify results.
In one RDS slow-query incident, the Agent took 10.16 minutes in total, invoked the diagnostic model 123 times, and earned a score of 94. When expanded to 10 incidents, the models took 214.3 minutes cumulatively. Target and data selection accounted for 54.7% of that time, while context organization accounted for 19.2%. The bottleneck is therefore not only model inference, but also how information moves through the workflow and how tasks continue.
Across 1,745 shared single-question inputs, the team’s proprietary Pilot model improved accuracy by 2.52 percentage points and cut P95 response time by 80.3%. When low-confidence cases were passed to the general-purpose Qwen3.8 Flash, 38 of 55 model-task combinations outperformed the stronger of the two individual models.
Industry view
Specialized models are more than small-model cost substitutes. They are faster on some judgments and better at choosing among candidate targets, tools, and evidence. This provides a basis for layered architectures: general-purpose large models handle complex reasoning, while smaller models execute high-frequency, low-tolerance scheduling actions.
But gains on individual steps do not consistently translate into benefits across complete tasks. On the same set of 10 incidents, placing Decision 1 at the task-routing position improved scores by 6.40 points; using Clef 27B for progress judgment improved scores by 7.20 points; and using Pilot for root-cause verification improved scores by 2.40 points while cutting execution time by 4.0%. Different models excel in different positions, so the results do not yet justify claiming that any one model can take over an Agent’s entire “decision layer.”
The risks are equally clear: incorrect routing can amplify costs, while missing context can reduce final quality. The research sample still includes only 10 complete RCA (root-cause analysis) cases, leaving it far short of long-term production validation. Specialized models save invocation overhead, but that does not mean responsibility for incident decisions can be outsourced to them.
Impact on regular people
For enterprise IT: Companies do not need to assign the most expensive large model to every Agent step. They can tier models by task risk, but should first validate the full workflow using real tickets and retain a mechanism for human escalation before deployment.
For individual professionals: More AI tools will soon select data sources, break down tasks, and advance workflows on their own. Workplace competitiveness will gradually shift from “writing good prompts” to defining objectives, supplying evidence, and judging whether an AI’s decisions are trustworthy.
For the consumer market: As decision models become faster and cheaper, Agents for customer support, device maintenance, and software troubleshooting will become more common. Users will see systems taking action automatically, making responsibility boundaries, explainability, and mechanisms for reversing decisions more important than model names.