After working with more than 30 companies over two years, AI consultant Hamel Husain offers a clear judgment: successful AI product teams measure their roadmaps primarily in “experiments,” not “features.” Tool selection is secondary. What ultimately determines product quality is the ability to see errors, validate changes quickly, and iterate continuously.
What this is
This is an AI product development method built around real error analysis: review user conversations one by one, classify recurring problems, create targeted tests for those problems, and then use small changes to test the results. Using this approach, Nurture Boss found that 66% of errors came from inaccurate date interpretation. After the fix, its success rate on date handling rose from 33% to 95%.
The supporting infrastructure does not necessarily require an expensive platform. A team can start with a spreadsheet and a custom data viewer, placing full context, open-ended notes, and quick filters on the same page. It can then use a large language model to help summarize failure types. Even without real users, teams can generate test questions with synthetic data, but those tests must incorporate real business rules and system constraints.
Industry view
We believe this reflects a pragmatic trend: competition in AI products is shifting from lists of models and features toward the speed at which an organization learns. With the same underlying model, the team that identifies errors earlier and knows how to improve them with data is more likely to deliver a consistently reliable experience. Having domain experts write and adjust prompts directly can also reduce information loss as requirements move between business and technical teams.
But this approach is not without risks. Excessive reliance on synthetic data can introduce the generator model’s biases into the evaluation system. Optimizing only a handful of high-frequency errors can also obscure low-frequency, high-cost scenarios. Moreover, Husain’s claim of “10 times faster iteration” lacks a standardized definition and should not be extrapolated to every team. More importantly, error analysis requires sustained human effort for review and labeling. If management continues to prioritize launch speed and feature count alone, this method will likely remain a recommendation on paper rather than become an operating practice.
Impact on regular people
For enterprise IT: Budgets may shift from one-time purchases of complex agent frameworks toward tools for data inspection, evaluation, and labeling. Technical leaders must manage model performance, business rules, and real errors simultaneously rather than simply tracking feature launches.
For careers: The value of AI-related work is moving from “connecting a model” toward identifying problems and validating results. People who understand the business, can judge output quality, and can turn ambiguous requirements into testable scenarios will have more room to collaborate than those who merely chase technology buzzwords.
For the consumer market: Companies using AI products will fix high-frequency mistakes faster, making experiences in scheduling, information retrieval, and customer service more reliable. Consumers should still pay attention to error types and business rules rather than assuming a polished demo means the product is mature.