Last week AI helped me draft a client quote — it was off by 30%

If that email had gone out, my friend Ah-Lin (who runs a brand consulting shop in Shanghai) would've lost the deal. Re-reading it, I caught that the AI had dropped the rush fee entirely. We've all been there — AI does the work, and the output is hit-or-miss.

Shreya and Hamel's approach: set rules first, then mine errors for patterns

I used to assume "looks fine" meant "is fine." That broke when I stumbled onto Shreya and Hamel's podcast (they run an AI evaluation course out of NYC, 4500+ students, teams from OpenAI and Google included). Turns out you need a system to check, not just a glance.

They split evaluation into two flavors:

  • Top-down: You set the rules first — "every bullet 240-330 words," "every quote must include tax."
  • Bottom-up: You dig through AI's actual mistakes for patterns — "it always forgets rush fees."

They built a free Error Discovery skill that runs inside Claude Code (Claude Code is a desktop AI tool — think "Claude, but it can also write code"). First time installing it, I got stuck searching for where the skill lives — turns out you trigger it with a slash command inside the app. Feed it your past client feedback and it auto-groups the errors AI tends to make. For non-coders like us, it's basically a free QA inspector on retainer.

What it takes to replicate this today

  • Money: $0. Claude Code free tier + Error Discovery skill are both free.
  • Time: First setup takes 2-3 hours, mostly walking through each prompt you actually use.
  • Tech barrier: If you can chat with Claude Code, you can do this. No code needed. If you've used ChatGPT or Claude web, you're already there.
  • First step: Download the Claude Code app (macOS / Windows), open it, type / to see the command list, search "Error Discovery" and install. Then feed it the last conversation where AI let you down.

Where are you in this?

  • Just starting, no clients yet: Skip this for now. You probably use AI sparingly — a manual glance is enough. Revisit when you're running AI on 10+ tasks a week.
  • 1-2 steady clients: I'd try it. Drop recent client feedback (like "the tone's off here") into Error Discovery and see if it spots a pattern.
  • Scaling — you're one person doing five jobs: Worth the afternoon to set up. Every output is now your "company face" — manual checking won't keep up.