I almost renewed last week because of an ad line
Last week I scrolled past a headline: an AI tool claiming it now handles 26% of its own internal R&D. Heart racing, finger hovering over the renew button. Then a friend pulled me back — that stat came straight from the AI company itself, never independently verified.
For us running small shops, leaning on AI to boost output, the scariest thing is vendor self-praise: when your tool says it got smarter, who actually decides that?
AI companies want regulation, but they hold the evidence
A reporter surfaced this recently. Anthropic claims Claude now drives 26% of its internal R&D. OpenAI says its AI hits "autonomous research intern" level. Sounds scary — but the ones publishing the numbers are the vendors themselves.
It's not that they shouldn't brag. The issue is: AI firms want oversight, yet they control the receipts. We paying users end up believing blind.
Concrete scene: last month Xiaozhao (a Beijing-based Xiaohongshu consultant) told me he spotted an AI writing tool claim "beats 90% of peers," and bought annual immediately. After using it, he found the "upgrade" was just interface copy changes. Textbook case of "vendor deals the cards, reader keeps the score."
What you can do right now: three things
Money: 0. Time: 1 hour. Tech barrier: a browser. First step: open your most-used AI tool this month, run your most common task (say, a product blurb), then save the output and compare it to the old version. Don't just stare at vendor changelogs.
I got stuck here too — saw a major model drop a new version, rushed to grab membership, and it actually regressed on my most-used scenario. Since then my rule: any AI tool update, test on old tasks for three days before pulling out the card.
Advice by stage
Just starting (barely using AI): skipping this is fine. Next time you're about to pay, come back and re-read.
Got 1-2 steady clients: I'd suggest setting up a "small exam" (say, a fixed client email template), run it once a month, archive and compare. Keep the proof in your own pocket.
Scaling (team of 3+): build a tool ledger. Three lines per tool: purchase date, use case, last test day. Vendor hype isn't scary — scary is having no ledger of your own.