I saw AI miscalculate a bill—and realized knowing answers isn’t the same as doing the job

Last Thursday night, I asked an AI assistant to handle a simulated contract. It could explain tax rules in conversation, but placed the “goodwill” line in the wrong section of an email. The resulting spreadsheet deducted $23,000 too little. I stared at the file, too nervous to send it to the client. I got stuck on a lesson I’d ignored before: if a model can explain an answer, that doesn’t mean it can complete work reliably.

This time, Latent Space invited Handshake AI founder Garrett Lord and his team to explain how they build training data for frontier AI labs. In plain terms, they give AI practice problems that resemble real work, then inspect the process rather than grading only the final answer. Finance scenarios simulate bank emails, contracts, and spreadsheets. Biology scenarios ask models to identify which two of nine brain tumor slices show the clearest vascular necrosis. Coding scenarios reveal which files the model read and which commands it changed.

The name that made the scenario real was Nick Magazine. In a 2025 podcast demo, a biology researcher selected slices H and I, but the model also chose G. He pointed at the tissue images and explained live what had been missed. This kind of “process autopsy” matters because the final response alone often hides the fact that the AI never checked the crucial material.

Replication cost: AI data work isn’t something you can start by buying software

According to public information, Handshake reached about $1 billion in annualized revenue in roughly a year—a number I can’t replicate by following a tutorial. For a smaller, custom practice-task project, a starting budget can run into the tens of thousands of dollars, while building the first version myself would take roughly one to three months. The technical barrier is twofold: engineering skills to inspect an AI workflow and domain experts who understand finance, research, or programming. The first step isn’t registering an account. It’s opening Handshake’s website, contacting the team, and asking whether they offer a small pilot.

If I don’t yet have the experts or budget, I might use task-recording software or ask colleagues to record their own workflows, then turn those recordings into cases with scoring criteria. If I don’t test it now, that’s fine too; this business is still a long way from what most side-project operators can reach.

At each stage, I’d start with the lightest step

If I’m just starting out, I’d first sell a “practice-task pack for one role.” I’d interview a legal assistant, analyst, or researcher and turn their work into 20 cases with checkpoints. I wouldn’t buy servers first.

If I already have one or two clients, I’d break the quote into task design, process annotation, and quality review, then begin with one fixed-scenario pilot. Being able to understand the task details matters more than writing complex code.

If I’m scaling, I’d prioritize expert review and a traceable inspection system over adding more tasks. The core lesson from Handshake is that customers ultimately pay for a reliable process, not a longer chat transcript.