This week, an article titled "Beyond Recall and the Illusion of Competence" circulated widely across the developer community, posing a sharp question: when we call today's AI models "capable," we often just mean they "have good memory."
What This Is
"Recall" (memorization/reproduction) and "competence" (real capability) represent a classic divide in evaluation. Traditional benchmarks mostly test whether a model can answer a question correctly or match a standard answer — essentially measuring "recitation + matching." But the work enterprises actually want AI to do — reading contracts, querying data, writing reports — has no standard answer; what matters is whether a model can do the right thing amid changing conditions.
The article argues this mismatch creates an "illusion of competence": models look like star students in tests but drop the ball in real workflows. This judgment has been repeatedly verified on the enterprise front line — many Agent projects (systems where AI autonomously completes multi-step tasks) get stuck at deployment. The issue isn't inadequate technology; it's the huge gap between "test-smart, runtime-dumb."
Industry View
Those supporting a "change of evaluation track": multiple researchers in the field — including Andrej Karpathy in several public talks — have pointed out that coding benchmarks have been "gamed" to the point of near-zero differentiation. Models score above 90% on test sets, but error rates rise significantly when placed in real codebases for continuous work over hours. Some leading labs have quietly shifted their recent launches toward showcasing real usage data rather than simply posting test scores.
The opposition / risk voice: we don't need to jump from one extreme to another. Completely abandoning benchmarks would cost the industry comparability — without a uniform yardstick, how do you judge that model A is stronger than model B? A more realistic path is stratification: use benchmarks to track trends in foundational capability, and use real tasks to evaluate deployment capability. But "real-task evaluation" is costly and hard to standardize, with no widely accepted solution today.
Another risk worth flagging: running large-scale real-task tests requires data and engineering investment that big players can afford, while SMEs must continue relying on public benchmarks — this will widen the evaluation gap into a product gap.
Impact on Regular People
For enterprise IT: when selecting AI vendors, don't just look at the "95% accuracy" they advertise — demand real test cases relevant to your business, or you'll end up buying an "exam-type player."
For individual careers: if your role is being disrupted by AI, first assess whether your work is "standard-answer type" (easily replaced) or "open-situation type" (AI still struggles with). The latter is safer in the short term.
For the consumer market: over the coming year you'll see more AI product marketing shift from "X% accuracy" to "real-world performance" — the direction is right, but watch out for old capabilities wrapped in new jargon. Don't get fooled by repackaged concepts a second time.