This week, a small open-source project called crewai-pse is making the rounds in the developer community — it uses 30 lines of regex (a text pattern-matching tool) to have one program check whether an AI-written article contains fabricated class names or file paths. This matters because it hits a question the industry has been avoiding: having AI audit AI is a logical dead loop.
What this is
The AI industry is currently enamored with "Multi-Agent collaboration" (an architecture where multiple AI roles divide the work). For writing an article, for example, planning, drafting, and review are each handled by a separate AI step. It sounds clearly divided — but the review step is itself a large language model (LLM). It audits another LLM's output, like letting a defendant pass judgment on their own testimony.
crewai-pse's solution is old-school: stop asking AI to do quality control. Instead, use regex to extract every backtick-wrapped code identifier in the article, then run grep across the disk to see whether the file actually exists. If it doesn't, flag it red and let a human fix it. Regex matching is deterministic — not found means not found. LLM evaluation is probabilistic: it will "feel" that a class name exists simply because it has seen similar naming patterns in its training data.
Industry view
Supporters say this echoes an old software engineering principle: when deterministic code can verify something, don't hand it to a probabilistic system. This isn't regression — it's getting clear-eyed about what AI cannot do.
The criticism holds too. This approach catches code references but misses fabricated data, invented people, and manufactured historical events — the real epicenter of AI hallucination. One commenter put it bluntly: "Using regex to validate AI output is like doing a modern audit with an abacus — usable, but nowhere near enough."
More realistically: most enterprises haven't even built the "AI-generates, AI-evaluates" pipeline yet, so debating replacements is premature. crewai-pse's real value isn't replicability — it's a reminder that AI system reliability cannot be solved by stacking more AI.
Impact on regular people
For enterprise IT and data teams: worth asking — in your AI document generation and reporting systems, who is handling the last-mile "quality check"? If it's another AI, think through who is on the hook when it fails.
For working professionals: proposals drafted by ChatGPT or Wenxin Yiyan — key figures, client names, project timelines — regex won't save you there. Only human eyes can. The more confident the AI sounds, the slower you should go.
For the consumer market: future AI assistants will "bullshit more fluently," and users need to build basic judgment — whenever an answer sounds too smooth and too complete, the right response is one more question: "Is this actually true?"