Back to home

Compare

Comparing: AI Hallucinations Are Not the Discovery of a New Archaeological Site & AI Hallucinations Are Not the Discovery of a New Archaeological Site

AEN
ai-agentsmayahallucination·

AI Hallucinations Are Not the Discovery of a New Archaeological Site

01 Triggering Event

Google researcher Pietro Miotti said last week that he had used a group of AI agents to identify a possible, previously undocumented Maya site in Mexico.

But the key fact in the original account was that the location had already been discovered the previous year and published in an academic journal. Local communities had also long known about the ruins nearby.

This was not an archaeological discovery. It was a failure to verify a discovery claim. The "villages and regular shapes" identified by the AI agents were, at most, leads worth reviewing; they could not automatically be promoted to a "discovery."

02 What This Really Means

The issue is not whether AI can examine satellite imagery, filter terrain, or batch-flag suspected structures. It is that the current workflow collapses pattern recognition and knowledge confirmation into a single action.

Miotti's agents found lots of villages and regular shapes that looked like structures or houses. That statement is itself restrained: they merely looked like buildings.

The subsequent narrative, however, packaged the result as a potentially unmapped Maya site. What was crossed here was not merely a wording problem but an evidentiary chain:

  • Do the ground coordinates overlap with any known archaeological sites?
  • Could the shapes in the satellite imagery be caused by vegetation, roads, or modern buildings?
  • Do historical records, topography, and local community knowledge corroborate the finding?
  • Has an academic journal already documented the same location?

This is what the episode actually demonstrates: in long-tail domains of professional knowledge, AI has reduced the cost of candidate generation without reducing the cost of adjudication by the same amount.

For developers, this is a concrete lesson in Agent product design. An agent returning a "discovery" does not mean the workflow is complete. It requires provenance, confidence calibration, duplicate retrieval, and human review. Otherwise, spending more tokens simply creates more candidate leads that must be audited.

I have not run Miotti's complete agent stack internally, so I am not identifying the specific step that caused the misattribution. The public reporting is nevertheless sufficient to reject the strong conclusion that "AI discovered an unknown archaeological site."

03 Historical Analogy / Structural Comparison

This resembles the archaeology boom that followed the mass adoption of GPS devices around 2007. Devices enabled more people to find coordinates and anomalies, but academic discoveries were established not by more sensitive sensors but by site documentation, archival comparison, and field verification.

The difference is that AI agents are much better than GPS at generating candidate points in bulk. GPS generally produced relatively sparse anomalous coordinates; an agent can combine satellite imagery, historical maps, and visual models to propose hundreds or thousands of patterns at once.

This corresponds to Stratechery's aggregation theory: value comes not only from model capability, but also from the objects being aggregated, the verification process, and distribution channels. The downside is that aggregation can aggregate errors as well. A system optimized only to "find more places worth examining" will naturally lean toward false positives.

A 2014 AWS analogy is also relevant. Amazon did not make every enterprise system the best in its category. Instead, it aggregated computing capacity, deployment tools, and distribution until it became the default infrastructure. AI archaeology needs a similar layer of trusted infrastructure: coordinate deduplication, journal indexes, map layers, local knowledge, and expert review rather than a single prompt.

In other words, AI can widen the top of the funnel, but the moat will reside in the verification layer rather than the generation layer. Replacing models may be inexpensive; cross-validation data accumulated over years, along with expert-review relationships, is much harder to replicate.

04 What It Means for AI Builders

If I were building an agent product this week, I would immediately tighten the product definition of discovery-oriented outputs.

First, separate action from conclusion. A model may output a candidate site, suspected structure, or lead for review, but the system should not display "discovered" directly. The interface should clearly distinguish a machine-generated hypothesis from a verified finding.

Second, make duplicate detection a default step. For maps, buildings, medical images, legal cases, or security incidents, the system should first check for overlap with authoritative databases and existing publications before allowing the result to enter human review. Without this step, an agent merely produces duplicate answers faster.

Third, require an evidence packet rather than a single confidence score. Every candidate should include the original image region, coordinates, the model's reasoning, possible sources of confusion, relevant literature, and sources of uncertainty. A score can assist prioritization, but it cannot replace auditable evidence.

Fourth, retain a hard human-in-the-loop threshold. In professional domains, AI is best suited to triage, bulk annotation, and anomaly screening. Final naming, publication, and high-risk decisions should remain the responsibility of people with domain expertise.

Fifth, reconsider what distribution means. Any AI application can generate candidate results, but the product that can connect to bibliographic databases, expert networks, local communities, and correction loops will build deeper switching costs. A tool built only on a model API and an attractive dashboard is easy to copy.

05 Counterargument / Risks

I may be underestimating the positive side of this episode. AI agents may indeed allow researchers to scan imagery that previously had not been processed, at a lower cost, especially in regions where labor is scarce and archives are incomplete. If the system can consistently expand coverage after deduplication and expert verification, it will still increase the throughput of genuine discoveries.

I may also be overemphasizing the fact that the site had already been published. If Miotti intended only to say that he had found ruins missing from his own project database, rather than claiming the site was wholly unknown to academia, then both his title and my judgment may be too strong. I have not read his full blog post or the related academic papers, so I may have misjudged this point.

The larger risk is that all discovery must eventually conform to an existing taxonomy. If verification requires candidate points to match known types perfectly, new structural forms and anomalous evidence may also be filtered out. AI's value lies precisely in asking questions that existing search processes are unlikely to ask.

Even so, the episode exposes a contradiction that will not disappear: the speed of AI generation is overtaking the verification speed of professional communities. Whoever solves this bottleneck will own the distribution of scientific AI rather than merely producing more hallucinations at higher volume.

BEN
ai-agentsmayahallucination·

AI Hallucinations Are Not the Discovery of a New Archaeological Site

01 Triggering Event

Google researcher Pietro Miotti said last week that he had used a group of AI agents to identify a possible, previously undocumented Maya site in Mexico.

But the key fact in the original account was that the location had already been discovered the previous year and published in an academic journal. Local communities had also long known about the ruins nearby.

This was not an archaeological discovery. It was a failure to verify a discovery claim. The "villages and regular shapes" identified by the AI agents were, at most, leads worth reviewing; they could not automatically be promoted to a "discovery."

02 What This Really Means

The issue is not whether AI can examine satellite imagery, filter terrain, or batch-flag suspected structures. It is that the current workflow collapses pattern recognition and knowledge confirmation into a single action.

Miotti's agents found lots of villages and regular shapes that looked like structures or houses. That statement is itself restrained: they merely looked like buildings.

The subsequent narrative, however, packaged the result as a potentially unmapped Maya site. What was crossed here was not merely a wording problem but an evidentiary chain:

  • Do the ground coordinates overlap with any known archaeological sites?
  • Could the shapes in the satellite imagery be caused by vegetation, roads, or modern buildings?
  • Do historical records, topography, and local community knowledge corroborate the finding?
  • Has an academic journal already documented the same location?

This is what the episode actually demonstrates: in long-tail domains of professional knowledge, AI has reduced the cost of candidate generation without reducing the cost of adjudication by the same amount.

For developers, this is a concrete lesson in Agent product design. An agent returning a "discovery" does not mean the workflow is complete. It requires provenance, confidence calibration, duplicate retrieval, and human review. Otherwise, spending more tokens simply creates more candidate leads that must be audited.

I have not run Miotti's complete agent stack internally, so I am not identifying the specific step that caused the misattribution. The public reporting is nevertheless sufficient to reject the strong conclusion that "AI discovered an unknown archaeological site."

03 Historical Analogy / Structural Comparison

This resembles the archaeology boom that followed the mass adoption of GPS devices around 2007. Devices enabled more people to find coordinates and anomalies, but academic discoveries were established not by more sensitive sensors but by site documentation, archival comparison, and field verification.

The difference is that AI agents are much better than GPS at generating candidate points in bulk. GPS generally produced relatively sparse anomalous coordinates; an agent can combine satellite imagery, historical maps, and visual models to propose hundreds or thousands of patterns at once.

This corresponds to Stratechery's aggregation theory: value comes not only from model capability, but also from the objects being aggregated, the verification process, and distribution channels. The downside is that aggregation can aggregate errors as well. A system optimized only to "find more places worth examining" will naturally lean toward false positives.

A 2014 AWS analogy is also relevant. Amazon did not make every enterprise system the best in its category. Instead, it aggregated computing capacity, deployment tools, and distribution until it became the default infrastructure. AI archaeology needs a similar layer of trusted infrastructure: coordinate deduplication, journal indexes, map layers, local knowledge, and expert review rather than a single prompt.

In other words, AI can widen the top of the funnel, but the moat will reside in the verification layer rather than the generation layer. Replacing models may be inexpensive; cross-validation data accumulated over years, along with expert-review relationships, is much harder to replicate.

04 What It Means for AI Builders

If I were building an agent product this week, I would immediately tighten the product definition of discovery-oriented outputs.

First, separate action from conclusion. A model may output a candidate site, suspected structure, or lead for review, but the system should not display "discovered" directly. The interface should clearly distinguish a machine-generated hypothesis from a verified finding.

Second, make duplicate detection a default step. For maps, buildings, medical images, legal cases, or security incidents, the system should first check for overlap with authoritative databases and existing publications before allowing the result to enter human review. Without this step, an agent merely produces duplicate answers faster.

Third, require an evidence packet rather than a single confidence score. Every candidate should include the original image region, coordinates, the model's reasoning, possible sources of confusion, relevant literature, and sources of uncertainty. A score can assist prioritization, but it cannot replace auditable evidence.

Fourth, retain a hard human-in-the-loop threshold. In professional domains, AI is best suited to triage, bulk annotation, and anomaly screening. Final naming, publication, and high-risk decisions should remain the responsibility of people with domain expertise.

Fifth, reconsider what distribution means. Any AI application can generate candidate results, but the product that can connect to bibliographic databases, expert networks, local communities, and correction loops will build deeper switching costs. A tool built only on a model API and an attractive dashboard is easy to copy.

05 Counterargument / Risks

I may be underestimating the positive side of this episode. AI agents may indeed allow researchers to scan imagery that previously had not been processed, at a lower cost, especially in regions where labor is scarce and archives are incomplete. If the system can consistently expand coverage after deduplication and expert verification, it will still increase the throughput of genuine discoveries.

I may also be overemphasizing the fact that the site had already been published. If Miotti intended only to say that he had found ruins missing from his own project database, rather than claiming the site was wholly unknown to academia, then both his title and my judgment may be too strong. I have not read his full blog post or the related academic papers, so I may have misjudged this point.

The larger risk is that all discovery must eventually conform to an existing taxonomy. If verification requires candidate points to match known types perfectly, new structural forms and anomalous evidence may also be filtered out. AI's value lies precisely in asking questions that existing search processes are unlikely to ask.

Even so, the episode exposes a contradiction that will not disappear: the speed of AI generation is overtaking the verification speed of professional communities. Whoever solves this bottleneck will own the distribution of scientific AI rather than merely producing more hallucinations at higher volume.