Common Crawl has crawled 300 billion pages cumulatively since 2007—it's the de facto training data starting point for nearly every large model. But what determines a model's capability ceiling isn't parameters or GPUs; it's the data fed in. Architectures can be copied, compute can be bought, but high-quality corpora are the secret each lab guards most tightly. What concerns us more is the other side: this data source is being choked off—book subsets are being withdrawn over copyright, Reddit and Stack Overflow are starting to charge licensing fees. Commercially usable full corpora are far harder to obtain than they were three years ago.
What This Is
Training a large model consumes several trillion to over ten trillion tokens of text (one token is roughly one Chinese character or half an English word). Few data sources can support this volume: public web pages make up the bulk; books boost long-form coherence but their public subsets are shrinking; GitHub code enhances logical reasoning; arXiv papers provide academic rigor; Wikipedia is a standard component of nearly every corpus.
Raw web pages are dirty—HTML tags, navigation bars, and ads take up most of the bytes. They pass through a funnel: language identification, body extraction (using libraries like Trafilatura to strip boilerplate), deduplication, and filtering of low-quality and toxic content. Each stage sheds another batch, leaving only a few percent of the original volume.
How the Industry Views It
The old saying goes: Quality In, Quality Out. With the same architecture and compute, different data yields a model one tier apart. Top labs treat their data recipes as the most closely guarded commercial secret.
But there are dissenting voices. One risk is copyright: if The New York Times v. OpenAI sets a precedent, the entire industry may need to renegotiate licensing, pushing training costs—already in the hundreds of millions of dollars—even higher. Another concern is "data hitting a wall": after the public web has been mined several rounds over, the marginal returns of pure data accumulation are diminishing—which is also why synthetic data has been seriously discussed over the past year.
Also worth noting: fastText, the standard tool for language identification, had its GitHub repo archived by Meta in 2024 and is no longer actively maintained.
Impact on Regular People
For Enterprise IT: The unique data you hold—industry reports, customer conversations, internal documents—has become a true moat in the AI era. How to use it and how to stay compliant matter more than algorithms.
For Individual Careers: Quality answers you've written in professional communities may be worth more than you think—AI companies are scraping this content, paying for it, or fighting over it in court. "Publishing quality content in public" is becoming a hidden asset.
For the Consumer Market: Homogenized products wrapped around generic LLM APIs will proliferate, because training data overlap heavily. What can truly differentiate are vertical private data and deep understanding of user context—neither of which cheap APIs can solve.