Common Crawl

CCBot

A non-profit that has been crawling the web since 2008 and gives away the results: a free archive of billions of web pages, used by researchers, and by almost every company building AI language models.

Worth knowing: filtered Common Crawl data made up about 60% of the training mix for GPT-3, the model that came before ChatGPT, and a 2024 Mozilla Foundation study concluded that generative AI in its current form would probably not be possible without it. Common Crawl has taken donations from AI companies, OpenAI and Anthropic among them.

Disputed: in November 2025 an investigation by The Atlantic said the archive holds millions of paywalled articles from outlets such as the New York Times, the Economist and the Atlantic itself, because its crawler never runs the code that puts the paywall up, and that articles publishers asked to have removed were hidden from Common Crawl's search tool rather than taken out of the archive. Its director, Rich Skrenta, told the magazine “The robots are people too.” Common Crawl called the article's claims false and misleading.

Why? Writing and pictures are the raw material for training AI models, and AI search and assistants fetch pages to answer people’s questions. The answer is given inside the chat, so the person asking often never comes to the site itself.