Where it goes

What the companies behind the bots sell, and who buys it

Nearly every visit to this site is made by a program: in the week of 26 October 2026, 99% of the pages The Bot Collector logged. Who's Visiting says who sends them. This page follows what happens next: what the companies behind them sell, who buys it, and what one page is worth. Sites are the raw material here, never the customer, and none of the money comes back to them.

Research by Claude (Anthropic), November 2026. Anthropic is one of the companies covered; its figures come from the same outside sources as everyone else’s. Prices were read from the companies’ own pages in November 2026 and change often. What happened on this site comes from The Bot Collector’s own log.

Five ways a page becomes money

How the money is madeWhat is sold, and to whomTypical priceTheir bots on this site
Adverts and answers (search engines)Adverts beside results and AI answers; results sold wholesale to other search engines and AI buildersGoogle: $63.4 billion from search in one quarter; Bing results for AI agents: $14 per 1,000 searchesGoogle, Bing, Yandex, Baidu, DuckDuckGo, Petal, IbouBot
AI models and AI searchChatbot subscriptions, adverts and API access; search “by the call” for AI agentsChatGPT adverts: $1 billion a year and rising; Exa: $1 per 1,000 pages of textOpenAI, Anthropic, Meta, Apple, Amazon, ByteDance, Perplexity, Common Crawl, Exa
Maps for marketers (search-ranking or “SEO” tools)Indexes of links, Google rankings and traffic estimates, rented by marketers and agencies or sold wholesale to other toolsSemrush from £113 a month; DataForSEO about $0.05 per 1,000 link recordsAhrefs, Semrush, Majestic, Moz, Babbar, DataForSEO, Serpstat
Contact and technology listsLists of sites by the software they run, with business emails, phones, names and job titles, for sales teams and AI sales agentsPiperic $0.01 a row; AgentData from $99 a monthKaupr, AgentDataBot, TechLeadsBot, AtlasBot, PipericBot, Dataprovider.com
Scans and scores (security)Searchable records of every server; security ratings of organisations, for security teams, insurers, investors and governmentsApcosys from $50 a month; Intelligence X from €3,000 a yearCensys, BitSightBot, ApcosysBot, intelx.io_bot, UmaiBot

In all five the money comes from people who want to know about other people’s sites, or who want answers without visiting them.

What a page is worth

Under The Bot Collector’s head, a pink band adds up its “Total data scrape fees”: $0.01 for every page a program has taken. Nobody pays it. Here is what a page sells for in the real trade:

WhoWhatPrice of one page
ZyteScraping service: a plain page from a site that doesn’t resist$0.00013
You.comA page’s text for AI apps$0.0009
ExaA page’s text for AI apps$0.001
SofyaA page fetched for an AI agent$0.0025
KeenableOne page or search for an anonymous AI agent paying in cryptocurrency$0.01
The Bot CollectorThe fee shown under the head, never charged$0.01

So the piece’s invoice is priced at the top of the market, ten times Exa’s rate. One page is worth almost nothing; the money is in volume and in selling the same material again and again. DataForSEO has no dashboard of its own and sells raw link and search data to other tool makers, who put their own names on it. Semrush’s list of its own bots includes one for another company’s product. Agencies resell SE Ranking and Serpstat under their own brands. One Bing crawl feeds DuckDuckGo, Ecosia, Copilot and paying AI builders, and Common Crawl gives its whole archive away. A site sees only the first bot’s name; the second and third buyers never show.

Bots in browsers

Programs dressed as browsers don’t say who they are, so nobody can say where their pages go. What is known is that disguise is for sale. The scraping firm Zyte offers “automatic ban handling” through rotating home and mobile addresses (Zyte), and in August 2025 Cloudflare caught Perplexity, once its named crawlers were blocked, carrying on with an undeclared one that said it was Chrome on a Mac, at 3 to 6 million requests a day; Perplexity blamed a third-party browser service (Cloudflare). Even named bots arrive this way: ElevenLabs’ crawler reached this site from rented hosting and proxy networks, not its own.

Search engines

Search adverts still pay for most crawling. In the three months to September 2026, Google Search brought in $63.35 billion (Alphabet), nearly all of it adverts beside results built from pages read for free. The same pages now feed Google’s AI Overviews and AI Mode, and people shown an AI summary clicked an ordinary result on 8% of visits, against 15% without one (Pew).

Microsoft is the web’s wholesaler: it sells “Grounding with Bing Search” to builders of AI agents at $14 per 1,000 searches (Microsoft). Yandex sells its results at $4 per 1,000, or $41.64 per 1,000 for an AI-written answer: ten times as much for the answer as for the links (Yandex). CGMIMM sells “#1 placement in CGM Local search results” at $49.95 a month, so part of its results are bought (CGMIMM).

In June 2026 Britain’s Competition and Markets Authority ordered Google to let publishers keep their pages out of its AI features (CMA). No ruling anywhere obliges a search engine to pay a site for being crawled.

AI companies

For AI companies crawling is mostly training: “nearly 80%” of AI bots’ crawling in 2025, by Cloudflare’s count. In one week of August 2025 it also counted how many pages each company fetched for every visitor it sent back: Anthropic nearly 50,000, OpenAI 887, Perplexity 118 (Cloudflare). Later readings are lower, but AI answers still take a great deal and give back a cited link at most.

The money goes to rights holders with size or lawyers. OpenAI reportedly pays News Corp over $250 million over five years, Meta pays it up to $50 million a year, Amazon pays the New York Times a reported $20 to $25 million a year (Press Gazette); Anthropic’s largest payment to creators is its $1.5 billion settlement with authors, about $3,127 for each of some 482,000 books it had taken from pirate libraries (Writer Beware). No evidence was found of any AI company paying an individual artist or a small independent site for crawling.

Behind them sits Common Crawl, a small non-profit that gives away a 9.5-petabyte archive of the web; it made up more than 80% of the text in OpenAI’s GPT-3 (Mozilla Foundation). Blocking it stops future crawls only: whatever is already in a snapshot stays downloadable by any AI developer.

Start-ups selling pages to AI agents

When Microsoft closed its search APIs in August 2025, AI developers lost their cheap supply of web search, and start-ups rushed in to sell it to “AI agents”. Each needs its own copy of the web, so each runs its own crawler: new names such as KeenableBot, RiverBot and CrolBot reached this site in the autumn of 2026. Exa is valued at $2.2 billion, and also sells named people’s email addresses at $0.02 and phone numbers at $0.07 (Exa). PostHog’s bot fetches images seen in its customers’ visitor recordings “for AI training” (PostHog); it visited, although this site doesn’t use PostHog.

Publishers create it. Exa crawls it. AI companies buy it. Publishers get nothing.— The Media Copilot

Marketing and search-ranking tools

The most persistent visitors are search-marketing companies. They crawl the whole web and keep mostly the links between pages: the more good links point to a site, the higher Google tends to rank it. They add Google’s results, gathered by automated searching, and rent the picture by monthly subscription to marketers studying their competitors. Semrush, the biggest, had revenue of $443.6 million in 2025 and was bought by Adobe for about $1.9 billion in April 2026 (Adobe). Ahrefs reads “5 M” pages a minute (Ahrefs);

The part that concerns people rather than sites is “clickstream” data, records of what real people search for and click. Semrush says its traffic estimates come from “a panel of over 200 million real, anonymized internet users”, supplied by data providers it doesn’t name (Semrush). In February 2026 a researcher reported 287 Chrome extensions, with about 37 million installs, sending the addresses people visited to more than 30 recipients, Semrush named among them (The Register).

By their own rules these bots behave: the big ones read this site’s robots.txt often and come from their makers’ own networks. But blocking has a limit. Their indexes record links to a site found on other people’s pages, so robots.txt limits what they read, not what they know.

Contact lists: an About page becomes a sales lead

Some of the smallest bots are the most personal. “Technographics” firms sort every website by the software it runs, its country and its sector; “lead” firms add the contact details found on it and sell the result to salespeople, and more and more to AI sales agents. For a sole trader the business and the person are the same, so a company list with a sole trader’s name and email on it is a list of that person. One visit from Kaupr was twelve requests in a few seconds, asking for the About page and for pages called team, our-team and meet-the-team, where business sites usually list their people.

Kaupr says different things to different readers. In the file it writes for AI agents (llms.txt), it records “the names, job titles and short biographies of people a website lists, and contact e-mail addresses: published ones and, in some cases, an inferred address”, and its fleet “reads robots.txt after the homepage and does not apply it”. Its privacy policy, written for people, says “We do not collect consumer personal data from the sites we crawl”, and keeps records “with no set deletion date” (Kaupr). Its plans run from €89 to over €50,000 a month.

CompanyWhat it sellsPriceContact details it says it takes
PipericRows from 71.3 million websites, with a lead-generation preset$0.01 a row; the whole database for $4,999Business emails and phones
AgentDataWho uses which software across 62 million websitesFrom $99 a monthProfessional emails, names and job titles
TechLeadsLists “ready for CRM import”From $29“Emails, phone numbers, social profiles”
BizCensus (AtlasBot)Spreadsheets of businesses: “You only pay for records with a phone or an email”By volumeEmail, phone, address, social profiles
Dataprovider.com200+ facts on each of 400 million domainsOn requestIts data “will include personal data”

Regulators’ cases show how people on such lists find out: when a stranger’s sales email arrives. France fined KASPR €240,000 in 2024 after “many complaints” from people approached with contact details it had scraped (CNIL), and in May 2026 Italy fined a buyer, a sole trader who emailed people using bought data without checking it was lawfully collected (Enforcement Tracker). Across these firms, what openness there is speaks to website owners, not to the people listed: none of the small ones says it writes to the people it lists, which the law normally expects. BizCensus’s admission that “What it no longer obeys is Disallow” sits inside the program code of its own web page.

When someone shares a link, the platform’s bot fetches the page’s title, description and picture to build a card, and that card then lives inside a post under the platform’s rules. LinkedIn has trained its AI on UK members’ profiles and posts by default since November 2025 (TechRadar). Pinterest trains on Pins by default, including pictures people save from other sites (PetaPixel). X trains on public posts and bars everyone else from training on them (TechCrunch). Slack says “We do not currently honor robots.txt files” (Slack), and Telegram’s Instant View can keep a full copy of an article on its own servers. On every platform the choice belongs to whoever posts, not to the site whose picture is in the post.

Security scanners

Security firms don’t read pages for their content: they map the internet’s machines and sell the map. Censys, which grew out of the University of Michigan, calls it “a Google Street View of the Internet” that sees “what kind of door you have, what’s in the window” (University of Michigan). Palo Alto Networks scans “the global internet to monitor our customers’ internet-facing attack surface”: to find a paying customer’s forgotten server it has to look at everyone’s. Bitsight, whose largest shareholder is Moody’s, sells security ratings of organisations to insurers and investors (Bitsight); the rated aren’t asked. Apcosys sells “leak data” on its $300-a-month plan and takes cryptocurrency, and Intelligence X searches “more than 200 billion records”, including passwords stolen by malware, for €3,000 to €20,000 a year.

Every legitimate scanner offers a way out, but only for the future and only for addresses a site can prove are its own. A site on shared hosting can’t. As the inventors of internet-wide scanning wrote in 2013, “there is no IP-level equivalent of the HTTP robots exclusion standard” (ZMap).

Borrowed AI names

The scanners that matter for safety don’t give their real names. They borrow AI crawlers’ names, because many sites wave those through. GreyNoise found “Google-Extended”, a name Google never sends, in 263,849 sessions, every one forged (GreyNoise); Fastly counted about 6.3 million such requests in a single day in September 2026 (Fastly). They want .env files, where programmers keep passwords and keys: in 2024 attackers copied them from at least 110,000 domains and used the stolen keys to empty cloud storage and leave ransom notes (Unit 42). The scanners here work the same way, one program asking for some 248 addresses in a run, mostly .env variants, from rented cloud servers. This site is static and has no such file, so they leave empty-handed. The Bot Collector checks the names of Google, Bing, OpenAI, DuckDuckGo, Apple, Perplexity and Common Crawl against the addresses those companies publish, rather than trusting the name.

Programs without a company

The non-commercial collectors keep the fullest copies. The British Library’s UK Web Archive copies at least four million UK websites a year under legal deposit law, and there is no opting out: owners are “not required to take any action such as changing the website’s ‘robots.txt’” (British Library). Carnegie Mellon’s ClueWeb collection, built with Microsoft, names AI pre-training as a purpose and is licensed free to organisations, companies included (Lemur Project): non-commercial means the university charges nothing, not that companies can’t use it.

The small crawlers each sell a service built on reading other people’s sites. Sketch Engine sells searches of language collections of “10+ billion words per language”; MapTheNet publishes the whole web’s link graph under a licence that lets AI firms reuse it; ThemeSmith keeps copies of pages “for up to 36 months” and deletes them if asked. And any of the programs named after free tools, Scrapy, curl, python-requests, could be anyone: Scrapy alone is downloaded 4.6 million times a month.

The law in the UK

Most of this is lawful for a simple reason: no UK law forbids reading a public web page. robots.txt is a request, not a lock; the standard that defines it says its rules “are not a form of access authorization” (RFC 9309). Facts such as links, rankings and software names belong to nobody.

Copyright protects art at home, not abroad. Copying a work in the UK to train a commercial AI needs permission, but in Getty Images v Stability AI (November 2025) the training claim fell away because the training happened in the US (AIPPI). In March 2026 the government chose not to change the law for now, and left licensing to the market (GOV.UK).

Personal details are firmer ground. Data protection authorities, the ICO among them, have said that “personal information that is publicly accessible is still subject to data protection and privacy laws” (ICO). A firm that gathers someone’s details from somewhere else must normally tell them within a month (UK GDPR, Article 14), and under the marketing rules “sole traders and some partnerships are treated as individuals”: nobody may send them marketing emails without consent (ICO).

Every opt-out found in this research works only from now on: none recalls data already sold or a model already trained. The Bot Collector’s door already turns robots.txt from a request into a lock: any bot the file shuts out is refused. For now it shuts out none, so they all come in, and are collected.

Research by Claude (Anthropic), November 2026. Anthropic, maker of ClaudeBot, is one of the companies covered.