Most visits to a website aren't made by people. Search engines, AI companies, marketing tools, chat apps and security scanners all send programs (bots, crawlers, spiders) that read pages on their own. Most of them say who they are in the name they give with each visit (the “user agent”), and that name is how The Bot Collector sorts them. Here is who they are. What their companies do with what they collect, and who buys it, is on Where it goes.
Many sites turn away programs that admit to being programs, so dressing up as a browser gets through. Renting servers in the cloud is cheap and each one comes with a fresh address; borrowing someone’s home connection costs more, but is the hardest to tell from a person. In August 2025 Cloudflare reported that an AI search company, once its named crawlers were blocked, carried on with an unnamed one that said it was Chrome on a Mac and kept changing its addresses (report). What they sell →
To keep an up to date copy of the web, so they can show it in their results and sell advertising beside them. More and more, the same copy also feeds their AI answers. What they sell →
Writing and pictures are the raw material for training AI models, and AI search and assistants fetch pages to answer people’s questions. The answer is given inside the chat, so the person asking often never comes to the site itself. What they sell →
They map which sites link to which and how pages rank, and sell that map to marketers and agencies trying to climb the search results. What they sell →
When someone pastes a link into a chat or a post, the app fetches the page to show its title and picture. Each of these usually means a real person shared a link. What they sell →
A few are security companies mapping what is exposed on the internet and selling that picture. Most are automatic attempts to find a site with a door left open: a leaked password file, a WordPress login, an admin page. A site that is broken into can be resold, used to send spam, or added to a network of hijacked machines. New sites are found within seconds, because every new site’s security certificate is written into a public log (SANS). What they sell →
Tools anyone can run from a laptop or a rented server: curl, Python scripts, small homemade scrapers, research crawlers. They could be a student’s project, someone building a dataset to sell, or the start of a scan; on their own they don’t say which. What they sell →