Most visits to a website aren't made by people. Search engines, AI companies, marketing tools, chat apps and security scanners all send programs (bots, crawlers, spiders) that read pages on their own. Most of them say who they are in the name they give with each visit (the “user agent”), and that name is how The Bot Collector sorts them. Here is who they are.
// Bots in browsers
Why? Many sites turn away programs that admit to being programs, so dressing up as a browser gets through. Renting servers in the cloud is cheap and each one comes with a fresh address; borrowing someone’s home connection costs more, but is the hardest to tell from a person. In August 2025 Cloudflare reported that an AI search company, once its named crawlers were blocked, carried on with an unnamed one that said it was Chrome on a Mac and kept changing its addresses (report).
// Search engines
Why? To keep an up to date copy of the web, so they can show it in their results and sell advertising beside them. More and more, the same copy also feeds their AI answers.
// AI companies
Why? Writing and pictures are the raw material for training AI models, and AI search and assistants fetch pages to answer people’s questions. The answer is given inside the chat, so the person asking often never comes to the site itself.
// Marketing and search-ranking tools
Why? They map which sites link to which and how pages rank, and sell that map to marketers and agencies trying to climb the search results.
// Link previews
Why? When someone pastes a link into a chat or a post, the app fetches the page to show its title and picture. Each of these usually means a real person shared a link.
// Security scanners
Why? A few are security companies mapping what is exposed on the internet and selling that picture. Most are automatic attempts to find a site with a door left open: a leaked password file, a WordPress login, an admin page. A site that is broken into can be resold, used to send spam, or added to a network of hijacked machines. New sites are found within seconds, because every new site’s security certificate is written into a public log (SANS).
// Programs without a company
Why? Tools anyone can run from a laptop or a rented server: curl, Python scripts, small homemade scrapers, research crawlers. They could be a student’s project, someone building a dataset to sell, or the start of a scan; on their own they don’t say which.
// People
Why? To look at the work.