Who's visiting

The visitors in The Bot Collector, and what they do

Most visits to a website aren't made by people. Search engines, AI companies, marketing tools, chat apps and security scanners all send programs (bots, crawlers, spiders) that read pages on their own. Most of them say who they are in the name they give with each visit (the “user agent”), and that name is how The Bot Collector sorts them. Here is who they are. What their companies do with what they collect, and who buys it, is on Where it goes.

Bots in browsers

Why? Many sites turn away programs that admit to being programs, so dressing up as a browser gets through. Renting servers in the cloud is cheap and each one comes with a fresh address; borrowing someone’s home connection costs more, but is the hardest to tell from a person. In August 2025 Cloudflare reported that an AI search company, once its named crawlers were blocked, carried on with an unnamed one that said it was Chrome on a Mac and kept changing its addresses (report). What they sell →

Peoplebots Unnamed programs using a browser name HeadlessChrome

Search engines

Why? To keep an up to date copy of the web, so they can show it in their results and sell advertising beside them. More and more, the same copy also feeds their AI answers. What they sell →

Google Bing Yandex Baidu DuckDuckGo Huawei (Petal) Seznam Qwant Mojeek ibou-search CGMSearchBot IbouBot

AI companies

Why? Writing and pictures are the raw material for training AI models, and AI search and assistants fetch pages to answer people’s questions. The answer is given inside the chat, so the person asking often never comes to the site itself. What they sell →

OpenAI Anthropic Meta Apple Amazon ByteDance Perplexity Common Crawl Cohere KeenableBot ElevenlabsCrawler ExaSearchBot Reflectionbot SofyaBot RiverBot YouBot ShapBot Claude-User PostHogImageFetcherBot

Marketing and search-ranking tools

Why? They map which sites link to which and how pages rank, and sell that map to marketers and agencies trying to climb the search results. What they sell →

Semrush Ahrefs Moz Majestic DataForSEO Serpstat Screaming Frog BLEXBot Barkrowler QlyzeBot SEOJuice-SearchBot SERankingBacklinksBot AwarioBot Dataprovider.com BarrigazBot CrolBot Kaupr webXtrendBot PipericBot AgentDataBot HaloBot DomainMetaDataBot LeadFinderBot TechLeadsBot webatlabot AtlasBot domainsbot FundigBot

Why? When someone pastes a link into a chat or a post, the app fetches the page to show its title and picture. Each of these usually means a real person shared a link. What they sell →

X (Twitter) LinkedIn Pinterest Slack Discord Telegram Link previews Are.naBot Embedly

Security scanners

Why? A few are security companies mapping what is exposed on the internet and selling that picture. Most are automatic attempts to find a site with a door left open: a leaked password file, a WordPress login, an admin page. A site that is broken into can be resold, used to send spam, or added to a network of hijacked machines. New sites are found within seconds, because every new site’s security certificate is written into a public log (SANS). What they sell →

Security scanners Censys Palo Alto Networks DeepSeekBot GrokBot MoonshotBot ChatGLM-Spider Qwenbot Bravebot ApcosysBot Kimi-SearchBot KimiBot PanguBot BitSightBot UmaiBot YiBot OAI-AdsBot intelx.iobot TLDWPBot

Programs without a company

Why? Tools anyone can run from a laptop or a rented server: curl, Python scripts, small homemade scrapers, research crawlers. They could be a student’s project, someone building a dataset to sell, or the start of a scan; on their own they don’t say which. What they sell →

Using a well-known bot's name curl python-requests Go-http-client Wget Scrapy Other named bots fresh-domain-watch X-Spider ClueWeb-Crawler jscrawler TernCrawler bl.uklddcbot MapTheNetBot DocSpider Java Google-CloudVertexBot wpbot xAI-SearchBot python-httpx hexidebot NVBot scrapecentral UKBusinessDirectoryBot MochiImageCrawler ThemeSmithResearchBot tphotobot CBSLeadsBot FindgBot SpiderLing TflowSitemapBot Top1mBot bot-research-checker eManagerBot NixieBot QueryWinBot FAB-Scraper GoldiBot Python-urllib AMLE-SiteCheck AntiCloneBot BitMoversBot domains-crawler LanternBot LeadGennyBot LumpsScraper MinakiBot RepetivBot aiohttp cyber-asset-fetch DeccanCorpusBot DomainBot GiantCrawler LuvaCheckBot OnlineShopsBot SiteDecayBot WebTunerBot Unnamed bots