
Almost certainly Pandalytics, the crawler of Domainsbot, a domain data company founded in Rome in 2004. Its name gives the address domainsbot.com/…, and The Bot Collector names a bot by the first word ending in “bot” it finds, so it was filed as domainsbot. It reads robots.txt and the homepage to note a site’s language, technology and security settings for Domainsbot’s domain data.
Worth knowing: Domainsbot says it follows more than 420 million registered domains across over 7,000 endings such as .com and .uk (Domainsbot).
On this site: Robots.txt and the homepage, Pandalytics’s usual round: one more domain checked for its language, technology and security, among hundreds of millions.
They map which sites link to which and how pages rank, and sell that map to marketers and agencies trying to climb the search results. What they sell →