
The crawler of webXtrend, a domain and DNS information service in Dresden. It publishes facts about domains (where they point, what else shares their server, their social media profiles) for market research, IT security and brand protection. It reads robots.txt, then the homepage only, and comes back now and then.
Worth knowing: It publishes the exact addresses it uses, twelve of them, and signs its visits to shops on Shopify (webXtrend).
How it makes money: It sells its domain data through an API and as downloads, including business contact details where sites publish them in structured data. Its crawler page shows no prices. (webxtrend.com)
On this site: Robots.txt, then the homepage, again and again, just as its page says: keeping this domain’s record up to date in a database for market research and brand protection.
They map which sites link to which and how pages rank, and sell that map to marketers and agencies trying to climb the search results. What they sell →