
The spider of Dataprovider.com, a Dutch company in Groningen that indexes over 400 million domains every month and sells a searchable database of what it finds on them. It follows robots.txt.
Worth knowing: In 2019 Statistics Netherlands took Dataprovider’s file of every Dutch company website, matched it to the national business register, and used it to map the Dutch internet economy (CBS).
How it makes money: It sells its database through a search tool, an API, an AI “Navigator” and data shares on Snowflake, BigQuery and Databricks, priced on request, to brand protection, payments, registries and registrars, and the public sector; PayPal, GoDaddy and MarkMonitor are among its clients. (dataprovider.com)
On this site: It read the homepage, the rules and most of the works: adding this site to a database of all the world’s websites, the kind a national statistics office once used to measure a country’s internet economy.
They map which sites link to which and how pages rank, and sell that map to marketers and agencies trying to climb the search results. What they sell →