
The crawler of Qlyze, a search-ranking (SEO) tool whose bot page is in German. Its page says it reads public pages to see how they link to each other, for a backlink index: a map of which sites link to which, the thing search-ranking tools sell to marketers. It doesn’t log in or fill in forms, and follows robots.txt.
On this site: It went through the whole site in a few long sessions, the last of them nearly 180 pages in under eight minutes, following links from page to page: mapping how this site’s pages link to each other and to the rest of the web, for marketers to study.
They map which sites link to which and how pages rank, and sell that map to marketers and agencies trying to climb the search results. What they sell →