
serpstatbot, the link crawler of Serpstat, a search-marketing platform that began in 2013 as an in-house tool at the Netpeak agency in Odesa, Ukraine. It follows links across the web to keep Serpstat’s backlink index up to date, and publishes its addresses.
Worth knowing: If it can’t read a site’s robots.txt (when the file is forbidden, say), it takes that as permission and crawls everything it can reach; it also follows links marked “nofollow” (Serpstat).
How it makes money: Subscriptions: Individual $50 a month, Team $100 and Agency $410, which includes “White Label Reports” for agencies to pass on under their own name and 2 million API credits. It claims over 1.1 million users. (serpstat.com)
On this site: It went through the site in a single day, Sketches from the Metaverse most: noting every link, so Serpstat’s customers can see who links where.
They map which sites link to which and how pages rank, and sell that map to marketers and agencies trying to climb the search results. What they sell →