
The crawler of Awario, a “social listening” tool that finds new pages mentioning its customers’ brands, from the company behind the SEO tool SEO PowerSuite. Awario says its bots collect only public information, follow robots.txt and ask for one page every three seconds at most.
Worth knowing: Awario says it began in 2015 as a little pet project that branched off SEO PowerSuite, another product of the same company, and now crawls 13 billion web pages a day (Awario).
On this site: It read the whole site properly, looking for mentions of the brands its customers watch: any brand named here, an advertising client or a company whose bot lands on the head, may have turned up in someone’s report.
They map which sites link to which and how pages rank, and sell that map to marketers and agencies trying to climb the search results. What they sell →