
A small crawler calling itself jscrawler, version 0.1, with a link to GitHub’s front page instead of to a project of its own, so it points to no one. The German bot directory WebRobots says only that it indexes and analyses the content of websites.
On this site: It went through the whole site twice in a row, about two pages a second each time, without asking for the rules, and hasn’t been back: someone’s small crawler, run once and then run again to be sure.
Tools anyone can run from a laptop or a rented server: curl, Python scripts, small homemade scrapers, research crawlers. They could be a student’s project, someone building a dataset to sell, or the start of a scan; on their own they don’t say which. What they sell →