Programs without a company
The painted mark for Scrapy
Placeholder: mark still to be painted

Scrapy

Scrapy, a free Python toolkit for building web scrapers (it calls them spiders): programs that collect information from websites. Anyone can run it, so a visit is someone’s own project. It is looked after by Zyte, a web scraping company.

Worth knowing: Scrapy was born at Mydeco, a London shopping site, with Insophia, a web consultancy in Montevideo, Uruguay, and first released in August 2008 (Wikipedia). Whether it reads robots.txt is up to whoever runs it: the setting is off by default “for historical reasons” (Scrapy).

What happens to it: Nobody sells Scrapy’s visits: each one is someone’s own project. Zyte, which looks after it, earns by selling the Zyte API, priced per 1,000 pages by how hard a site is to scrape, from $0.13. (zyte.com)

On this site: It only ever looks at the homepage, never the rules: by the look of it, one person’s small scheduled job, checking for changes.

Why? Tools anyone can run from a laptop or a rented server: curl, Python scripts, small homemade scrapers, research crawlers. They could be a student’s project, someone building a dataset to sell, or the start of a scan; on their own they don’t say which. What they sell →