Programs without a company
The painted mark for MapTheNetBot
Placeholder: mark still to be painted

MapTheNetBot

The crawler of MapTheNet, a map of the internet that draws each website as a dot, placed by which sites link to it and which it links to. It reads a few pages of each site only to see their links; its page says it keeps no page content, addresses or titles.

Worth knowing: The map shows more than 46 million domains (MapTheNet). It was inspired by The Internet Map of 2011, which drew about 350,000 websites and was never updated (MapTheNet).

What happens to it: It’s given away: the rankings and the whole link map can be downloaded free under a Creative Commons licence (BY-SA 4.0) that also allows commercial reuse, so anyone may build on it. (mapthenet.org)

On this site: It reads a handful of pages each time, starting with the rules, from a German fibre broadband line rather than a data centre: perhaps a one-person project mapping the internet from home, with this site as one more dot.

Why? Tools anyone can run from a laptop or a rented server: curl, Python scripts, small homemade scrapers, research crawlers. They could be a student’s project, someone building a dataset to sell, or the start of a scan; on their own they don’t say which. What they sell →