Programs without a company
The painted mark for Wget
Placeholder: mark still to be painted

Wget

GNU Wget, a free tool from 1996 for downloading files from the command line. It comes ready installed on many Linux computers and servers, it can pick up a download where it broke off, and it can follow every link on a site to save a copy of the whole thing, reading robots.txt first when it does.

People use it to save a site to read offline or as a backup, in server scripts that fetch a file every night, and to save sites before they disappear: Archive Team, the volunteers who rescue websites before they shut down, use it a great deal (Archive Team).

Worth knowing: It was first called Geturl, and was renamed when its author found there was already an Amiga program called GetURL; since 2012 it can save what it downloads in WARC, the web archives’ own format (Wikipedia).

What happens to it: Nobody sells these visits: Wget is free, and each one is whoever wrote or ran the program, who decides what happens to what it fetched.

On this site: A visit under its own name would mean someone saving pages from this site to keep. Wget can also be told to send no name, and then it lands among the unnamed bots.

Why? Tools anyone can run from a laptop or a rented server: curl, Python scripts, small homemade scrapers, research crawlers. They could be a student’s project, someone building a dataset to sell, or the start of a scan; on their own they don’t say which. What they sell →