Programs without a company
The painted mark for ClueWeb-Crawler
Placeholder: mark still to be painted

ClueWeb-Crawler

A research crawler from Carnegie Mellon University in Pittsburgh. It collects pages for ClueWeb, a series of very large copies of the web that the university gives to researchers working on search engines and language technology.

Worth knowing: The last one, ClueWeb22, holds 10 billion web pages. It was made with Microsoft, from pages a commercial search engine’s crawler had found (arXiv).

What happens to it: The pages become research datasets: there is “no fee to license” ClueWeb22, only “a modest fee” for distributing it, companies can sign a licence too, and its makers named pre-training AI language models as one use. (ar5iv.labs.arxiv.org)

On this site: It keeps coming back from the university’s own network and reads the works one by one: this site may well end up in the next ClueWeb, studied by researchers for years after the pages themselves have changed.

Why? Tools anyone can run from a laptop or a rented server: curl, Python scripts, small homemade scrapers, research crawlers. They could be a student’s project, someone building a dataset to sell, or the start of a scan; on their own they don’t say which. What they sell →