
A research crawler from Carnegie Mellon University in Pittsburgh. It collects pages for ClueWeb, a series of very large copies of the web that the university gives to researchers working on search engines and language technology.
Worth knowing: The last one, ClueWeb22, holds 10 billion web pages. It was made with Microsoft, from pages a commercial search engine’s crawler had found (arXiv).
What happens to it: The pages become research datasets: there is “no fee to license” ClueWeb22, only “a modest fee” for distributing it, companies can sign a licence too, and its makers named pre-training AI language models as one use. (ar5iv.labs.arxiv.org)
On this site: It keeps coming back from the university’s own network and reads the works one by one: this site may well end up in the next ClueWeb, studied by researchers for years after the pages themselves have changed.
Tools anyone can run from a laptop or a rented server: curl, Python scripts, small homemade scrapers, research crawlers. They could be a student’s project, someone building a dataset to sell, or the start of a scan; on their own they don’t say which. What they sell →