Programs without a company
The painted mark for bl.uklddcbot
Placeholder: mark still to be painted

bl.uklddcbot

The UK’s legal deposit web crawler, run by the British Library for the UK Web Archive (it calls itself bl.uk_lddc_bot; the log leaves out the underscores). Since 2013 the law has let the UK’s legal deposit libraries collect UK websites as they collect printed books: once a year they take a snapshot of the UK web, at least four million websites, and smaller collections through the year.

Worth knowing: The archive’s own website was offline for three years after the cyber attack on the British Library in October 2023, coming back only in October 2026 (Wikipedia), but the Library says its curators kept on collecting all along, with crawlers running on Amazon’s cloud (British Library).

What happens to it: Kept by law: the Library must keep at least one copy, may use it only for its own non-commercial research, and shows it mainly on reading-room computers, one reader at a time, with no opt-out for site owners. (legislation.gov.uk)

On this site: It came from the UK and from Amazon’s cloud, as the Library says its crawlers do: copies of this website are probably in the national archive now, kept with the country’s books.

Why? Tools anyone can run from a laptop or a rented server: curl, Python scripts, small homemade scrapers, research crawlers. They could be a student’s project, someone building a dataset to sell, or the start of a scan; on their own they don’t say which. What they sell →