
BitMoversBot
“A small research crawler run by Byte Dev Limited from the UK”, under the name Bit Movers. It reads public pages, mostly homepages, contact and about pages, to map how sites link to each other, what technology they use and what they are about, and keeps a searchable index that includes the contact details sites publish. It says it doesn’t sell or publish the data.
Worth knowing: It publishes its addresses in the same format Google uses for Googlebot, and the list holds just one, on a fibre broadband line (Bit Movers).
How it makes money: Not the data, it says: its company, Byte Dev Limited, sells “Advanced data extraction, migration, integration and data warehousing”. (bitmovers.co.uk)
On this site: The rules, then the homepage 34 seconds later, from the one address it publishes, on a fibre broadband line: a small, careful crawler that takes its time, perhaps run from someone’s home or office.
Tools anyone can run from a laptop or a rented server: curl, Python scripts, small homemade scrapers, research crawlers. They could be a student’s project, someone building a dataset to sell, or the start of a scan; on their own they don’t say which. What they sell →