
Requests is the most common way for a Python program to fetch something from the web. It is a free add-on, so nobody runs “the” Requests bot: a visit in its name is someone’s script that didn’t give itself a name, a data project, a scraper or a bot someone wrote themselves. Whether it reads robots.txt is up to whoever wrote it.
Worth knowing: It is downloaded around one and a half billion times a month (PyPI Stats). Its author, Kenneth Reitz, handed it to the Python Software Foundation in 2019 (Wikipedia).
What happens to it: Nobody sells these visits: Requests is a free library, and each one is whoever wrote or ran the program, who decides what happens to what it fetched.
On this site: Scripts from home connections and cloud servers alike, asking for the homepage, the rules, ads.txt and news feed addresses: people’s own small programs, looking for something to read, or for adverts to sell.
Tools anyone can run from a laptop or a rented server: curl, Python scripts, small homemade scrapers, research crawlers. They could be a student’s project, someone building a dataset to sell, or the start of a scan; on their own they don’t say which. What they sell →