
Amazonbot, Amazon’s crawler. Amazon says it is used to improve its products and services and may be used to train Amazon’s AI models. Two newer Amazon agents have names of their own: Amzn-SearchBot, for search in Amazon’s products, and Amzn-User, which fetches pages when someone asks Alexa something. Amazon publishes the addresses of all three.
Worth knowing: Amazon’s crawlers ignore crawl-delay, the robots.txt line asking bots to slow down, keep a copy of a site’s robots.txt for up to 30 days, and read the old “noarchive” tag in a new way: “do not use the page for model training” (Amazon).
How it makes money: Indirectly, through the products and AI models it helps improve. Amazon pays some big publishers for their work, the New York Times a reported $20 to $25 million a year and the UK’s Reach by use, but not small sites. (pressgazette.co.uk)
On this site: It comes almost every day from Amazon’s own cloud and reads the rules only now and then, as its 30-day copy allows, The Bot Collector most. Whatever it takes may be used to train Amazon’s AI.
Writing and pictures are the raw material for training AI models, and AI search and assistants fetch pages to answer people’s questions. The answer is given inside the chat, so the person asking often never comes to the site itself. What they sell →