
Applebot, Apple’s crawler. What it collects powers search in Siri, Spotlight and Safari and, unless a site says no with a separate robots.txt name, Applebot-Extended, it may also be used to train Apple’s AI. Apple publishes the addresses it crawls from.
Worth knowing: When Apple announced the AI models behind Apple Intelligence in June 2024, it said they were trained on licensed data and “publicly available data collected by our web-crawler, AppleBot” (Apple). Applebot-Extended is a name with no robot behind it: “Applebot-Extended does not crawl webpages”; it only says whether what Applebot has collected may be used for AI (Apple).
How it makes money: Indirectly: Apple makes its money from its devices and services, and what Applebot collects makes features in them more useful. Small sites aren’t paid for it. (support.apple.com)
On this site: It comes from Apple’s own network and reads The Bot Collector most: for Siri and Spotlight, and perhaps, unless this site says no, for Apple’s AI.
Writing and pictures are the raw material for training AI models, and AI search and assistants fetch pages to answer people’s questions. The answer is given inside the chat, so the person asking often never comes to the site itself. What they sell →