
The crawler of You.com, the AI search engine: it reads public pages for You.com’s search results, follows robots.txt and can be checked by its addresses. But not every YouBot that came here was You.com’s.
Worth knowing: You.com was founded in 2020 by two former Salesforce researchers, Richard Socher and Bryan McCann. In December 2022 it launched YouChat, one of the first chatbots to answer with live results from the web (Wikipedia).
How it makes money: It now sells its search to other companies’ AI by the call: $5 per 1,000 searches or answers, 100 free a day, and says its API handles over a billion queries a month, for customers such as DuckDuckGo. (you.com)
On this site: The real crawler came the proper way, robots.txt then one page; the rest under its name were a scanner borrowing it, asking for addresses that would give away a server’s insides. One name, two very different visitors.
Writing and pictures are the raw material for training AI models, and AI search and assistants fetch pages to answer people’s questions. The answer is given inside the chat, so the person asking often never comes to the site itself. What they sell →