ByteDance

Bytespider

ByteDance is the Chinese company that owns TikTok. Its crawler, Bytespider, is widely thought to gather material for ByteDance's AI models (such as its Doubao chatbot) and search.

Worth knowing: in 2024 the web-security company Cloudflare reported Bytespider as the busiest AI crawler it saw across the sites it protects, and Fortune reported that it ignores robots.txt. Research by TollBit covering the first half of 2026 found it was one of three bots that reached blocked pages on close to half of the European sites that named them, alongside OpenAI's ChatGPT‑User and Youbot.

But: ByteDance publishes nothing about Bytespider, not even a way to check that a visit really is its crawler. So any program can call itself Bytespider, and studies that count by name may blame ByteDance for impostors. Some of the ByteDance visits here may not be ByteDance at all.

Why? Writing and pictures are the raw material for training AI models, and AI search and assistants fetch pages to answer people’s questions. The answer is given inside the chat, so the person asking often never comes to the site itself.