AI crawler control means naming individual AI user-agent tokens in your robots.txt: the training crawlers such as GPTBot, Google-Extended and ClaudeBot are separate bots from the live answer-fetchers such as ChatGPT-User and PerplexityBot, so each is permitted or refused on its own User-agent line. Choose which AI crawlers may use your content for training or answers, and generate the robots.txt rules to enforce it — all in your browser.
Append these lines to the robots.txt at your site root (/robots.txt) — don’t overwrite the whole file. robots.txt is a voluntary standard: well-behaved AI crawlers honor it, but it cannot physically stop a crawler that ignores it.
Each checkbox is an AI crawler; checked means blocked. Start from a preset — Block all AI, Allow all AI, or Block training only (which blocks the model-training bots but leaves the live search and assistant bots free to fetch answers) — then fine-tune individual bots. Set the Disallow path (/ blocks your whole site; a folder like /blog/ blocks just that section), and the robots.txt rules build as you go. Copy the block or download robots.txt.
The tool covers the AI crawler user-agent tokens the major providers publish, split into two groups. The training crawlers, which collect content that feeds model training, are GPTBot (OpenAI), Google-Extended (Gemini and Vertex AI), Applebot-Extended (Apple Intelligence), ClaudeBot and the legacy anthropic-ai (Anthropic), CCBot (Common Crawl, which feeds many models), Bytespider (ByteDance), Meta-ExternalAgent, Amazonbot, cohere-ai, Diffbot, Omgilibot, Timpibot and ImagesiftBot. The search and assistant crawlers, which fetch a page live to answer a question, are OAI-SearchBot and ChatGPT-User (OpenAI), PerplexityBot and Perplexity-User, Claude-Web, YouBot, Meta-ExternalFetcher and DuckAssistBot. Block training only leaves that second group free, so your pages can still be fetched and cited in AI answers while staying out of training sets.
These lines belong in the robots.txt file at your site root, at /robots.txt. Append them to any rules you already have rather than replacing the file. Remember that robots.txt is a voluntary standard: the major AI companies publicly document that their crawlers obey it, but it is not technically enforced — for a hard guarantee, block the bots server-side by user agent or IP. Need general crawl rules too? Use the robots.txt generator, confirm the result with the robots.txt tester, and point the crawlers you do welcome at your best pages with an llms.txt file.