AI Crawler Control

Choose which AI crawlers may use your content for training or answers, and generate the robots.txt rules to enforce it — all in your browser.

AI training crawlers

Append these lines to the robots.txt at your site root (/robots.txt) — don't overwrite the whole file. robots.txt is a voluntary standard: well-behaved AI crawlers honor it, but it cannot physically stop a crawler that ignores it.

How to use AI Crawler Control

Each checkbox is an AI crawler; checked means blocked. Start from a presetBlock all AI, Allow all AI, or Block training only (which blocks the model-training bots but leaves the live search and assistant bots free to fetch answers) — then fine-tune individual bots. Set the Disallow path (/ blocks your whole site; a folder like /blog/ blocks just that section), and the robots.txt rules build as you go. Copy the block or download robots.txt.

These lines belong in the robots.txt file at your site root, at /robots.txt. Append them to any rules you already have rather than replacing the file. Remember that robots.txt is a voluntary standard: the major AI companies publicly document that their crawlers obey it, but it is not technically enforced — for a hard guarantee, block the bots server-side by user agent or IP. Need general crawl rules too? Use the robots.txt generator.

Frequently Asked Questions

Does robots.txt actually stop AI crawlers?
Only voluntarily. robots.txt is a request, not a wall. Major AI companies publicly document that their crawlers — such as GPTBot, ClaudeBot, Google-Extended and PerplexityBot — obey robots.txt, so adding a disallow rule reliably stops those well-behaved bots. But the standard is not technically enforced: it cannot physically prevent a crawler that chooses to ignore it. If you need a hard guarantee, block the bots server-side by user agent or IP instead.
What is Google-Extended?
Google-Extended is a separate robots.txt token that controls whether Google may use your content to train and ground its generative AI models, such as Gemini and Vertex AI. It is not a real crawler with its own user agent — it only governs AI use. Because it is separate from Googlebot, blocking Google-Extended does NOT remove your site from Google Search; your pages are still crawled and ranked normally.
Will blocking GPTBot remove my site from ChatGPT?
No. GPTBot is OpenAI's crawler for gathering training data, so blocking it stops your content being used to train future models. Live answers in ChatGPT are fetched by different agents — ChatGPT-User (browsing on a user's request) and OAI-SearchBot (OpenAI's search index) — which you can allow or block separately in this tool. Blocking GPTBot alone leaves those live-fetch bots unaffected.
Where do these rules go?
Append the generated lines to the robots.txt file at your site root, reachable at /robots.txt. Do not overwrite the whole file — add these User-agent and Disallow blocks alongside your existing rules. If you do not yet have a robots.txt, create one at the root and paste these rules in.