robots.txt Generator

Build a valid robots.txt for your site — allow everything, block everything, or set custom allow and disallow rules with a crawl-delay and sitemap URL. Copy or download, all in your browser.

How to use the robots.txt generator

Pick how you want crawlers treated: Allow all lets every compliant crawler fetch every path, Block all keeps them out entirely, and Custom rules lets you target a specific User-agent and list the paths to Disallow (and optionally Allow). Add a Sitemap URL so crawlers can find your sitemap, and a Crawl-delay if you want to throttle aggressive bots. The file updates as you type — copy it or download it, then place it at the root of your domain so it is served at yoursite.com/robots.txt.

Remember that robots.txt is a crawling directive, not a security control: it asks well-behaved crawlers not to fetch paths, but it does not hide anything from people or from bots that ignore it. To keep a page out of search results, use a noindex tag instead. Counting words on a page instead? Try the word counter.

Frequently Asked Questions

What is a robots.txt file?
robots.txt is a plain text file at the root of your site (example.com/robots.txt) that tells search-engine crawlers which paths they may or may not request. It uses User-agent lines to target specific crawlers and Allow/Disallow lines to permit or block paths. It is a crawling directive, not a security control — it does not hide pages from people or non-compliant bots.
Where do I put the robots.txt file?
At the very root of your domain, so it is served at example.com/robots.txt. Each subdomain needs its own file. Once uploaded, most search engines pick up changes within a day or two; you can also request a re-fetch in their webmaster tools.
Does Disallow hide a page from Google?
Not exactly. Disallow stops compliant crawlers from fetching a path, but a blocked URL can still appear in results if other pages link to it. To keep a page out of results, use a noindex meta tag or header (and don't also block it in robots.txt, or the crawler can't see the noindex), or protect it behind authentication.
Should I list my sitemap in robots.txt?
Yes — it's a helpful, widely supported way to point crawlers at your sitemap: add a line like Sitemap: https://example.com/sitemap.xml. This tool adds it when you fill in the sitemap URL. You can still also submit the sitemap directly in each search engine's webmaster tools.