Free, no sign-up
Built in your browser
Works offline
The LK Forge robots.txt Generator assembles the file crawlers look for at your site root, writing the User-agent, Allow, Disallow, Crawl-delay and Sitemap lines for you. Build a valid robots.txt for your site — allow everything, block everything, or set custom allow and disallow rules with a crawl-delay and sitemap URL. Copy or download, all in your browser.
Pick how you want crawlers treated: Allow all lets every compliant crawler fetch every path, Block all keeps them out entirely, and Custom rules lets you target a specific User-agent and list the paths to Disallow (and optionally Allow). Add a Sitemap URL so crawlers can find your sitemap, and a Crawl-delay if you want to throttle aggressive bots. The file updates as you type — copy it or download it, then place it at the root of your domain so it is served at yoursite.com/robots.txt.
Remember that robots.txt is a crawling directive, not a security control: it asks well-behaved crawlers not to fetch paths, but it does not hide anything from people or from bots that ignore it. To keep a page out of search results, use a noindex tag instead. Already have a file live? Check whether a URL is blocked with the robots.txt tester, and build the file it points at with the XML sitemap generator.