—
- Effective group—
- Matched rule—
- Tested path—
This implements the standard matching rules Google and Bing document, but individual crawlers can differ in edge cases. robots.txt is advisory — only well-behaved bots obey it.
robots.txt is the Robots Exclusion Protocol file specified by RFC 9309: a crawler obeys the single group whose User-agent token best matches its name, and inside that group the longest matching Allow or Disallow path decides the URL, with Allow taking a tie. Paste a robots.txt and test any URL against any crawler user-agent — the verdict updates live to show whether the page is allowed or blocked, and which rule decided it. Everything runs in your browser.
This implements the standard matching rules Google and Bing document, but individual crawlers can differ in edge cases. robots.txt is advisory — only well-behaved bots obey it.
Paste your robots.txt into the box, type the URL or path you want to check, and pick the crawler user-agent — the verdict updates as you type. The tester follows the standard algorithm: it picks the most specific User-agent group that matches your chosen crawler — matching is case-insensitive and the longest matching token wins, falling back to the * group — then compares every Allow and Disallow rule in that group against the path. The rule with the longest matching path wins, and on a tie Allow wins. Wildcards * (any sequence) and $ (end of URL) are supported, an empty Disallow: is treated as allow-all and never wins, and a path with no matching rule is allowed. Comments after # are stripped, and Sitemap: lines are ignored because they are informational rather than access rules.
Need to build the file first? Use the robots.txt generator, or manage AI crawlers specifically with AI Crawler Control. To build the file that a Sitemap: line should point at, use the sitemap generator.