robots.txt Tester

Paste a robots.txt and test any URL against any crawler user-agent — the verdict updates live to show whether the page is allowed or blocked, and which rule decided it. Everything runs in your browser.

—
  • Effective group—
  • Matched rule—
  • Tested path—

This implements the standard matching rules Google and Bing document, but individual crawlers can differ in edge cases. robots.txt is advisory — only well-behaved bots obey it.

How to use the robots.txt tester

Paste your robots.txt into the box, type the URL or path you want to check, and pick the crawler user-agent — the verdict updates as you type. The tester follows the standard algorithm: it picks the most specific User-agent group that matches your chosen crawler (falling back to the * group), then compares every Allow and Disallow rule in that group against the path. The rule with the longest matching path wins, and on a tie Allow wins. Wildcards * (any sequence) and $ (end of URL) are supported.

Need to build the file first? Use the robots.txt generator, or manage AI crawlers specifically with AI Crawler Control.

Frequently Asked Questions

What is a robots.txt file?
A robots.txt file is a plain text file at the root of your domain, served from /robots.txt, that tells web crawlers which parts of your site they may or may not request. It uses simple User-agent, Disallow and Allow rules plus an optional Sitemap line. It controls crawling, not indexing or access: it is a set of instructions that well-behaved crawlers choose to follow.
Is robots.txt necessary, and is it still used?
Yes, robots.txt is still an active, widely supported standard. Google, Bing and the major AI crawlers all read it, and it was formalized as RFC 9309 in 2022. It is not strictly required: a site with no robots.txt is simply treated as fully crawlable. You want one when you need to steer crawlers away from low-value or duplicate paths, point them to your sitemap, or manage crawl budget on a large site. Remember it only guides crawling, so use a noindex tag to keep a page out of search results and authentication to truly block access.
How does robots.txt decide allowed vs blocked?
A crawler first picks the most specific group whose User-agent matches its name, falling back to the User-agent: * group if there is one. Within that group it looks at every Allow and Disallow rule whose path matches the URL, and the rule with the longest matching path wins. If an Allow and a Disallow match with the same length, Allow wins. If no rule matches, the URL is allowed.
Does a Disallow guarantee a page won't be crawled?
No. robots.txt is advisory. Well-behaved crawlers such as Googlebot and Bingbot obey it, but it is not an access control and cannot force a crawler to stay away. To truly block access, use authentication or server-side rules; to keep a page out of search results, use a noindex directive on a page crawlers are allowed to fetch.
What do * and $ mean in robots.txt?
Inside a path, * matches any sequence of characters, so Disallow: /*.pdf blocks any URL containing .pdf. A $ anchors the rule to the end of the URL, so Disallow: /*.pdf$ blocks only URLs that end in .pdf. These wildcards are supported by Google and Bing and this tester implements them the same way.
Is my robots.txt uploaded?
No. Everything runs in your browser. The robots.txt you paste, the URL you test and the user-agent you choose never leave your device and are not uploaded, logged or stored.

Related SEO tools

robots.txt Generator

Build a valid robots.txt with allow/disallow rules, crawl-delay and sitemap URL.

SEO Tools

→

AI Crawler Control

Allow or block GPTBot, ClaudeBot and other AI crawlers with ready-made rules.

SEO Tools

→

Sitemap Generator

Paste your page URLs and build a valid sitemap.xml with lastmod and priority.

SEO Tools

→

XML Sitemap Validator

Check your sitemap.xml for valid structure, URL limits and common errors.

SEO Tools

→

Meta Tag Generator

Create SEO title, description, Open Graph and Twitter Card tags for your page.

SEO Tools

→