Skip to content
Toolzami
Search tools
Menu

Robots.txt Validator

Paste a robots.txt file to check every line against RFC 9309 and Google's rules, then test a URL for Googlebot, an AI crawler or any user agent: see whether it may be crawled, which line decides and which group applies. Nothing is fetched or sent.

Runs locally in your browser

Input

12 lines · 183 characters
# robots.txt for www.claystudio.example
User-agent: *
Disallow: /checkout/
Disallow: /search
Allow: /search/help
Crawl-delay: 5

User-agent: GPTBot
Disallow: /

Sitemap: /sitemap.xml

Test a URL

Enter a path or a URL to see whether the crawler may fetch it.

What robots.txt does

robots.txt is a plain-text file at the root of a site, such as https://example.com/robots.txt, that tells crawlers which URLs of that host they may fetch. It covers only the host, protocol and port that serve it: https://shop.example.com/ needs a file of its own. Crawlers that follow the standard, RFC 9309 (opens in a new tab), read it before they crawl. It is a request, not access control: anyone can still open the pages, so it is no way to hide private content.

robots.txt also controls crawling, not indexing. Google can list a URL it may not crawl, without a description, when other pages link to it. To keep a page out of search, let crawlers fetch it and add a noindex robots meta tag or X-Robots-Tag header.

How to test a robots.txt file

  1. Paste your robots.txt, open the file, or start from a template.
  2. Fix the lines the editor marks: crawlers skip a line with an error and read the rest.
  3. Enter a path or a full URL, and choose a crawler or type another user agent.
  4. Read the verdict: whether the crawler may fetch the URL, which line decides and which group it follows. Show the rule selects that line.
  5. Download the file as robots.txt and upload it to the root of your site.

Examples

An Allow that makes an exception, tested for Googlebot

Input

User-agent: *
Disallow: /admin/
Allow: /admin/public/

Output

Googlebot /admin/public/a  Allowed  Allow: /admin/public/ (line 3)
Googlebot /admin/x  Blocked  Disallow: /admin/ (line 2)
Googlebot /  Allowed  no rule matches
A wildcard and an end anchor

Input

User-agent: *
Disallow: /*.pdf$

Output

Googlebot /files/a.pdf  Blocked  Disallow: /*.pdf$ (line 2)
Googlebot /files/a.pdf?dl=1  Allowed  no rule matches
A group of its own replaces the * group

Input

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

Output

Googlebot /page  Allowed  Allow: / (line 5)
GPTBot /page  Blocked  Disallow: / (line 2)

Line 2, column 1: Disallow: / in the * group blocks the whole site for every crawler without a group of its own. That's right for a staging site; on a live site it keeps pages out of search. (ROBOTS_BLOCK_ALL)

How a crawler picks its group and its rule

  • Groups. A group starts with one or more User-agent lines, and the Allow and Disallow lines after them belong to it, up to the next User-agent line. Blank lines, comments and sitemaps don't end a group.
  • The crawler's group. A crawler looks for the groups that name its product token, the name before any version: Googlebot/2.1 is googlebot, in any case. Groups that name the same crawler are read as one.
  • The * group. A crawler without a group of its own follows the * group; with no * group either, nothing is blocked. It never adds the * group's rules to its own.
  • Documented exceptions. Google's AdsBot and Mediapartners-Google ignore the * group, and Applebot follows Googlebot's group when no group names Applebot.
  • The deciding rule. Within the group, the rule with the longest path decides, counted in characters with any * and $. When an Allow and a Disallow are equally long, the Allow wins. With no matching rule the URL is allowed, and /robots.txt itself always is.

Wildcards, the end of a path, and encoding

  • A rule matches the start of the URL's path and query: Disallow: /fish blocks /fish, /fish.html and /fish/salmon?size=2, but not /Fish, since paths are case-sensitive.
  • * stands for any run of characters, slashes included: Disallow: /*.pdf blocks PDF files in every folder.
  • $ at the end means the path ends there: Disallow: /*.pdf$ blocks /a.pdf but not /a.pdf?dl=1. A * at the end adds nothing.
  • Rules and URLs are compared percent-encoded: /ü and /%C3%BC are the same path, and so are /%7Ea and /~a, while %2F stays different from /. To match a * or $ that is part of a URL, write %2A or %24.
  • The host and the part after # are never compared.

Directives only some crawlers read

Google reads four directives: user-agent, allow, disallow and sitemap. A sitemap line takes the sitemap's full URL, may appear anywhere and as often as needed, and belongs to no group. Other directives depend on the crawler:

  • Crawl-delay asks a crawler to wait between requests. Google ignores it, Anthropic's crawlers honor it, and Applebot doesn't.
  • Host named a site's main mirror for Yandex; Yandex's robots.txt guide no longer lists it.
  • Clean-param tells Yandex which URL parameters, such as UTM tags, to ignore when indexing.
  • Noindex isn't supported in robots.txt. Use the robots meta tag or the X-Robots-Tag header.

AI crawlers and training controls

AI companies document separate crawlers for training, for their search features and for pages a user asks for, and some publish a token that only controls how crawled pages may be used. As their operators describe them in November 2026:

AI crawlers and tokens, by operator
TokenOperatorWhat it does
Google-ExtendedGoogleWhether content may train and ground Gemini and Vertex AI; no effect on Search. A control token: no crawler sends it.
GPTBotOpenAITraining OpenAI's models.
OAI-SearchBotOpenAIChatGPT search results.
ChatGPT-UserOpenAIPages ChatGPT users ask for. OpenAI says robots.txt rules may not apply to these user-initiated fetches.
ClaudeBotAnthropicTraining Anthropic's models.
Claude-SearchBotAnthropicClaude's search results.
Claude-UserAnthropicPages Claude users ask for.
PerplexityBotPerplexityPerplexity's search results; not used for training.
Perplexity-UserPerplexityPages Perplexity users ask for. Perplexity says this fetcher generally ignores robots.txt.
Applebot-ExtendedAppleWhether content may train Apple's generative models. A control token: no crawler sends it.

Blocking a training token doesn't change search: Google says Google-Extended has no effect on Google Search, and Apple says it doesn't use Applebot-Extended rules for ranking. Fetches a user asks for may not follow robots.txt, as OpenAI and Perplexity say of theirs.

Common errors and fixes

Common errors and how to fix them
CodeWhat it meansHow to fix it
ROBOTS_MISSING_COLONErrorA line has no colon after its name, like Disallow /private/. The standard says to skip it, though Google's parser reads it as if the colon were there.Write a name, a colon and a value: Disallow: /private/.
ROBOTS_RULE_OUTSIDE_GROUPErrorAn Allow or Disallow line comes before the first User-agent line, so it belongs to no group.Put a User-agent line above it, such as User-agent: *.
ROBOTS_EMPTY_USER_AGENTErrorA User-agent line has no value.Name a crawler, or write * for every crawler without a group of its own.
ROBOTS_UNKNOWN_DIRECTIVEWarningA line's name isn't a robots.txt directive, often a typo such as Dissallow.Use the directive the problem suggests, or remove the line.
ROBOTS_UNSUPPORTED_DIRECTIVENoteCrawl-delay, Host, Clean-param or Noindex: directives Google doesn't read and other crawlers read differently.Keep it only for the crawlers that read it; the note says which.
ROBOTS_RELATIVE_SITEMAPWarningA Sitemap line has a path, such as /sitemap.xml, instead of a full URL.Write the sitemap's full address, with https:// and the host.
ROBOTS_PATH_NOT_ABSOLUTEWarningA rule's pattern doesn't start with / or *, so it can't match a URL's path.Start the pattern with /.
ROBOTS_BLOCK_ALLWarningDisallow: / in the * group blocks the whole site for every crawler without a group of its own.Keep it for a staging site only.
ROBOTS_GROUPS_MERGEDNoteTwo groups name the same crawler, so crawlers read their rules as one group.Keep one group per crawler.
ROBOTS_NO_GROUPSNoteThe file has no User-agent lines, so every crawler may fetch everything.Add a group, such as User-agent: * with the rules you want.
ROBOTS_TOO_LARGEWarningThe file is larger than the 500 KB Google reads; it ignores the rest.Shorten it: remove old rules, or cover paths with * patterns.
ROBOTS_TEST_URL_INVALIDErrorThe URL to test isn't a path starting with / or a full http:// or https:// URL.Enter a path such as /blog/, or a full URL.
INPUT_TOO_LARGEErrorThe file is longer than the 2 MB this tool checks.Check a smaller file, or the part you changed.

Privacy

Robots.txt Validator runs entirely in your browser. What you type, paste or open is processed on your device: it is not sent to Toolzami, and Toolzami does not store it. Only the options you choose are remembered, in this browser.

Frequently asked questions

Does blocking a page in robots.txt remove it from Google?

No. robots.txt controls crawling, not indexing: a blocked URL can still appear in results, without a description, when other pages link to it. To keep a page out of search, let Google crawl it and add a noindex robots meta tag or X-Robots-Tag header.

Which rule wins when Allow and Disallow both match?

The one with the longer path, counted in characters with any * and $. When an Allow and a Disallow are equally long, the Allow wins. A crawler reads only the group that names it, or the * group if none does, so rules of other groups never count.

How do I block AI training without leaving search?

Disallow the tokens that control training, such as GPTBot, ClaudeBot, Google-Extended and Applebot-Extended, and leave Googlebot and the search crawlers alone. Google-Extended and Applebot-Extended don't crawl: their rules only decide whether pages may train AI models, so search isn't affected.

Last updated