A robots.txt file can look like a top-to-bottom list of instructions. Crawlers do not simply stop at the first line that resembles the URL.
They first select the applicable crawler group, then compare matching path rules by specificity.
Select the crawler group first
A group begins with one or more User-agent lines followed by its rules. A crawler looks for the most specific user-agent token that matches its name. The wildcard group applies when no more specific group matches.
For example:
User-agent: ExampleBot
Disallow: /private/
User-agent: *
Allow: /ExampleBot uses its named group rather than combining it with the wildcard group. Google documents this group selection in its robots.txt specification guidance.
Use the longest matching path
Within the applicable group, compare every Allow and Disallow path that matches the requested URL path. The rule matching the most characters is the most specific.
User-agent: *
Disallow: /private/
Allow: /private/public-example/The path /private/public-example/page matches both rules. The longer Allow rule wins, so the crawler is allowed to request that path.
The standardized Robots Exclusion Protocol defines this most-specific-match behaviour in RFC 9309.
Allow wins an equal-length conflict
When an Allow and Disallow rule match the same number of characters, the Allow rule is used under the RFC model. Google describes the same least-restrictive result for equal rules.
Rule order is therefore not a reliable way to resolve a tie. Make the intended exception explicit and test the exact path.
Wildcards and end anchors need care
Google supports * as a wildcard and $ as an end-of-path marker in rule values. A pattern such as /*.pdf$ is narrower than a plain / block, but small punctuation changes can alter which paths match.
Crawler implementations are not identical. Test named crawlers against the syntax they document rather than assuming every agent supports every extension.
No matching rule normally means allowed
An empty Disallow value does not block the site. When no applicable Allow or Disallow path matches, crawling is normally allowed.
That does not guarantee that the crawler will visit the page. It only means this robots.txt evaluation did not prohibit the request.
What a robots test cannot prove
A deterministic tester can identify the applicable group, matching rule, and resulting Allow or Disallow decision for the text it receives. It cannot prove that a crawler will visit, render, index, cite, or rank the page.
Robots.txt is also not authentication. The file is public, and disallowed URLs can still be discovered from links. Google’s robots.txt introduction warns against using it to keep web pages out of Search or to protect sensitive information.
A safe rule-testing workflow
- Name the exact crawler you want to test.
- Use the URL path, including relevant query text when the crawler’s syntax supports it.
- Test the current live file and any proposed replacement separately.
- Record the selected group and exact matching rule.
- Check important public paths as well as intended blocked paths.
- Deploy the file at the site root as
/robots.txt. - Fetch the public file and repeat the same tests.
Use the free Robots.txt Tester to evaluate a named crawler and path against live or pasted rules.
Method note: the tester uses deterministic group and longest-path matching for the supplied crawler and rules. It reports policy evidence, not crawler behaviour or indexing. Last reviewed 23 August 2026.
