Pre-deployment policy test
Paste proposed rules and test important public and private paths before replacing the live file.
Crawling and indexing tool
Robots.txt Tester
Test whether a named crawler can request a path under the declared robots rules. The result shows the selected group and longest matching rule so the decision is inspectable.
FreeNo loginNo data stored
Crawler diagnosis
See which exact group and rule applies to a named crawler.
Launch checklist
Test representative paths after moving a staging configuration into production.
What it checks
Use the result as evidence, not a score.
- Crawler-specific group selection
- Allow and Disallow path matching
- Longest matching rule with Allow winning an equal-length tie
- Live robots.txt or pasted pre-deployment rules
What it cannot check
- Whether a crawler will actually visit, index, cite, or recommend the page
- Undocumented crawler behavior or identity spoofing
- The business or legal intent behind an access policy
How it works
Input, inspect, improve, verify.
The tool processes what you enter, shows a plain-language result, and gives you a safer next step. Make the change on your website, deploy it, then test the live page again.
Popular use cases
Useful before and after publishing.
Pre-deployment policy test
Paste proposed rules and test important public and private paths before replacing the live file.
Crawler diagnosis
See which exact group and rule applies to a named crawler.
Launch checklist
Test representative paths after moving a staging configuration into production.
Pro tips
Small checks that prevent avoidable mistakes.
- Test the crawler and exact path that matter; the home page alone is not enough.
- Use specific Allow exceptions carefully inside broader Disallow patterns.
- Keep training-crawler policy separate from search and answer-retrieval access.
- Remember that robots.txt controls crawling, not guaranteed indexing or removal.
Common issues and solutions
What to review when the result looks wrong.
No matching group appears
The crawler uses default open behavior unless a wildcard group applies.
An Allow rule wins unexpectedly
The most specific matching path wins, and Allow wins when matching rules have equal length.
The live file differs from the pasted test
Clear CDN caches and re-fetch the public /robots.txt response before relying on the change.
Frequently asked questions
Short answers before you make a change.
Does Disallow remove a URL from Google?
No. It controls crawling and is not a guaranteed removal or noindex mechanism.
Can I test AI crawlers?
Yes, when you choose a documented crawler name. Declared access still does not prove visits or citations.
Are robots rules case-sensitive?
Path matching is generally case-sensitive, while user-agent matching is handled without case sensitivity.
Read the practical guide
Read the guide →