Pre-deployment policy test
Paste proposed rules and test important public and private paths before replacing the live file.
Crawling and indexing tool
Robots.txt Tester
Enter your website URL, choose a crawler, and test an exact path. Use live robots.txt or paste proposed rules to see which directive allows or blocks that request.
FreeNo loginNo data stored
Crawler diagnosis
See which exact group and rule applies to a named crawler.
Launch checklist
Test representative paths after moving a staging configuration into production.
What it checks
Use the result as evidence, not a score.
- Crawler-specific group selection
- Allow and Disallow path matching
- Longest matching rule with Allow winning an equal-length tie
- Live robots.txt or pasted pre-deployment rules
What it cannot check
- Whether a crawler will actually visit, index, cite, or recommend the page
- Undocumented crawler behavior or identity spoofing
- The business or legal intent behind an access policy
How it works
Input, inspect, improve, verify.
The tool processes what you enter, shows a plain-language result, and gives you a safer next step. Make the change on your website, deploy it, then test the live page again.
Understand the result
How to check robots.txt for Googlebot
Enter the site address in Website URL, keep Googlebot selected in Crawler, and enter the path in Path to test. Leave Optional pasted robots.txt empty to test the site’s live file, then select Test access.
Read the crawler, path, rules source, applicable groups, matching rule, and decision together. Select another supported crawler when its access policy is the question. This is a rules test, not a rendered Googlebot browser or evidence of actual crawler visits.
- Test the specific path rather than only the homepage.
- Non-empty pasted rules replace the live file for this test.
- Pasted rules are sent to the server for evaluation; the test does not publish them.
Understand the result
Find the rule behind a blocked-by-robots.txt result
To try a controlled example, paste three lines: User-agent: *, Disallow: /draft/, and Allow: /draft/example/. Put each directive on its own line. With Googlebot selected, test /draft/ first, then /draft/example/.
The broader draft path is disallowed; the more specific example path is allowed. Inspect the matching directive before editing a real rule, and preserve deliberate restrictions. A rules result cannot establish what Google currently has indexed or why an impression count changed.
- Use the related generator to prepare a draft rather than editing the live file during diagnosis.
- An Allowed result does not test authentication, firewalls, JavaScript rendering, or indexing.
- Clear pasted text before rechecking the live robots.txt file.
Understand the result
Compare a draft test with the published file
Before deployment, test important public paths and intentional restrictions against the proposed text. After deployment, remove pasted rules and repeat those tests against the live host. A successful draft test does not prove the same rules were published.
A matching group and rule explain this decision for the selected crawler and path. They do not prove that other crawlers follow the same rules, that a crawler is authentic, or that the whole site is ready for search.
- Use View example to check the Overhyped homepage with Googlebot.
- Use Clear to reset the URL, path, pasted text, and result.
- Review missing or unavailable live-file evidence before changing a crawl policy.
Popular use cases
Useful before and after publishing.
Pre-deployment policy test
Paste proposed rules and test important public and private paths before replacing the live file.
Crawler diagnosis
See which exact group and rule applies to a named crawler.
Launch checklist
Test representative paths after moving a staging configuration into production.
Pro tips
Small checks that prevent avoidable mistakes.
- Test the crawler and exact path that matter; the home page alone is not enough.
- Use specific Allow exceptions carefully inside broader Disallow patterns.
- Keep training-crawler policy separate from search and answer-retrieval access.
- Remember that robots.txt controls crawling, not guaranteed indexing or removal.
Common issues and solutions
What to review when the result looks wrong.
No matching group appears
The crawler uses default open behavior unless a wildcard group applies.
An Allow rule wins unexpectedly
The most specific matching path wins, and Allow wins when matching rules have equal length.
The live file differs from the pasted test
Clear CDN caches and re-fetch the public /robots.txt response before relying on the change.
Frequently asked questions
Short answers before you make a change.
Is this an official Google robots.txt tester?
No. It is an Overhyped tool for evaluating declared rules. Use Search Console for Google’s property-specific reports and indexing evidence.
Does Allowed mean Google can index the page?
No. It means the evaluated robots rules allow this crawler and path. Other access controls and indexing signals need separate checks.
Why does my pasted test differ from the live result?
Non-empty pasted text is tested instead of the fetched file. Clear that text, confirm the host and exact path, and test again after deployment.
Does Disallow remove a URL from Google?
No. It controls crawling and is not a guaranteed removal or noindex mechanism.
Can I test AI crawlers?
Yes, when you choose a documented crawler name. Declared access still does not prove visits or citations.
Are robots rules case-sensitive?
Path matching is generally case-sensitive, while user-agent matching is handled without case sensitivity.
Read the practical guide
How robots.txt rule matching actually works
Understand the evidence behind this tool, where its result stops, and the safer next steps to verify on a live page.
Read the guideEvidence · Limitations · Verification