New website baseline
Start with a readable, conservative file and add only intentional restrictions.
Crawling and indexing tool
Robots.txt Generator
Enter the paths you intend to disallow, any allow exceptions, and an optional sitemap URL. Review the generated robots.txt draft and test it before publishing.
FreeNo loginNo data stored
Staging review
Prepare a production draft without copying a whole-site staging block by mistake.
Policy cleanup
Replace duplicated rules with a smaller file that can be tested path by path.
What it checks
Use the result as evidence, not a score.
- One wildcard crawler group
- Normalized Allow and Disallow paths
- Optional absolute sitemap declaration
- A visible warning when Disallow: / blocks the whole site
What it cannot check
- Which paths your business should expose or protect
- Whether sensitive content is secured by authentication
- Indexing, ranking, crawler visits, or legal policy
How it works
Input, inspect, improve, verify.
The tool processes what you enter, shows a plain-language result, and gives you a safer next step. Make the change on your website, deploy it, then test the live page again.
Understand the result
How to create a robots.txt file with this generator
Enter one path per line in Paths to disallow, one per line. Add any narrower exceptions in Exceptions to allow, one per line. If you have a sitemap, enter its complete public URL in Sitemap URL. The draft updates as you type.
Use Copy file to copy the text into a plain-text file named robots.txt. This generator runs in your browser and creates one User-agent: * group. It does not inspect an existing file, choose your site’s policy, or upload changes.
- Keep intentional rules from the existing file when reviewing a replacement.
- Blank disallow inputs produce Allow: /; they do not copy restrictions from your website.
- Use the related Robots.txt Tester before deployment.
Understand the result
An example robots.txt file with an allow exception
For a hypothetical site, put /draft/ in Paths to disallow and /draft/example/ in Exceptions to allow. The draft contains User-agent: *, Disallow: /draft/, and Allow: /draft/example/, each on its own line.
The narrower allow rule is intended to keep that example path crawlable inside the broader blocked path. Test both paths for the crawlers that matter before using the file. This example demonstrates rule structure, not a recommendation to block a real directory.
- Entering / as a disallowed path triggers a whole-site warning.
- The generator normalizes missing leading slashes but does not validate every policy or rule pattern.
- Robots.txt is public crawler guidance; protect confidential content with authentication.
Understand the result
Review the default robots.txt draft before replacing a live file
View example fills the Overhyped sitemap URL without adding disallowed paths. Clear removes your inputs and returns the open draft. Copying either draft is a preparation step; it does not change your website.
Publish the reviewed UTF-8 text file at the root of the host it should govern, then fetch and test the live file. CMS and hosted-platform settings can generate or override robots.txt, so identify the existing owner before replacing it.
- This tool is not a WordPress or Blogger plugin.
- It does not generate crawler-specific groups or Crawl-delay.
- Crawling permissions do not guarantee indexing, traffic, or visits.
Popular use cases
Useful before and after publishing.
New website baseline
Start with a readable, conservative file and add only intentional restrictions.
Staging review
Prepare a production draft without copying a whole-site staging block by mistake.
Policy cleanup
Replace duplicated rules with a smaller file that can be tested path by path.
Pro tips
Small checks that prevent avoidable mistakes.
- Never use robots.txt as access control for private or sensitive information.
- Avoid Disallow: / in production unless blocking the entire site is deliberate.
- Use complete sitemap URLs and publish the file at the site root.
- Deploy, fetch the live file, and test important paths with named crawlers.
Common issues and solutions
What to review when the result looks wrong.
The entire site is blocked
Remove Disallow: / unless the whole-site restriction is intentional and approved.
Private content still opens
Robots.txt is public guidance for crawlers, not authentication. Protect private content at the application layer.
A crawler ignores Crawl-delay
Not every crawler supports that directive. This generator leaves it out of the safe minimum.
Frequently asked questions
Short answers before you make a change.
Can I use this robots.txt generator for WordPress or Blogger?
It creates a plain-text draft that you can review for your site. It does not install a plugin or change hosting settings. Check whether your CMS or host already manages the file before replacing it.
Why does the default draft contain Allow: /?
With no disallowed paths, this generator outputs an open wildcard group. It has not fetched your existing rules. Review your intended policy before publishing the draft.
Can I generate rules for a specific crawler?
This generator creates one User-agent: * group. It does not create separate crawler groups. Use a reviewed custom configuration when different crawlers need different policies.
Does the generator upload the file?
No. It creates text in your browser. You choose whether and where to deploy it.
Should every site list a sitemap?
It can help discovery, especially on larger or changing sites, but it is not required for every website.
Can robots.txt protect confidential pages?
No. Use authentication and authorization. A robots.txt file is public and its rules are voluntary crawler controls.
Read the practical guide
How to create a safe robots.txt file
Understand the evidence behind this tool, where its result stops, and the safer next steps to verify on a live page.
Read the guideEvidence · Limitations · Verification