Robots.txt Generator
Generate a correct robots.txt file to control which pages search engines crawl.
How to Generate a Robots.txt File
- Choose "Allow All," "Custom Rules," or "Block All."
- For custom rules, list any paths to disallow or explicitly allow.
- Optionally add your sitemap URL.
- Copy the result and upload it as robots.txt at your site's root.
Frequently Asked Questions
Where do I put the generated robots.txt file?
Upload it to the root of your domain so it's reachable at yourdomain.com/robots.txt — search engines specifically look for it at that exact location and won't find it anywhere else.
Does blocking a page in robots.txt guarantee it stays out of Google's index?
Not entirely — robots.txt only tells crawlers not to fetch a page's content. If other sites link to that URL, Google can sometimes still index the URL itself (without its content) unless you also use a noindex meta tag on the page, which does require it to be crawlable.
What does the Sitemap line actually do?
It's a courtesy pointer telling any crawler exactly where your sitemap.xml lives, so it can find and process all your important URLs efficiently, rather than relying purely on discovering links by crawling around your site.
Is the information I enter sent anywhere?
No — the file is built entirely in your browser using JavaScript, so nothing you enter is sent to a server.