What is robots.txt?

Robots.txt is a small text file in your site root that tells search engine crawlers which parts of your site they may and may not visit. It is guidance, not a password wall, but most well-behaved crawlers follow it.

Use it to keep crawlers away from admin areas, search result pages and files that waste their time, while letting them freely index your real content.

How to use this generator

1. Enter your site URL so the sitemap path is built automatically.
2. Set a rule for each crawler you want to control.
3. Add allow and disallow paths for each rule.
4. Add your sitemap URL, or let the tool suggest one.
5. Copy the file or download robots.txt.
6. Upload it to the root folder of your website.

Writing rules that make sense

• Start with a rule for all crawlers, then add specific ones.
• Disallow only what you genuinely want blocked.
• Use Allow before Disallow when one path inside a blocked folder should stay open.
• Test any rule change before leaving it live for a long time.

A warning about blocking things you need

If you block the folder that holds your CSS or JavaScript, search engines can still index the page but may render it poorly. Keep the file simple, and never use robots.txt to hide content you simply do not want ranked.