Robots.txt generator

Builds a robots.txt: what to hide from search engines and AI crawlers, where the sitemap is.

Template

Disallow (one path per line)Allow despite the rules aboveSitemap.xml address

AI crawlers

Stop AI crawlers from collecting the site for model training

Google and Bing search crawlers keep working as usual. GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, PerplexityBot, Bytespider, meta-externalagent, Amazonbot

robots.txt

User-agent: *
Disallow: /admin/
Disallow: /search
Disallow: /*?utm_

Sitemap: https://example.com/sitemap.xml

Put the file at the root of the site so it opens at /robots.txt. Robots.txt is a request, not protection: put pages with personal data behind a password.

Robots.txt generator

Template

AI crawlers

robots.txt

Other tools