robots.txt generator for your website
Choose which bots may crawl your site, block folders and AI training bots, and add your sitemap.
About robots.txt generator
Choose your settings, then copy or download robots.txt and put it at your site's root (e.g. https://example.com/robots.txt). List paths to keep out of crawlers one per line, such as /admin/ or /search.
robots.txt is a request that well-behaved bots follow, not security — protect private pages with a password, and use a noindex tag to keep a page out of Google.
FAQ
Can I stop AI training on my content?
Tick “Block AI bots” to add GPTBot, ClaudeBot, Google-Extended, CCBot and others; the major companies say they respect robots.txt.
More tools here
IP lookup IPv6 test SSL checker HTTP header checker SEO meta checker Heading checker Email checker llms.txt generator UTM builder Meta tag generator