AI robots.txt Generator: allow AI search, block AI training
Pick a preset or choose bot by bot. Your robots.txt updates live and is ready to copy.
Checked = allowed. Unchecked = blocked.
Which AI bots should you allow?
| If you want | Allow | Can block |
|---|---|---|
| To be cited in ChatGPT | OAI-SearchBot, ChatGPT-User | GPTBot |
| To be cited in Perplexity | PerplexityBot, Perplexity-User | – |
| To be cited in Claude | Claude-SearchBot, Claude-User | ClaudeBot |
| Google AI Overviews | Googlebot | Google-Extended |
| Microsoft Copilot | Bingbot | – |
Remember: a bot with its own User-agent group ignores the * group, and the longest matching rule wins. After updating, run the Free AEO Checker to confirm AI search bots can reach your page.
AI robots.txt questions
How do I allow ChatGPT but block AI training?
Allow OAI-SearchBot and ChatGPT-User (they power ChatGPT search and user requests) and block GPTBot (OpenAI training). This generator does that with the "Allow AI search, block training" preset.
Does blocking Google-Extended remove me from Google AI Overviews?
No. Google-Extended only controls whether Google may use your content for Gemini training and grounding. AI Overviews are part of Google Search and use Googlebot.
Where do I put robots.txt?
At the root of your domain, so it loads at https://yourdomain.com/robots.txt. In WordPress, most SEO plugins let you edit it; otherwise upload the file with your host's file manager.
Do all AI bots obey robots.txt?
The major AI companies state that their crawlers respect robots.txt, but it is a voluntary standard. For hard blocking use your firewall or CDN bot controls.