Control AI crawlers without losing citations.
Pick which AI search bots, user-triggered fetchers and training crawlers can reach your site, and get a clean robots.txt to paste in.
Choose your crawlers
Ticked bots are allowed, unticked bots are blocked. Start from a preset.
Your robots.txt
Save as robots.txt at the root of your domain.
Search bots, user fetchers and training bots
AI companies increasingly separate their crawlers by purpose. Search bots build the index an assistant cites from; block them and you disappear from those answers. User-triggered fetchers visit a page because a person asked about it. Training bots collect content that may be used to train future models; blocking them does not remove you from search-based answers.
For most businesses, being cited is the goal, so allowing everything is the simplest choice. Publishers whose product is the content itself often allow search and block training. Crawler names change, so check each company's documentation before relying on this list, and remember that robots.txt is a request that reputable crawlers follow, not access control.
More detail: AI crawlers and robots.txt, a decision with a trade-off.
Frequently asked questions
Should I block AI crawlers?
For most businesses, no. Being cited by AI assistants is a growing source of visibility. Blocking search bots removes you from their answers. Publishers who sell their content sometimes block training bots while allowing search bots.
Does blocking GPTBot remove me from ChatGPT?
No. OpenAI documents GPTBot for training and OAI-SearchBot for ChatGPT search results; they are controlled separately.
Does Google-Extended affect AI Overviews?
No. Google-Extended controls use of content for Gemini training and grounding. AI Overviews are part of Google Search and use Googlebot.
Where does robots.txt go?
At the root of your domain, for example yourcompany.com/robots.txt. Each subdomain needs its own file.
Related reading
- AI crawlers and robots.txt
- The technical floor for AI search
- llms.txt generator
Want us to do this for you?
Get a free AI visibility report: how ChatGPT, Gemini and Google AI describe your business, and what to fix first.
