AI crawlers and robots.txt, a decision with a trade-off.

There are two confident camps on this and both are oversimplifying. Blocking AI crawlers protects content from being used without compensation. It also removes you from the answers those systems generate. Which matters more depends on your business model, and it is worth deciding deliberately rather than by default.

What the crawlers do differently

It helps to separate two functions that often use different user agents. Some crawling gathers training data. Other crawling happens live, at the moment a user asks a question, to retrieve current information.

That distinction matters because the trade-off differs. Blocking training crawlers protects your content from being absorbed into a model. Blocking retrieval crawlers removes you from answers being generated right now, which is a much more immediate commercial cost.

Publishers frequently want to block the first and allow the second. That is a coherent position, and whether it is achievable depends on the specific agents each operator documents.

Who should consider blocking

• Publishers whose product is the content itself. If people pay for access to what you write, having it summarized for free is a direct substitution.
• Sites with genuinely proprietary research. Where the material is the moat.
• Anyone under contractual restrictions. Licensed content may not be yours to expose.

For almost everyone else — service businesses, local companies, ecommerce, B2B — blocking is usually the wrong call. Your content is marketing, and being summarized with attribution is the outcome you want.

Example robots.txt configurations

For a service business that wants to appear in AI answers but prefers not to contribute to model training, a reasonable configuration allows the search and user agents and disallows the training agents:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

For most businesses, though, the simplest choice is best: allow everything, because being summarized with attribution is the goal. Either way, write the decision down with the date and the reasoning, so the next person to edit the file knows it was deliberate.

Checking what actually reaches you

Before deciding anything, look at your server logs. Filter for the documented user agents of the major operators and see what is genuinely crawling, how often, and what it fetches.

This frequently produces surprises in both directions: crawlers nobody expected, and absent crawlers explaining why a site is invisible in a particular assistant. It is also the cheapest diagnostic in this whole discipline, and it requires no tool.

Check too whether anything is blocking them unintentionally — a CDN rule, a bot-protection service, or a robots.txt line inherited from a previous team. Unintentional blocking is a more common cause of AI invisibility than most people assume.

What robots.txt cannot do

robots.txt is a request, not a lock. Reputable operators follow it, but it does not prevent access, and user-triggered fetchers may be treated differently because a person asked for the page. It also does not remove content already collected; it only affects future crawling.

If content genuinely must not be accessed, it needs authentication, not a robots.txt line. And if the goal is to keep content out of search results, use noindex rather than robots.txt, because a blocked page cannot be crawled to see the noindex instruction.

Reviewing the decision

The crawler landscape changes: new operators appear, existing ones split or rename agents, and the value of each assistant as a traffic source shifts. Review your robots.txt and firewall rules every six months against the operators' current documentation.

Compare it with what the logs show and with what your monitoring shows about citations. If you are blocking something and still appear in that assistant, or allowing something and never appear, the reason is worth understanding before the next review.

Frequently asked questions

Should I block AI crawlers in robots.txt?

For most service businesses, no. Your content is marketing and being cited is the goal. Blocking makes sense mainly for publishers whose product is the content itself, sites with proprietary research, or anyone under contractual restrictions on the material.

Does blocking AI crawlers hurt SEO?

It does not directly affect traditional search rankings, since those crawlers are separate. It does remove you from AI-generated answers, which is an increasingly significant discovery channel in its own right.

How do I know which AI crawlers visit my site?

Check server logs for the documented user agents of the major operators. This is the cheapest diagnostic available and frequently reveals both unexpected crawlers and unintentional blocking by a CDN or bot protection service.

What is the difference between GPTBot and OAI-SearchBot?

OpenAI documents GPTBot as the crawler used for training data and OAI-SearchBot as the crawler for ChatGPT search results. They can be allowed or blocked separately.

Can I block AI crawlers without blocking Google Search?

Yes. Google-Extended controls use of content for Gemini training and grounding without affecting Googlebot and Google Search.