
AI companies use three kinds of bots: search crawlers that decide whether you can be cited, training crawlers that collect content for future models, and user-initiated fetchers that visit a page when someone asks a question. Google and Apple also use control tokens that do not crawl at all. This list covers every agent we could find documented on the vendor's own page, checked in November 2026, with the exact robots.txt token and a link to the source. Agents without vendor documentation are left out.
How to read this list
- Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the index an assistant searches. Block one and you drop out of that assistant's answers.
- Training crawlers (GPTBot, ClaudeBot, CCBot) collect content for future models. Blocking them does not remove you from today's search answers.
- User-initiated fetchers (ChatGPT-User, Claude-User, Perplexity-User) visit a page because a person asked. Several vendors say robots.txt may not apply to these.
- Control tokens (Google-Extended, Applebot-Extended) have no crawler of their own. They tell the company how it may use pages its main crawler already fetched.
Whether to block anything is a business decision, covered in AI crawlers and robots.txt. This page is the reference.
The full list
| Company | robots.txt token | Type | What it does (vendor's words, paraphrased) | Follows robots.txt? | Source |
|---|---|---|---|---|---|
| OpenAI | OAI-SearchBot | Search | Surfaces sites in ChatGPT search answers | Yes; changes take about 24 hours | Docs |
| OpenAI | GPTBot | Training | Crawls content that may train OpenAI's foundation models | Yes | Docs |
| OpenAI | ChatGPT-User | User fetch | Visits pages when a ChatGPT user or custom GPT asks | May not apply | Docs |
| OpenAI | OAI-AdsBot | Ads check | Checks landing pages submitted as ChatGPT ads; not used for training | Only visits ad pages | Docs |
Googlebot | Search | Crawls for Google Search, including AI Overviews and AI Mode | Yes | Docs | |
Google-Extended | Control token | Controls Gemini training and grounding in Gemini Apps; no separate crawler | Yes; does not affect Search | Docs | |
Google-CloudVertexBot | Site-owner crawl | Crawls sites whose owners asked to build Vertex AI Agents | Yes | Docs | |
| Anthropic | ClaudeBot | Training | Collects web content for model training | Yes; supports Crawl-delay | Docs |
| Anthropic | Claude-SearchBot | Search | Improves Claude's search results | Yes | Docs |
| Anthropic | Claude-User | User fetch | Fetches pages when a person asks Claude a question | Yes; blocking stops retrieval | Docs |
| Perplexity | PerplexityBot | Search | Surfaces and links sites in Perplexity; not for training | Yes | Docs |
| Perplexity | Perplexity-User | User fetch | Visits a page to help answer a user's question | Generally ignores it | Docs |
| Microsoft | bingbot | Search | Crawls for Bing; no separate Copilot crawler is documented | Yes | Docs |
| Apple | Applebot | Search | Powers Spotlight, Siri and Safari search; may also be used for training | Yes; no Crawl-delay | Docs |
| Apple | Applebot-Extended | Control token | Opts content out of Apple foundation model training; does not crawl | Yes | Docs |
| Meta | meta-externalagent | Training / indexing | Trains foundation AI models or indexes content directly | Blocking examples given | Docs |
| Meta | meta-webindexer | Search | Improves Meta AI search results | Allow to be cited | Docs |
| Meta | meta-externalfetcher | User fetch | Fetches individual links at a user's request | May bypass it | Docs |
| Amazon | Amazonbot | Training / products | Improves Amazon products; may train Amazon AI models | Yes; no Crawl-delay | Docs |
| Amazon | Amzn-SearchBot | Search | Search experiences such as Alexa; not for training | Yes | Docs |
| Amazon | Amzn-User | User fetch | Fetches live information for Alexa queries | May not follow all rules | Docs |
| Common Crawl | CCBot | Open archive | Builds an open web archive widely used for AI training | Yes | Docs |
| DuckDuckGo | DuckAssistBot | Search | Real-time crawling for DuckDuckGo's AI-assisted answers; not for training | Yes; takes 72 hours | Docs |
| Mistral | MistralAI-Index | Search | Indexes the web for Mistral search; not for training | Yes | Docs |
| Mistral | MistralAI-Training | Training | Builds datasets to train Mistral models | Yes | Docs |
| Mistral | MistralAI-User | User fetch | Visits pages when a user asks Mistral's Vibe a question | Governs user requests | Docs |
Write the tokens exactly as shown. "Follows robots.txt?" reflects each vendor's own statement, not independent testing.
Want to see your own results?
AEO Armory runs the full audit free: real ChatGPT and Google AI Mode answers, 9 readiness checks and a prioritized fix list.
Run your free AEO audit →Details and full user-agent strings
OpenAI
OpenAI treats each setting as independent: you can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot. Example strings from OpenAI's docs (version numbers change):
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
OpenAI says sites that block OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.
Googlebot crawls for Search, and Google says a page must be indexed and eligible to show a snippet to appear as a supporting link in AI Overviews or AI Mode. Google-Extended has no user-agent string of its own. It controls whether crawled content may train future Gemini models and be used for grounding in Gemini Apps and Vertex AI, and Google states it does not affect inclusion or ranking in Google Search.
Anthropic
Anthropic's help page names ClaudeBot, Claude-User and Claude-SearchBot, says to add rules for every subdomain, and supports the non-standard Crawl-delay line. It does not publish full user-agent strings on that page.
Perplexity
Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)
Microsoft
Bing's crawler identifies as bingbot/2.0, for example Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36. Microsoft documents no separate AI crawler. To limit use in its chat answers, Bing introduced NOCACHE and NOARCHIVE meta tag values in September 2023.
Apple, Meta, Amazon and others
- Applebot:
... Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot). Apple says Applebot-Extended does not crawl and pages that disallow it can still appear in Apple's search results. - Meta:
meta-externalagent/1.1,meta-webindexer/1.1,meta-externalfetcher/1.1; Meta says robots.txt changes can take up to 24 hours. - Amazon:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36, with Amzn-SearchBot/0.1 and Amzn-User/0.1 in the same format. - Common Crawl:
CCBot/2.0 (https://commoncrawl.org/faq/). - DuckDuckGo:
DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html). - Mistral:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots), with MistralAI-User and MistralAI-Training in the same format.
Copy-paste robots.txt examples
Allow AI search, block training
The most common choice for businesses that want to be cited but not used as training data. Several user-agent lines can share one group of rules.
# Allow AI search and live answers
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Amzn-SearchBot
User-agent: DuckAssistBot
User-agent: MistralAI-Index
User-agent: meta-webindexer
Allow: /
# Block model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
User-agent: MistralAI-Training
Disallow: /
Sitemap: https://www.example.com/sitemap.xmlTwo cautions. Blocking Google-Extended also opts you out of grounding in Gemini Apps, not just training. Amazonbot is left out because Amazon says it may train models and also improves its products; decide for yourself.
Allow everything
If your robots.txt has no AI rules and a general allow, every bot that honors robots.txt may crawl.
# Allow every crawler, including AI
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlBlock AI crawlers but keep search engines
This keeps Googlebot and bingbot, so your Google and Bing listings are unaffected, but it removes you from ChatGPT, Claude, Perplexity and other AI search answers. User-initiated fetchers are not listed because vendors say they may not follow robots.txt anyway.
# Block AI crawlers that honor robots.txt (search engines stay allowed)
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: CCBot
User-agent: DuckAssistBot
User-agent: MistralAI-Index
User-agent: MistralAI-Training
Disallow: /Test any change before you publish it. The AEO Armory dashboard has a free AI robots.txt generator and robots.txt tester.
Verify a bot before you trust it
User-agent strings are easy to fake, and Common Crawl warns that it sees crawlers falsely claiming to be CCBot. Check the IP address against the vendor's published list or reverse DNS:
- OpenAI: openai.com/searchbot.json, openai.com/gptbot.json and openai.com/chatgpt-user.json
- Perplexity: perplexity.ai/perplexitybot.json and perplexity.ai/perplexity-user.json
- Anthropic: claude.com/crawling/bots.json (Anthropic says blocking by IP may stop its bots reading robots.txt)
- Apple: reverse DNS in the applebot.apple.com domain
- Common Crawl: reverse DNS in crawl.commoncrawl.org, or index.commoncrawl.org/ccbot.json
- DuckDuckGo: duckduckgo.com/duckassistbot.json; Amazon publishes separate IP pages per agent
Also check your CDN and firewall. A bot-protection rule can block AI crawlers that robots.txt allows, which is a common reason a site is missing from AI answers.
Who is not on this list
We only include agents the company documents itself. ByteDance's Bytespider is the best-known omission: third-party trackers list it, but we could not find a ByteDance page that documents it or commits to robots.txt, so we cannot verify its purpose or behavior. You can still name it in robots.txt; just do not count on it.
Frequently asked questions
What is the user agent for ChatGPT?
ChatGPT uses three: OAI-SearchBot for ChatGPT search, GPTBot for training data, and ChatGPT-User when it visits a page because a user asked. Each is controlled separately in robots.txt.
Does blocking Google-Extended remove me from AI Overviews?
No. Google says Google-Extended does not affect inclusion in Google Search. AI Overviews and AI Mode use pages Googlebot indexed; to limit them, Google points to nosnippet and noindex controls.
Which AI crawlers ignore robots.txt?
By their vendors' own statements, user-initiated fetchers may: OpenAI says robots.txt may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores it, and Meta says meta-externalfetcher may bypass it.
Should I block AI crawlers?
Most businesses should allow AI search crawlers so they can be cited, and decide separately about training crawlers. Publishers who sell their content have the strongest case for blocking.
Sources & further reading
- Overview of OpenAI Crawlers — OpenAI developer docs
- Google's common crawlers (incl. Google-Extended) — Google for Developers
- AI features and your website — Google Search Central
- Does Anthropic crawl data from the web, and how can site owners block the crawler? — Claude Help Center
- Perplexity crawlers — Perplexity docs
- Announcing user-agent change for Bing crawler bingbot — Bing Webmaster Blog, Apr 2022
- Announcing new options for webmasters to control usage of their content in Bing Chat — Bing Webmaster Blog, Sep 2023
- About Applebot — Apple Support
- Meta Web Crawlers — Meta for Developers
- Amazon Web Search crawlers (Amazonbot, Amzn-SearchBot, Amzn-User) — Amazon Developer, updated Oct 2026
- CCBot — Common Crawl
- Is DuckAssistBot related to DuckDuckGo? — DuckDuckGo Help Pages
- Mistral robots (MistralAI-User, MistralAI-Index, MistralAI-Training) — Mistral AI docs
- Introduction to robots.txt — Google Search Central


