Every AI crawler and user agent, in one list (2026).

26 AI crawlers, user agents and control tokens from 11 companies, each checked against the vendor's own documentation, with what it does, whether it follows robots.txt, and ready-to-paste rules.

Every AI crawler and user agent, in one list (2026).

AI companies use three kinds of bots: search crawlers that decide whether you can be cited, training crawlers that collect content for future models, and user-initiated fetchers that visit a page when someone asks a question. Google and Apple also use control tokens that do not crawl at all. This list covers every agent we could find documented on the vendor's own page, checked in November 2026, with the exact robots.txt token and a link to the source. Agents without vendor documentation are left out.

How to read this list

Whether to block anything is a business decision, covered in AI crawlers and robots.txt. This page is the reference.

AI CRAWLERS BY PURPOSECompanyModel trainingSearch indexUser-triggered fetchOpenAIGPTBotOAI-SearchBotChatGPT-UserAnthropicClaudeBotClaude-SearchBotClaude-UserPerplexity(none listed)PerplexityBotPerplexity-UserGoogleGoogle-Extended*Googlebotuser fetchersMicrosoft(none listed)Bingbot(Copilot via Bing)Blocking a training bot does not remove you from search answers; blocking a search bot does.*Google-Extended is a robots.txt token, not a separate crawler. Check each vendor's docs for current names.
Major AI crawlers grouped by what they do. Names change; confirm against each vendor's documentation before editing robots.txt.

The full list

Companyrobots.txt tokenTypeWhat it does (vendor's words, paraphrased)Follows robots.txt?Source
OpenAIOAI-SearchBotSearchSurfaces sites in ChatGPT search answersYes; changes take about 24 hoursDocs
OpenAIGPTBotTrainingCrawls content that may train OpenAI's foundation modelsYesDocs
OpenAIChatGPT-UserUser fetchVisits pages when a ChatGPT user or custom GPT asksMay not applyDocs
OpenAIOAI-AdsBotAds checkChecks landing pages submitted as ChatGPT ads; not used for trainingOnly visits ad pagesDocs
GoogleGooglebotSearchCrawls for Google Search, including AI Overviews and AI ModeYesDocs
GoogleGoogle-ExtendedControl tokenControls Gemini training and grounding in Gemini Apps; no separate crawlerYes; does not affect SearchDocs
GoogleGoogle-CloudVertexBotSite-owner crawlCrawls sites whose owners asked to build Vertex AI AgentsYesDocs
AnthropicClaudeBotTrainingCollects web content for model trainingYes; supports Crawl-delayDocs
AnthropicClaude-SearchBotSearchImproves Claude's search resultsYesDocs
AnthropicClaude-UserUser fetchFetches pages when a person asks Claude a questionYes; blocking stops retrievalDocs
PerplexityPerplexityBotSearchSurfaces and links sites in Perplexity; not for trainingYesDocs
PerplexityPerplexity-UserUser fetchVisits a page to help answer a user's questionGenerally ignores itDocs
MicrosoftbingbotSearchCrawls for Bing; no separate Copilot crawler is documentedYesDocs
AppleApplebotSearchPowers Spotlight, Siri and Safari search; may also be used for trainingYes; no Crawl-delayDocs
AppleApplebot-ExtendedControl tokenOpts content out of Apple foundation model training; does not crawlYesDocs
Metameta-externalagentTraining / indexingTrains foundation AI models or indexes content directlyBlocking examples givenDocs
Metameta-webindexerSearchImproves Meta AI search resultsAllow to be citedDocs
Metameta-externalfetcherUser fetchFetches individual links at a user's requestMay bypass itDocs
AmazonAmazonbotTraining / productsImproves Amazon products; may train Amazon AI modelsYes; no Crawl-delayDocs
AmazonAmzn-SearchBotSearchSearch experiences such as Alexa; not for trainingYesDocs
AmazonAmzn-UserUser fetchFetches live information for Alexa queriesMay not follow all rulesDocs
Common CrawlCCBotOpen archiveBuilds an open web archive widely used for AI trainingYesDocs
DuckDuckGoDuckAssistBotSearchReal-time crawling for DuckDuckGo's AI-assisted answers; not for trainingYes; takes 72 hoursDocs
MistralMistralAI-IndexSearchIndexes the web for Mistral search; not for trainingYesDocs
MistralMistralAI-TrainingTrainingBuilds datasets to train Mistral modelsYesDocs
MistralMistralAI-UserUser fetchVisits pages when a user asks Mistral's Vibe a questionGoverns user requestsDocs

Write the tokens exactly as shown. "Follows robots.txt?" reflects each vendor's own statement, not independent testing.

Want to see your own results?

AEO Armory runs the full audit free: real ChatGPT and Google AI Mode answers, 9 readiness checks and a prioritized fix list.

Run your free AEO audit →

Details and full user-agent strings

OpenAI

OpenAI treats each setting as independent: you can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot. Example strings from OpenAI's docs (version numbers change):

OpenAI says sites that block OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.

Google

Googlebot crawls for Search, and Google says a page must be indexed and eligible to show a snippet to appear as a supporting link in AI Overviews or AI Mode. Google-Extended has no user-agent string of its own. It controls whether crawled content may train future Gemini models and be used for grounding in Gemini Apps and Vertex AI, and Google states it does not affect inclusion or ranking in Google Search.

Anthropic

Anthropic's help page names ClaudeBot, Claude-User and Claude-SearchBot, says to add rules for every subdomain, and supports the non-standard Crawl-delay line. It does not publish full user-agent strings on that page.

Perplexity

Microsoft

Bing's crawler identifies as bingbot/2.0, for example Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36. Microsoft documents no separate AI crawler. To limit use in its chat answers, Bing introduced NOCACHE and NOARCHIVE meta tag values in September 2023.

Apple, Meta, Amazon and others

Copy-paste robots.txt examples

Allow AI search, block training

The most common choice for businesses that want to be cited but not used as training data. Several user-agent lines can share one group of rules.

# Allow AI search and live answers
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Amzn-SearchBot
User-agent: DuckAssistBot
User-agent: MistralAI-Index
User-agent: meta-webindexer
Allow: /

# Block model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
User-agent: MistralAI-Training
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Two cautions. Blocking Google-Extended also opts you out of grounding in Gemini Apps, not just training. Amazonbot is left out because Amazon says it may train models and also improves its products; decide for yourself.

Allow everything

If your robots.txt has no AI rules and a general allow, every bot that honors robots.txt may crawl.

# Allow every crawler, including AI
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Block AI crawlers but keep search engines

This keeps Googlebot and bingbot, so your Google and Bing listings are unaffected, but it removes you from ChatGPT, Claude, Perplexity and other AI search answers. User-initiated fetchers are not listed because vendors say they may not follow robots.txt anyway.

# Block AI crawlers that honor robots.txt (search engines stay allowed)
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: CCBot
User-agent: DuckAssistBot
User-agent: MistralAI-Index
User-agent: MistralAI-Training
Disallow: /

Test any change before you publish it. The AEO Armory dashboard has a free AI robots.txt generator and robots.txt tester.

Verify a bot before you trust it

User-agent strings are easy to fake, and Common Crawl warns that it sees crawlers falsely claiming to be CCBot. Check the IP address against the vendor's published list or reverse DNS:

Also check your CDN and firewall. A bot-protection rule can block AI crawlers that robots.txt allows, which is a common reason a site is missing from AI answers.

Who is not on this list

We only include agents the company documents itself. ByteDance's Bytespider is the best-known omission: third-party trackers list it, but we could not find a ByteDance page that documents it or commits to robots.txt, so we cannot verify its purpose or behavior. You can still name it in robots.txt; just do not count on it.

Frequently asked questions

What is the user agent for ChatGPT?

ChatGPT uses three: OAI-SearchBot for ChatGPT search, GPTBot for training data, and ChatGPT-User when it visits a page because a user asked. Each is controlled separately in robots.txt.

Does blocking Google-Extended remove me from AI Overviews?

No. Google says Google-Extended does not affect inclusion in Google Search. AI Overviews and AI Mode use pages Googlebot indexed; to limit them, Google points to nosnippet and noindex controls.

Which AI crawlers ignore robots.txt?

By their vendors' own statements, user-initiated fetchers may: OpenAI says robots.txt may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores it, and Meta says meta-externalfetcher may bypass it.

Should I block AI crawlers?

Most businesses should allow AI search crawlers so they can be cited, and decide separately about training crawlers. Publishers who sell their content have the strongest case for blocking.

Sources & further reading

  1. Overview of OpenAI Crawlers — OpenAI developer docs
  2. Google's common crawlers (incl. Google-Extended) — Google for Developers
  3. AI features and your website — Google Search Central
  4. Does Anthropic crawl data from the web, and how can site owners block the crawler? — Claude Help Center
  5. Perplexity crawlers — Perplexity docs
  6. Announcing user-agent change for Bing crawler bingbot — Bing Webmaster Blog, Apr 2022
  7. Announcing new options for webmasters to control usage of their content in Bing Chat — Bing Webmaster Blog, Sep 2023
  8. About Applebot — Apple Support
  9. Meta Web Crawlers — Meta for Developers
  10. Amazon Web Search crawlers (Amazonbot, Amzn-SearchBot, Amzn-User) — Amazon Developer, updated Oct 2026
  11. CCBot — Common Crawl
  12. Is DuckAssistBot related to DuckDuckGo? — DuckDuckGo Help Pages
  13. Mistral robots (MistralAI-User, MistralAI-Index, MistralAI-Training) — Mistral AI docs
  14. Introduction to robots.txt — Google Search Central
Call now