Can Google and ChatGPT crawl that page?
Paste your robots.txt, enter a URL, and see which crawlers are allowed or blocked, from Googlebot to GPTBot, OAI-SearchBot and ClaudeBot, with the exact rule that decided.
Your robots.txt
Open yourdomain.com/robots.txt, copy everything and paste it here.
Who can crawl it?
| Crawler | Result | Matched rule |
|---|
Follows Google's rules: the most specific user-agent group applies, the longest matching path rule wins, and Allow wins a tie. Supports * and $ wildcards.
Blocked from Google, or from AI search?
One misplaced line in robots.txt can hide a whole site from Google, or quietly keep you out of ChatGPT and Perplexity answers. AI companies run separate crawlers for search and for training, so you can block training (GPTBot, ClaudeBot, Google-Extended) while still letting their search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) fetch your pages so you can be cited.
Robots.txt controls crawling, not indexing: a blocked page can still appear in results if other sites link to it. To keep a page out of Google, allow crawling and add a noindex tag instead. Need to change your rules? Use the free AI robots.txt generator and read AI crawlers and robots.txt.
Frequently asked questions
How do I test my robots.txt file?
Paste the contents of yourdomain.com/robots.txt into this tester, enter a URL or path, and it shows whether Googlebot, Bingbot and the main AI crawlers are allowed to fetch it and which rule decided.
Does blocking GPTBot keep me out of ChatGPT search?
No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, and user-triggered fetches use ChatGPT-User. Allow those if you want to be cited.
Which rule wins when Allow and Disallow both match?
The longest (most specific) matching rule wins. If they are the same length, Allow wins. This follows Google's documented behavior.
Is this robots.txt tester free?
Yes, with no signup. Everything runs in your browser; nothing you enter is sent anywhere.
Related reading
Want us to do this for you?
Get a free AI visibility report: how ChatGPT, Gemini and Google AI describe your business, and what to fix first.
