Can AI assistants read your site?
Check which AI crawlers a website's robots.txt allows or blocks, from GPTBot and ClaudeBot to PerplexityBot and Google-Extended, and what each one means for you.
Ready-made robots.txt rules
Add one of these to the robots.txt at the root of your site. Search engines such as Googlebot and Bingbot are not affected.
Stay in AI answers, opt out of training
Blocks the crawlers that collect training data. Assistants can still find and cite your pages.
# Stay in AI search and answers, opt out of AI training User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: cohere-ai Disallow: /
Block every AI crawler listed here
Your pages will not be used for training, AI search or user-requested fetches by these operators.
# Block all known AI crawlers User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Amazonbot User-agent: Meta-ExternalAgent User-agent: CCBot User-agent: Bytespider User-agent: cohere-ai User-agent: MistralAI-User User-agent: DuckAssistBot Disallow: /
The AI crawlers we check
Each operator publishes the name its crawler uses in robots.txt. Blocking one does not block the others.
| Crawler | Operator | Purpose | If blocked |
|---|---|---|---|
| GPTBot | OpenAI | Model training | Keeps your pages out of future OpenAI model training. |
| OAI-SearchBot | OpenAI | AI search and answers | Your pages will not appear as sources in ChatGPT search results. |
| ChatGPT-User | OpenAI | Fetches for a user | ChatGPT cannot open your pages when a user asks it to. |
| ClaudeBot | Anthropic | Model training | Keeps your pages out of future Anthropic model training. |
| Claude-SearchBot | Anthropic | AI search and answers | Your pages will not be used in Claude search results. |
| Claude-User | Anthropic | Fetches for a user | Claude cannot open your pages when a user asks it to. |
| PerplexityBot | Perplexity | AI search and answers | Your pages will not appear as sources in Perplexity answers. |
| Perplexity-User | Perplexity | Fetches for a user | Perplexity cannot open your pages when a user asks it to. |
| Google-Extended | Model training | Your content will not be used for Gemini models. This does not affect Google Search. | |
| Applebot-Extended | Apple | Model training | Your content will not be used to train Apple's AI models. This does not affect Apple search features. |
| Amazonbot | Amazon | AI search and answers | Amazon services, including its assistants, cannot use your pages. |
| Meta-ExternalAgent | Meta | Model training | Keeps your pages out of Meta's AI training and products. |
| CCBot | Common Crawl | Model training | Keeps your pages out of the Common Crawl archive many AI models are trained on. |
| Bytespider | ByteDance | Model training | Keeps your pages away from ByteDance's crawler. |
| cohere-ai | Cohere | Model training | Keeps your pages out of Cohere's models. |
| MistralAI-User | Mistral | Fetches for a user | Le Chat cannot open your pages when a user asks it to. |
| DuckAssistBot | DuckDuckGo | AI search and answers | Your pages will not be used in DuckDuckGo's AI answers. |
AI crawlers and robots.txt
What is the difference between training and search crawlers?
Training crawlers such as GPTBot and ClaudeBot collect pages to train future AI models. Search crawlers such as OAI-SearchBot and PerplexityBot fetch pages so an assistant can cite them in its answers. User agents such as ChatGPT-User fetch a page when a person asks the assistant to open it. You can allow some and block others.
Does blocking Google-Extended remove my site from Google?
No. Google-Extended only controls whether your content is used for Gemini models. Google Search uses Googlebot, which this check does not change.
Do AI crawlers obey robots.txt?
The major operators listed here say their crawlers follow robots.txt. It is a request, not a lock: to see which crawlers actually visit, look at your server logs.
My robots.txt is fine but bots are still blocked. Why?
A firewall or CDN setting can block or allow AI crawlers regardless of robots.txt. Some CDNs can also add their own AI rules to your robots.txt, which this check would show.
Do you store the websites I check?
No. The result is shown once and not saved. To prevent abuse, the number of checks per visitor is limited for a few minutes.
More free checks
No sign-up, nothing stored. See all free tools.
See which AI bots actually visit.
robots.txt says who may crawl. GoTrace reads your server logs to show which AI crawlers really come, which pages they read, and how many visitors AI assistants send back.
AI traffic analytics