Definition
AI crawlersAI crawlers are bots run by AI companies that fetch web pages. Some collect training data for future models, some build the search index an AI engine retrieves from, and some fetch a page on demand when a user asks a question. Each type has its own user agent, so robots.txt can allow one and block another.
How Searchable measures this
Searchable's AI Crawlers view shows which AI bots visit your site, how often, and how much of the site they reach, for OpenAI, Anthropic, Google, Perplexity, Microsoft, DeepSeek, xAI, and Meta, so you can confirm search crawlers can read the pages you want cited.
The 3 types of AI crawlers
AI companies run separate bots for separate jobs, and the job decides whether blocking one helps or hurts you. The bots fall into three groups:
- Training crawlers collect pages that may be used to train future models. Blocking one signals that your content shouldn't be used for training, and for OpenAI it has no effect on ChatGPT search, which uses a separate crawler.
- Search crawlers build the index an AI engine searches before it answers. Blocking one removes or reduces your pages in that engine's answers and citations.
- User-triggered fetchers visit a page because a person asked a question that needs it, or shared the link directly. They act more like a browser than a crawler.
Not every company runs all three. Perplexity documents no training crawler, and Google controls AI training through a robots.txt token instead of a separate bot. The split matters because a robots.txt rule written to stop training can also remove you from AI search if it blocks the wrong user agent.
The main AI crawlers and what they do
These are the user agents the major AI companies document. The robots.txt behavior comes from each company's own documentation.
| Company | User agent | Type | Respects robots.txt |
|---|---|---|---|
| OpenAI | GPTBot | Training | Yes |
| OpenAI | OAI-SearchBot | Search (ChatGPT search) | Yes |
| OpenAI | ChatGPT-User | User-triggered | May not apply |
| Anthropic | ClaudeBot | Training | Yes |
| Anthropic | Claude-SearchBot | Search | Yes |
| Anthropic | Claude-User | User-triggered | Yes |
| Perplexity | PerplexityBot | Search | Yes |
| Perplexity | Perplexity-User | User-triggered | Generally ignores it |
| Google-Extended | Robots.txt token for Gemini training and grounding | Yes (token only) |
OpenAI's crawler documentation says sites that opt out of OAI-SearchBot won't be shown in ChatGPT search answers, and that robots.txt rules may not apply to ChatGPT-User. Anthropic's help center says blocking Claude-SearchBot or Claude-User may reduce a site's visibility in Claude's search results. Perplexity's documentation says PerplexityBot isn't used to train foundation models.
GPTBot
GPTBot is OpenAI's training crawler. Disallowing it tells OpenAI not to use your content to train its foundation models, and it doesn't change whether you appear in ChatGPT search. That job belongs to OAI-SearchBot, so a site can block GPTBot and allow OAI-SearchBot at the same time.
Google-Extended
Google-Extended isn't a crawler at all. It's a robots.txt token that Googlebot checks. Google's documentation says it controls whether content can be used to train Gemini models and for grounding in Gemini Apps, and that it doesn't affect inclusion or ranking in Google Search. Google AI Overviews and AI Mode are part of Search, so they follow Googlebot's rules.
How to manage AI crawlers in robots.txt
Decide by crawler type rather than by company. If you want AI visibility, allow every search crawler and user-triggered fetcher, then decide separately about training crawlers. This setup blocks OpenAI training while keeping ChatGPT search:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
OpenAI and Perplexity both say robots.txt changes can take about 24 hours to take effect. A firewall or bot-protection rule can also block AI crawlers without anyone editing robots.txt, so check your CDN settings when a search crawler never shows up in your logs.
Why AI crawlers matter for AI visibility
An AI engine can only cite a page its crawler can read. If a search crawler is blocked, times out, or gets a page whose content loads only through JavaScript, that page can't be retrieved for the answer, however good it is.
JavaScript is a common hidden problem. Vercel's analysis of AI crawler traffic found that GPTBot and ClaudeBot fetch JavaScript files but don't execute them. Pages that build their content in the browser can look empty to these bots, which is why server-side rendering matters for AI search.
How to check AI crawler activity on your site
Server logs are the most reliable record. Search them for the user agents in the table above, then check which pages each bot requests and what status codes it receives. A search crawler that only ever gets 403 or 404 responses is a problem to fix before any content work.
Don't trust the user agent alone. Anyone can send a request that claims to be GPTBot, so verify important bots against the IP addresses each company publishes.
OpenAI lists ranges in its crawler documentation, and Anthropic publishes its crawler IPs at claude.com/crawling/bots.json. Anthropic also warns that blocking by IP can stop its bots from reading robots.txt, so use robots.txt to opt out.
For how we serve AI crawlers on our own site, see how we built a dual-served content system for AI crawlers.
Frequently asked questions
Block training crawlers if you don't want your content used to train models, but allow search crawlers if you want to appear in AI answers. Blocking GPTBot doesn't remove you from ChatGPT search, while blocking OAI-SearchBot does.
See how AI engines talk about your brand
Get a free report on where ChatGPT, Perplexity, and Google AI name and cite you.