The short version: three kinds of AI crawler
Most AI companies no longer run one bot. They run several, and each does a different job. Your robots.txt file can treat them differently.
| Job | What it means for you | Examples |
|---|---|---|
| Search | Finds and links your pages in an AI search product. Block it and you may not be shown there. | OAI-SearchBot, PerplexityBot, Claude-SearchBot |
| Training | Collects content that may be used to train models. Block it and your pages are asked to stay out of training. | GPTBot, Google-Extended, ClaudeBot, Applebot-Extended |
| User request | Visits one page because a person asked a question. | ChatGPT-User, Perplexity-User, Claude-User |
If you want to appear in AI answers, the search row is the one that matters. It is one part of AI search optimization, and a quick one to check.
What each crawler does
OpenAI: OAI-SearchBot, GPTBot and ChatGPT-User
OpenAI’s crawler page lists three. OAI-SearchBot is used to surface websites in ChatGPT’s search features. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. GPTBot crawls content that may be used to train OpenAI’s generative AI foundation models, and disallowing it indicates your content should not be used in training. ChatGPT-User visits pages when a user asks ChatGPT something or uses a custom GPT action. OpenAI says it is not used to crawl the web automatically, and that because a user starts the visit, robots.txt rules may not apply.
OpenAI also says each setting is independent. You can allow OAI-SearchBot and disallow GPTBot. For search, changes can take about 24 hours to take effect.
Google: Googlebot and Google-Extended
Googlebot is the crawler for Google Search, and Google says its robots.txt rules affect Google Search. For AI Overviews and AI Mode, Google says there are no additional requirements: a page has to be indexed and eligible to show in Search with a snippet. For managing how your sites are crawled for Search, Google names robots.txt directives for Googlebot as the control.
Google-Extended is a separate token. Google describes it as a standalone product token that controls whether content Google crawls may be used for training future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. It has no separate user agent string, and Google says it does not impact a site’s inclusion in Google Search and is not a ranking signal.
To limit what Google shows from your pages in Search, including in AI features, Google points to preview controls such as nosnippet, data-nosnippet, max-snippet and noindex. Those also limit how much of you can be quoted, so use them only when you mean it.
Perplexity: PerplexityBot and Perplexity-User
Perplexity says PerplexityBot is designed to surface and link websites in Perplexity search results, and is not used to crawl content for AI foundation models. It recommends that sites allow it. Perplexity-User supports user actions: when someone asks a question, it may visit a page and link to it. Perplexity says that, since a user requested the fetch, it generally ignores robots.txt rules. Changes can take up to 24 hours.
Anthropic and Apple
Anthropic’s help article lists three crawlers. ClaudeBot collects web content for model training. Claude-User retrieves pages when a Claude user asks a question. Claude-SearchBot indexes content to improve search result quality. Anthropic says to add a rule for each bot in your top-level robots.txt and repeat it for every subdomain.
Apple’s Applebot gathers data for search features in Spotlight, Siri and Safari. Applebot-Extended does not crawl pages. It only controls whether Applebot’s data is used to train Apple’s generative AI models. Apple says pages that disallow Applebot-Extended can still appear in search results.
A robots.txt example: allow search, block training
This example is for a business that wants to appear in AI search but does not want its pages used for model training. It is an example, not a recommendation. Edit it to match your decision.
# Example robots.txt for a local service business
# Replace example.com with your domain
# Search and answer crawlers: allowed
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Applebot
Allow: /
# Training controls: asked to stay out of model training
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
# Everyone else, including Googlebot and Bingbot
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.com/sitemap.xmlA few notes on reading it:
- Each bot gets its own group. Google’s documentation says a crawler matches only one rule set: the first, most specific group that matches its user agent. So the
*group does not add to the named groups. Repeat any rule you need inside a named group. - Lines starting with
#are comments. - The
/wp-admin/lines are a common WordPress pattern. Delete them if you do not use WordPress. - The ChatGPT-User, Claude-User and Perplexity-User bots are left out on purpose. They act for a person, and OpenAI and Perplexity both say their robots.txt rules may not apply to them.
How to add or change your robots.txt file, step by step
- Open your current file. Go to
yourdomain.com/robots.txtin a browser. Google says the file must sit at the root of the host it applies to, and it applies only to the protocol, host and port where it is posted. A file onwww.example.comdoes not covershop.example.com. - Decide on training. There are two honest positions. You may be happy for AI companies to learn from your public pages, in which case you can leave the training bots alone. Or you may want them out, in which case you disallow them. Google says Google-Extended is not a ranking signal in Search.
- Check search access first. Make sure OAI-SearchBot and PerplexityBot are not disallowed, and that no broad rule blocks them by accident. Check any plugin, host or security setting that edits robots.txt or blocks bots for you.
- Edit the file. On WordPress, an SEO plugin usually has a robots.txt editor. On other platforms, look in site settings or ask your web host. Paste in the groups you want. Rules are case-sensitive, and a Disallow path must start with
/. - Save, then reload
yourdomain.com/robots.txtand confirm the change is live. - Wait about a day. OpenAI and Perplexity both say changes can take around 24 hours.
- Write the date and your decision in a note. Crawler names change. The next person who edits the file should know why it looks the way it does.
If you want a second pair of eyes on this, our GEO and AEO audit includes a crawl-access check.
Mistakes to avoid
- Blocking everything with
Disallow: /underUser-agent: *. That also blocks Googlebot and the search bots you wanted. - Treating robots.txt as security. Google says robots.txt rules cannot enforce crawler behavior and that private files should be password-protected. It is also not a way to keep a page out of Google: Google says to use
noindexfor that. - Blocking by IP address. Anthropic says IP blocking may not reliably give an opt-out, because it can stop its bots from reading your robots.txt in the first place. If your firewall or security service blocks bots automatically, check it separately from robots.txt.
- Blocking a search bot and expecting to be recommended. OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers.
- Copying a list from a forum. Bot names change, and some lists mix up search and training bots. Use the vendors’ own pages (see sources below).
- Expecting robots.txt alone to bring you into AI answers. It only decides who may read your pages. What the pages say, and whether other sites confirm it, is a separate job. See how to show up in ChatGPT.
How this fits with llms.txt and schema
robots.txt controls access. llms.txt is a proposed reading list, and Google says Google Search ignores it. Schema markup labels facts inside a page; see schema markup for local business. The three are different tools, and only robots.txt is a long-established, widely honored way to say who may crawl.
Nobody controls what ChatGPT, Google AI Overviews, AI Mode or Perplexity recommend. Letting a search bot in is a precondition, not a promise.
Work with Magnet Mint
We are a Boston studio that builds websites for local service businesses and works with clients across the US, remotely. We review robots.txt, sitemap and bot access as part of the GEO and AEO audit. Price: from $750. Not included: domain and hosting, new photography, third-party software subscriptions (billed at cost), and AI usage above a plan allowance. To see how customers and AI assistants view your site today, ask for a free demo.
Published . Updated .