AI crawlers for B2B SaaS sites: which to allow and why
The search crawlers, if the goal is to be cited. For ChatGPT that is OAI-SearchBot, for Claude it is Claude-SearchBot and for Perplexity it is PerplexityBot. Each company runs separate crawlers for model training, such as GPTBot and ClaudeBot, and treats the two settings independently. A B2B SaaS site can allow search crawlers and block training crawlers, or allow both. Blocking OAI-SearchBot, OpenAI says, keeps a site out of ChatGPT search answers.
Allow the search crawlers: OAI-SearchBot, Claude-SearchBot and PerplexityBot. Decide on training crawlers separately.
OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers.
A CDN or firewall that blocks AI crawlers even when robots.txt allows them. Google names the CDN as a place to check.
Open the site's robots.txt and check each AI user agent listed on this page against the site's intent.
Which AI crawlers should a B2B SaaS website allow?
It depends on what each crawler is for. The table groups them by job.
| Company | Crawler | Job | Effect of blocking it, per the company |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Search in ChatGPT | "Will not be shown in ChatGPT search answers" |
| OpenAI | GPTBot | Model training | Content "should not be used in training" |
| OpenAI | ChatGPT-User | Fetches pages when a user asks | "robots.txt rules may not apply" |
| Anthropic | Claude-SearchBot | Search quality in Claude | May reduce visibility in user search results |
| Anthropic | ClaudeBot | Model training | Future content excluded from training |
| Anthropic | Claude-User | Fetches pages when a user asks | Stops retrieval for user queries |
| Perplexity | PerplexityBot | Search results in Perplexity | Perplexity recommends allowing it |
| Perplexity | Perplexity-User | Fetches pages when a user asks | "Generally ignores robots.txt rules" |
| Google-Extended | Gemini training and grounding | No effect on Google Search inclusion or ranking |
For a B2B SaaS company that wants to be cited, the search crawlers are the ones that matter most. The training crawlers are a separate business decision.
What is the difference between search and training crawlers?
Search crawlers fetch pages so an AI tool can find and cite them in answers. Training crawlers collect content that may be used to train future models. The companies document them separately and say the settings are independent.
OpenAI is explicit: "Each setting is independent of the others." A site can allow OAI-SearchBot and disallow GPTBot. OpenAI adds that if both are allowed, it may use one crawl for both purposes. Anthropic describes the same split: ClaudeBot for training, Claude-SearchBot for search quality and Claude-User for fetches made on a user's behalf.
Perplexity says PerplexityBot is not used to crawl content for AI foundation models. It is a search crawler only.
What about crawlers that fetch pages for users?
They work differently. ChatGPT-User, Claude-User and Perplexity-User fetch a page when a person asks the AI tool something that needs it. Because a person started the request, OpenAI says "robots.txt rules may not apply," and Perplexity says its user fetcher "generally ignores robots.txt rules."
Anthropic's documentation describes Claude-User as honoring robots.txt like its other bots, and says disabling it stops Claude retrieving the content for user queries. The practical point for a B2B SaaS site: these fetchers are how an AI tool reads a pricing or product page when a buyer asks about it. Blocking them can hide the site from buyers who are already asking about it.
Does Google-Extended affect AI Overviews?
Google does not say it does. Google describes Google-Extended as a token for managing whether content may be used "for training future generations of Gemini models," and for grounding. It states: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."
The same page does not mention AI Overviews. Google's AI features page says the control for AI Overviews and AI Mode is ordinary: robots.txt rules for Googlebot, plus snippet controls such as nosnippet and max-snippet. Those are covered on the Google AI Overviews page.
Why might allowed crawlers still be blocked?
Because robots.txt is not the only gate. Google's AI features guidance lists "Ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure." Bot protection on a CDN or firewall can block AI crawlers that robots.txt allows.
Three checks catch most problems.
- Read the live robots.txt. Check every AI user agent named on this page.
- Check the CDN and firewall rules. Look for bot-blocking settings that apply to AI crawlers by default.
- Check the server logs. Requests from OAI-SearchBot, Claude-SearchBot and PerplexityBot should appear with successful responses.
Anthropic also warns that blocking by IP address "may not work correctly or persistently guarantee an opt-out," because it can stop the bot from reading robots.txt at all. A sample file is in robots.txt for AI crawlers.
How long do changes take?
About a day for the two companies that state it. OpenAI says "it can take ~24 hours from a site's robots.txt update for our systems to adjust." Perplexity says changes may take up to 24 hours. Anthropic's article does not give a time.
After a change, the server logs are the quickest confirmation that the crawlers are back. Whether the site is then cited is a separate question, covered on the tracking AI visibility page.
Frequently asked questions
What is OAI-SearchBot?
OAI-SearchBot is OpenAI's search crawler. OpenAI says it is used to surface websites in ChatGPT's search features, and that sites opted out of it will not be shown in ChatGPT search answers, though they may still appear as navigational links. It is separate from GPTBot, which collects content for training.
Should a B2B SaaS site block GPTBot?
It is a business choice, not an AEO requirement. OpenAI says disallowing GPTBot signals that content should not be used for training its models, and that each crawler setting is independent. Blocking GPTBot does not, per OpenAI's documentation, remove a site from ChatGPT search answers. OAI-SearchBot controls that.
What is the difference between ClaudeBot and Claude-SearchBot?
ClaudeBot collects content that may help train Anthropic's models. Claude-SearchBot navigates the web to improve search result quality for Claude users. Anthropic says blocking Claude-SearchBot may reduce a site's visibility and the accuracy of how it is represented in user search results.
Does blocking Google-Extended remove a site from Google Search?
No. Google states that Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. It manages whether crawled content may be used for training future Gemini models and for grounding. Google's documentation does not say it controls AI Overviews.
Do AI crawlers follow robots.txt?
The search and training crawlers documented by OpenAI, Anthropic and Perplexity do. User-triggered fetchers are different: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Anthropic says its bots honor robots.txt.
Figures on this page come from published studies and platform documentation. Where the numbers come from
Talk your AI search visibility through with an expert
A free consultation. Bring your AI search, crawler or content question and brainstorm the next step.