robots.txt for AI crawlers: a setup guide for B2B SaaS sites

Allow the search crawlers of each AI tool, decide separately on the training crawlers, and check that nothing upstream blocks either. For AI search, the user agents are OAI-SearchBot for ChatGPT, Claude-SearchBot for Claude and PerplexityBot for Perplexity. For training, they are GPTBot, ClaudeBot and the Google-Extended token. OpenAI says each setting "is independent of the others," so a site can be cited without its content being used for training.

Every crawler rule on this page comes from the documentation of OpenAI, Anthropic, Perplexity and Google, read in October 2026. The sample file is an invented example to illustrate the rules, not a recommendation for every site.

How should a B2B SaaS site set robots.txt for AI crawlers?

In four steps.

Which user agents matter?

These are the crawlers and tokens documented by the four companies.

Google-Extended has no user agent of its own. Google describes it as a token read in robots.txt, used "in a control capacity." Googlebot itself controls access for Google Search, including AI Overviews.

What does a sample file look like?

The example below allows AI search crawlers and blocks training crawlers. It is an invented example to show the syntax. A site may just as well choose to allow the training crawlers too.

# AI search: allow, so the site can be cited
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Model training: a separate business choice
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# Everyone else, including Googlebot
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Anthropic notes that rules must be set in the top-level robots.txt file of each subdomain. A product on app.example.com and docs on docs.example.com each need their own file.

Should training crawlers be blocked?

It depends on the company's view, not on AEO. The documentation is clear that the choices are separate. OpenAI: "Each setting is independent of the others." A site can allow OAI-SearchBot and disallow GPTBot.

Points that weigh on the decision for a B2B SaaS company:

• Allowing training may mean the brand and product are better known to future models, including in answers given without web search. No study read for this site measures that effect.
• Blocking training keeps content out of future training sets, per OpenAI and Anthropic, without removing the site from their search features.
• Google-Extended is separate from Google Search. Google says it does not affect inclusion or ranking.

The wider comparison is on the AI crawlers page.

What about crawlers that act for users?

They follow different rules. ChatGPT-User, Claude-User and Perplexity-User fetch pages when a person asks the AI tool a question. OpenAI says "robots.txt rules may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores robots.txt rules." Anthropic says its bots honor robots.txt.

For a B2B SaaS site, these fetchers matter at the moment a buyer asks an AI tool about the product. Blocking them, where it works, can stop the tool reading the pricing or product page the buyer asked about.

How can the setup be checked?

Three checks after any change.

1. Fetch the live file. Open /robots.txt on the main domain and each subdomain and read it.
2. Check the CDN and firewall. Google lists "any CDN or hosting infrastructure" alongside robots.txt as places crawling must be allowed. Bot protection can block unfamiliar crawlers.
3. Read the server logs. Look for OAI-SearchBot, Claude-SearchBot and PerplexityBot with successful responses.

Allow for propagation. OpenAI says "it can take ~24 hours" for its search systems to adjust, and Perplexity says changes may take up to 24 hours. Anthropic warns that blocking by IP address "may not work correctly or persistently guarantee an opt-out," because it can stop the bot from reading robots.txt at all.

Does robots.txt affect Google AI Overviews?

Through Googlebot only. Google's AI features page says robots.txt directives for Googlebot are the control for crawl access, and that pages must be indexed and eligible for a snippet to appear. Snippet controls such as nosnippet and max-snippet limit how a page is used. Google-Extended, Google says, does not affect inclusion in Google Search. More is on the Google AI Overviews page.

FAQ

Does blocking GPTBot remove a site from ChatGPT search?

Not according to OpenAI. GPTBot controls whether content may be used for training. OAI-SearchBot controls whether a site appears in ChatGPT search answers, and OpenAI says each setting is independent. A site can block GPTBot and still be cited in ChatGPT search if OAI-SearchBot is allowed.

What robots.txt rule allows Claude to cite a site?

Allowing Claude-SearchBot, which Anthropic says navigates the web to improve search result quality for users. Anthropic says blocking it may reduce a site's visibility in user search results. Claude-User, which fetches pages when a user asks, should also be allowed if the site wants to be read on request.

How long do robots.txt changes take for AI crawlers?

OpenAI says it can take about 24 hours from a robots.txt update for its search systems to adjust. Perplexity says changes may take up to 24 hours. Anthropic's documentation does not state a time. Server logs show when the crawlers return after a change.

Does Google-Extended need a user agent rule?

It is written like one, as User-agent: Google-Extended, but Google says it has no separate HTTP user agent. It is a token read in robots.txt that manages whether crawled content may be used for training Gemini models and for grounding. It does not affect Google Search.

Why do AI crawlers still fail when robots.txt allows them?

Usually because something upstream blocks them. Google lists CDN and hosting infrastructure alongside robots.txt as places crawling must be allowed. Bot protection on a CDN or firewall can return errors to AI crawlers that robots.txt permits. Server logs show whether requests succeed.

Conclusion

A B2B SaaS site that wants to be cited in AI search should allow OAI-SearchBot, Claude-SearchBot and PerplexityBot, and decide on training crawlers as a separate matter. OpenAI says the settings are independent. User-triggered fetchers follow their own rules, and CDN or firewall settings can block crawlers robots.txt allows. A short check of the live file, the CDN and the server logs after each change confirms the setup works.

Sources

• Overview of OpenAI crawlers. OpenAI.
• Does Anthropic crawl data from the web, and how can site owners block the crawler?. Claude Help Center.
• Perplexity Crawlers. Perplexity.
• Google's common crawlers. Google Search Central.
• AI features and your website. Google Search Central.