How to Allow AI Crawlers in robots.txt (Every User Agent, Copy and Paste)

To allow AI crawlers in robots.txt, add an explicit Allow: / rule for each AI user agent you want to permit, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended. A permissive User-agent: * block usually covers them already. Note that robots.txt is advisory: if a firewall or security plugin is refusing these crawlers, the file cannot override it, and the fix belongs at the edge instead.

That is the whole answer. The rest of this page is the file itself, what each agent actually does, and the three ways sites block these crawlers by accident.

Why this matters more than it looks

Answer engines reach your site through two doors. One is the training snapshot, which is months or years old and changes slowly. The other is live retrieval, where an assistant fetches your page in the moment somebody asks a question.

Live retrieval is the fast door. It does not care how old your domain is or how many backlinks you have. It cares whether the fetch succeeds. If a crawler gets blocked, rate limited or served a challenge page, you are simply not a candidate, no matter how good the content behind that URL is.

This is the single most common technical reason a business is absent from AI answers, and it is usually not a decision anyone made on purpose.

The robots.txt, copy and paste

Four lines. On a normally configured site this is the whole job, and it is what this site serves.

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Save that as robots.txt in your site root, so it lives at yourdomain.com/robots.txt. Change the sitemap line to your own URL. That is the entire implementation for most sites.

User-agent: * already covers every crawler named in this post. Adding a group per agent does not grant access the wildcard was withholding, because it was not withholding any. It is worth understanding why the longer version below exists before you reach for it, because the rule that makes it useful is the same rule that makes it a liability if you get it wrong: a crawler with its own group reads only that group and ignores User-agent: * entirely.

What this file cannot do

robots.txt is a set of voluntary instructions that well-behaved crawlers choose to follow. It is not an access control, and it cannot override anything. Google sets out what the file is for in Introduction to robots.txt.

If your CDN, WAF, rate limiter or security plugin is blocking a crawler, robots.txt will not change that. Your file says yes and your edge says no, and the edge wins every time. Listing an agent explicitly does not protect you from that, and anyone telling you otherwise has confused two different layers.

Access is configured at the edge and verified by looking at real responses. Request a page while sending each crawler's user agent and read the status code that comes back. A 403, a 429 or a challenge page tells you something between robots.txt and the crawler is refusing, and that something is where the fix belongs.

The longer file below is still worth having if you want to treat agents differently, allowing a search crawler while disallowing a training crawler, because a crawler with its own group reads only that group. It buys you per agent control. It does not buy you protection from anything upstream.

# Allow AI crawlers

User-agent: *
Allow: /

# OpenAI
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

# Anthropic
User-agent: ClaudeBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

# Perplexity
User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

# Google
User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Allow: /

# Microsoft
User-agent: bingbot
Allow: /

# Apple
User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Allow: /

# Meta
User-agent: meta-externalagent
Allow: /

# Common Crawl
User-agent: CCBot
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Same location, same sitemap line to change. If you deploy this version, re-request your robots.txt afterwards and read what comes back rather than what you committed. The entire reason for using it is that something upstream may be editing the file, and that something does not stop just because you rewrote it.

What each user agent actually does

Allowing everything is the right default for most businesses, but you should know what you are agreeing to, because these agents do genuinely different jobs.

GPTBot collects training data for future OpenAI models. Allowing it means your content may inform how a model describes your category in a year's time. Blocking it does not affect whether ChatGPT can cite you today.

OAI-SearchBot builds the index behind ChatGPT's search feature. This is a citation crawler. Block it and your pages stop being candidates for the source list users see.

ChatGPT-User fetches pages live when a user's prompt triggers browsing, or when someone clicks a citation link. OpenAI has also extended it to cover Custom GPT requests and GPT Actions. For most businesses this is the single most important agent to allow.

ClaudeBot, Claude-User and Claude-SearchBot are Anthropic's equivalents, split along roughly the same lines between training, user-initiated fetch and search indexing.

PerplexityBot indexes for Perplexity's answers. Perplexity-User handles live fetches during a session.

Google-Extended is not a crawler and has no user agent of its own. It is a control token governing whether content Googlebot already fetched may be used for training and grounding in Gemini Apps and the Vertex AI API.

It is widely misunderstood, so this is worth stating plainly. Disallowing Google-Extended does not remove you from AI Overviews or AI Mode. Those are features inside Google Search, drawing on the standard Search index through Googlebot. Appearing in them requires only that a page is indexed and eligible to appear in Search with a snippet. There is no separate AI index and no separate submission step.

The controls that do affect AI features are the ordinary preview controls: nosnippet, data-nosnippet, max-snippet and noindex. Google has also begun rolling out a setting in Search Console, under Settings then Search generative AI, that excludes a site from AI Overviews and AI Mode without removing it from normal Search. It is reaching site owners gradually. Google documents all of this in AI features and your website.

The mistake to avoid in the other direction is worse. Blocking Googlebot to "stop AI" removes you from Google Search entirely, along with everything built on it.

Applebot-Extended works the same way for Apple's models.

CCBot is Common Crawl, which feeds a large number of downstream training corpora. Its influence is indirect but broad.

One detail worth knowing: OpenAI's documentation notes that if you allow both GPTBot and OAI-SearchBot, it may use the result of a single crawl for both purposes rather than fetching your pages twice. That is a bandwidth saving, not a loophole.

The current canonical list lives in OpenAI's crawler documentation and Google's crawler overview. Check both when you make changes, because names get added.

The three ways sites block AI crawlers by accident

Getting robots.txt right is necessary and not sufficient. Three things override it silently.

1. Your firewall or security plugin

This is the most common cause by a wide margin. WAF rulesets, bot protection modules and security plugins frequently ship with AI crawlers blocked or rate limited by default, and they do not announce it. Your robots.txt says yes and your edge says no.

The symptom is a 403, a 429 Too Many Requests, or a challenge page returned to a crawler that robots.txt explicitly allows. From the crawler's side, a 429 looks like a site that does not want the traffic, and it backs off.

If you use Cloudflare, check the AI crawler controls in the dashboard directly rather than assuming. Most managed hosts have an equivalent setting buried somewhere in security. Cloudflare specifically splits AI bots into three behavior categories rather than one on or off switch, covered in is Cloudflare blocking ChatGPT.

2. A staging rule that shipped to production

Disallow: / on a staging site is correct. Disallow: / copied into production is a site that vanishes. It happens more often than anyone admits, and because most people never open robots.txt after launch, it can sit there for months.

Open yourdomain.com/robots.txt in a browser right now. If you see Disallow: / under User-agent: *, that is your entire problem and nothing else on this page matters until you fix it.

3. Content that only exists after JavaScript runs

Robots.txt permits the fetch. It does not guarantee the fetcher sees anything useful. If your key content renders client side, a crawler that does not execute JavaScript receives an empty shell.

Test it by disabling JavaScript in your browser and reloading a service page. Whatever remains visible is roughly what a non-rendering fetch gets.

Verifying the crawler is real

Any script can send a header saying it is GPTBot. Spoofing is common, and some of it is malicious scraping hiding behind a name you have chosen to trust.

Verification takes two checks. The user agent string has to match, and the source IP has to fall inside the published range for that agent. OpenAI publishes machine readable IP lists for each of its three bots. Anthropic and Perplexity publish equivalents.

A request that passes the name check and fails the IP check is an impostor. Block it freely. Doing so has no effect on your real visibility in ChatGPT, because the real crawler comes from the published ranges.

What to do after you publish the file

Give it about a day. Robots.txt changes take roughly twenty four hours to propagate through search systems, and longer through the crawlers that visit less often.

Then check your logs, and read them carefully, because this is where people reach the wrong conclusion.

Absent visits are inconclusive. "Not visited" and "visited and denied" are different states with different causes. A crawler that has not found you yet, or has not prioritized you, produces exactly the same empty log as one being blocked. New sites and low authority domains genuinely do get crawled rarely, and no crawler visits on a schedule you can predict.

What logs do prove is the positive case. Hits from an agent confirm access works for that agent. Requests that arrive and receive a 403 or a 429 confirm something is refusing them. Both are evidence. An empty log is not.

If robots.txt is permissive, your edge returns 200 to a test request, and logs are still empty after several weeks, you have a discovery problem rather than an access problem. Submit a sitemap, make sure the pages are internally linked, and give it time.

Access is a starting line, not a finish. Once the crawlers can reach you, the question becomes whether your pages state their facts clearly enough to be worth quoting, which is a content problem rather than a technical one. There is more on that in the notes on AEO and GEO.