OpenAI runs three separate web crawlers. GPTBot collects training data for future models, OAI-SearchBot builds the index behind ChatGPT's search feature, and ChatGPT-User fetches pages live when someone's prompt triggers browsing. Only the last two affect whether ChatGPT can cite your business today, which means a robots.txt written around GPTBot alone is solving the wrong problem.
That distinction is worth more than it sounds. A large share of the sites that are invisible in ChatGPT blocked the wrong bot, or allowed the wrong one, and have been waiting for results that were never going to arrive.
Why OpenAI split them up
The three bots exist because the three activities warrant genuinely different consent decisions.
Training is a one to many operation. A single crawl feeds a model that answers millions of future questions, and your content becomes part of a system you have no visibility into. Plenty of publishers reasonably want to opt out of that.
Search indexing is closer to what Bing and Google have always done. Your page gets stored so it can be retrieved and linked when relevant.
Live fetching is different again. It happens because a specific human asked a specific question and the assistant went to look. It is closer to a browser request made on somebody's behalf than to autonomous crawling.
Bundling those three under one user agent would have forced an all or nothing choice. Splitting them lets a publisher say no to training and yes to citation, which is exactly what many news organizations have done.
OpenAI is not the only one that splits its crawlers this way. Anthropic runs the same pattern with ClaudeBot, Claude-SearchBot and Claude-User, and Perplexity separates its own search crawler from the bot that fetches a page only when a person asks it a direct question, detailed in the PerplexityBot user agent breakdown.
GPTBot
- User agent token:
GPTBot - Purpose: collects training data for future OpenAI models
- Effect on citations today: none
GPTBot has been running since August 2023 and it is the one everybody has heard of, which is precisely why it causes so much confusion. Blocking GPTBot stops your content contributing to future training runs. It does not remove you from ChatGPT, and it does not stop ChatGPT citing you.
There is one long term consideration. The model's internal picture of your business, the thing that lets it name you without going to look, is built from training data. Blocking GPTBot forever means your business only ever exists to ChatGPT through live retrieval. For a publisher protecting an archive that trade may be worth it. For a service business trying to get recommended, it rarely is.
Its share of AI crawling traffic has grown substantially, roughly doubling between mid 2024 and mid 2025, which tells you how much more OpenAI is now collecting.
OAI-SearchBot
- User agent token:
OAI-SearchBot - Purpose: builds the index that ChatGPT Search retrieves from
- Effect on citations today: direct and significant
This is the one most people miss. OAI-SearchBot behaves like a conventional search crawler. It visits, it indexes, and the resulting index is what ChatGPT queries when a user asks something that needs current information.
Block it and your pages stop being candidates for citation. Not deprioritized. Removed from the pool. Everything else you do for AI visibility is downstream of this bot being able to reach you.
OpenAI's documentation notes something useful here: if you allow both GPTBot and OAI-SearchBot, it may use a single crawl to serve both purposes rather than fetching your pages twice. That coordination explains a pattern site owners have observed in logs, where multiple OpenAI agents appear around the same prompt.
ChatGPT-User
- User agent token:
ChatGPT-User - Purpose: live fetches triggered by a user's request
- Effect on citations today: none directly. It retrieves on request, it does not decide inclusion
When a ChatGPT user asks something that needs a live look, or clicks a source link, or uses a Custom GPT that reaches out to your site, this is the agent that arrives. OpenAI has expanded its documented scope to include Custom GPT requests and GPT Actions, which makes it less a crawler and more a proxy for a human browsing on request.
A correction worth making explicitly, because getting this backwards is common. ChatGPT-User does not determine whether you appear in ChatGPT Search. OpenAI's own overview of OpenAI crawlers is clear on that. It handles fetches triggered by a specific user action, and blocking it stops those individual retrievals, but it is not the inclusion control.
OAI-SearchBot is the agent that matters for citation. It builds the index ChatGPT Search retrieves from. If you allow only one OpenAI agent, allow that one.
The clean split is: GPTBot for training, OAI-SearchBot for search inclusion, ChatGPT-User for user-triggered fetches. Three agents, three different decisions.
There is a useful diagnostic hidden in this. Because ChatGPT-User only fetches when a user's question sends it, hits from this agent in your logs are a rough proxy for which of your pages are actually surfacing in conversations. If one page is getting fetched repeatedly, that page is doing work.
The configurations people actually run
Allow all three. The right default for any business that wants to be recommended. You accept training use in exchange for full citation eligibility.
Block GPTBot, allow OAI-SearchBot and ChatGPT-User. The publisher position. You keep your archive out of training and stay fully citable. Common among news organizations and paid content businesses.
Allow GPTBot, block the other two. Almost always a mistake, and it happens by accident more than by choice. You are contributing content to training while blocking the retrieval surface that turns that contribution into visibility. If you find this on your site, it was not a decision.
Block all three. Defensible if you are a subscription business with nothing to gain from AI referral. Understand that you are opting out of a discovery channel, not just a training set.
Testing your configuration
The three tokens are distinct, so you can test each independently rather than guessing.
Request a page from your own site while sending each user agent string, and check what
comes back. A 200 with your real HTML means access is working. A
403, a 429, or a challenge page means something between
robots.txt and the crawler is intervening, usually a firewall rule or a bot protection
module rather than the file itself.
Do this against a real content page, not the homepage. Firewall rules are often path specific, and the homepage frequently gets an exemption the rest of the site does not.
Then check your logs. Filter for the three tokens and count hits over the last thirty days. If robots.txt permits all three and none of them appear, robots.txt is not the thing making the decision on your server.
Verifying the bot is actually OpenAI's
The user agent header is trivially spoofable, and scrapers impersonate these bots constantly to get access that would otherwise be denied.
Real verification is a two step check. The token has to match, and the source IP has to fall inside OpenAI's published range for that specific agent. OpenAI publishes machine readable IP lists for each of the three, linked from its bots documentation.
A request that passes the name check and fails the IP check is an impostor, and blocking it costs you nothing in real visibility. Build this into your log analysis rather than checking ad hoc, because impostor traffic will otherwise inflate your bot numbers and make your access look healthier than it is.
What this does not solve
Crawler access is the entry requirement, not the outcome. Once all three agents can reach you and you can prove it from your logs, the remaining question is whether your pages state their facts clearly enough that a model chooses to quote them over a competitor's.
That is a content and entity problem. It is answered by putting the complete answer in the first sentence of each section, stating facts as facts rather than implying them through images or logos, and making sure the description of your business is identical everywhere a model might check it. The notes on AEO and GEO cover both in more detail.
If your logs show healthy crawl volume and you still are not being named, that is genuinely good news. It means the problem has moved from access to content, which is the part you control directly.