Anthropic explains its three web crawlers in its own support article on crawling and blocking, and the short version is that ClaudeBot, Claude-SearchBot and Claude-User are not the same bot wearing three names. Each one does a different job, and blocking all three with one robots.txt rule often gives up more than a business actually meant to.
What each crawler does
| Crawler | What it does | What blocking it does |
|---|---|---|
| ClaudeBot | Collects web content that could contribute to model training. | Signals that future material should be excluded from Anthropic's AI model training datasets. |
| Claude-SearchBot | Navigates the web to improve search result quality inside Claude. | Stops indexing for search optimization, which may reduce visibility and accuracy when Claude answers a search style question. |
| Claude-User | Accesses a website when a person asks Claude a direct question. | Stops retrieval for that user's query, which may reduce visibility in user directed web search. |
That table comes straight from Anthropic's own wording. The useful part for a business owner is what it implies: you can treat training and citation as two separate decisions, because Anthropic built them as two separate bots.
The setup most businesses actually want
If the goal is to stop your content feeding a training dataset but still be named when someone asks Claude a question, the answer is to block ClaudeBot and allow Claude-SearchBot and Claude-User. That is not a workaround. It is exactly what the two bots are built to let you do separately, and it is the same logic this site recommends for OpenAI's three crawlers, which split the same way.
Copy and paste robots.txt
Option A: allow all three. The simplest setup, and the one to use if you have no training objection.
# Anthropic, allow all three
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /Option B: block training, allow citation. Keeps Claude-SearchBot and Claude-User working, stops ClaudeBot.
# Anthropic, block training only
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
Anthropic's own examples use exactly this pattern: a bare User-agent: ClaudeBot block with either
Crawl-delay: 1 to rate limit it or Disallow: / to block it outright. The file has to sit in the top
level directory of your subdomain, the same place every other robots.txt rule lives. A copy sitting on a different subdomain
does not apply.
Crawl-delay and the IP list
Anthropic states that its crawlers respect robots.txt and support the non-standard Crawl-delay extension where
appropriate, so a line like Crawl-delay: 1 under User-agent: ClaudeBot is a real option if you want to
slow it down rather than block it.
Anthropic also publishes an IP address list for verification, at claude.com/crawling/bots.json. It is worth reading Anthropic's own caveat here: the company says IP blocking may not reliably guarantee an opt-out, and recommends robots.txt as the primary way to block its crawlers. Treat the IP list as a way to confirm a request really came from Anthropic in your server logs, not as your main defense.
How this compares to Perplexity
Anthropic says its bots respect robots.txt. That is not universal among AI companies. Perplexity's own documentation says Perplexity-User, the bot that fetches a page when a Perplexity user asks a direct question, generally ignores robots.txt rules, because the fetch was requested by a person rather than run as a background crawl. Anthropic's Claude-User is the closest equivalent on its side, and Anthropic does not carve out the same exception for it in the article cited above. If you are trying to reason about these bots from first principles, do not assume every company's user-triggered bot behaves the same way; check that company's own documentation.
Frequently asked questions
Does Anthropic have more than one crawler?
Yes, three: ClaudeBot, Claude-SearchBot and Claude-User. They do different jobs, so blocking all three with one rule can cost you more than you meant to give up.
Can I block ClaudeBot and still show up when someone asks Claude about my business?
Yes. ClaudeBot is the one that gathers content for training. Claude-User is the one that fetches your page when someone asks Claude a question. Blocking ClaudeBot and allowing Claude-User and Claude-SearchBot is a normal setup.
Does Anthropic respect robots.txt?
Anthropic's support article says its crawlers respect robots.txt and the non-standard Crawl-delay extension where appropriate. That is stronger than some other AI companies state for their user-triggered bots.
Is blocking Anthropic's IP addresses enough?
Anthropic itself says IP blocking may not reliably guarantee an opt-out, and recommends robots.txt instead. The IP list is published for verification, not as the primary way to block the crawlers.
Where does the robots.txt rule for ClaudeBot need to go?
In the robots.txt file at the top level directory of your subdomain, the same place every other robots.txt rule lives. A rule on a different subdomain does not apply.
Not sure what is actually reaching your site?
Not sure whether AI crawlers can reach your site? The $49 audit checks crawler access, firewall rules and rendering, alongside 25 real buyer prompts. Get the $49 audit.
Sources: Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler (updated April 7, 2026, fetched October 11, 2026) Anthropic crawler IP list Perplexity, Bots documentation