To check whether your site is blocking AI crawlers, run four tests in order: read your robots.txt in a browser, request a page while sending each AI user agent string, check your server logs for real crawler hits, and load a page with JavaScript disabled. Most blocks turn out to sit in a firewall or security plugin rather than in robots.txt, which is why the file alone is not enough to confirm access.
Work through them in that order. Each one rules out a different cause, and stopping at the first is how most people end up confidently wrong.
Check 1: Read your robots.txt
Open yourdomain.com/robots.txt in a browser. Sixty seconds.
You are looking for three things.
A blanket block. If you see Disallow: / under
User-agent: *, your site is closed to everything. This is usually a staging
configuration that shipped to production, and it is more common than anyone likes to
admit. Fix it and stop reading, because nothing else matters until you do.
Named AI agents being disallowed. Look for GPTBot,
OAI-SearchBot, ChatGPT-User, ClaudeBot,
PerplexityBot, Google-Extended, CCBot and
Applebot-Extended. A Disallow: / under any of these is a
deliberate looking block that frequently nobody deliberately made. Security plugins and
CMS templates add these lines on their own.
Path level blocks that catch your content. A
Disallow: /resources/ or Disallow: /blog/ cuts off exactly the
pages you most want quoted. Check that no rule covers a directory holding real content.
If robots.txt is clean, do not conclude access is working. It is the file that gets checked, and it is rarely the thing making the decision.
Check 2: Request your site as each crawler
This is the test that finds the real problem.
Request one of your content pages while sending an AI crawler's user agent string, and
look at what comes back. Do this for each agent you care about:
GPTBot, OAI-SearchBot, ChatGPT-User,
ClaudeBot, PerplexityBot.
What the responses mean:
A 200 with your real HTML means access is working for that
agent. Confirm the HTML actually contains your content, not just a shell.
A 403 Forbidden means something is refusing the request by
name. This is almost always a firewall rule or a bot protection module, and it overrides
robots.txt entirely. A 405 Method Not Allowed is a related but different
problem, usually a server or CDN rule rather than a bot block; both are covered in
how to fix AI crawler 403 and 405 errors.
A 429 Too Many Requests is the quiet killer. A rate limit
is treating the crawler as suspicious traffic. From the crawler's side a
429 reads as a site that does not want the visit, and it backs off,
sometimes for a long time. This has been one of the most common causes of missing
ChatGPT citations on sites whose robots.txt was perfectly permissive.
A challenge or CAPTCHA page returned as a 200 is the most
deceptive result, because the status code says success. Look at the body, not the code.
Two important details. Test a real content page rather than the homepage, because firewall rules are often path specific and homepages frequently get exemptions the rest of the site does not. And test from outside your own network, since your IP may be allow-listed in ways a crawler's is not.
Check 3: Look at your server logs
The first two checks tell you what should happen. Logs tell you what did.
Filter your access logs for the AI user agent tokens over the last thirty days and count hits per agent. This is the only real proof that access is open, and it answers a question the other tests cannot.
Zero hits across thirty days with a clean robots.txt and clean
curl responses means something intermittent is intervening, or the crawlers
simply have not found reason to visit. New sites and low authority domains genuinely do
get crawled rarely, which is a discovery problem rather than an access problem.
Hits from ChatGPT-User specifically are the most
interesting line in your logs. That agent only fetches when a real user's question sends
it, so those hits are a rough map of which of your pages are surfacing in live
conversations. If one URL appears repeatedly, that page is earning its keep.
Hits that look wrong are worth a second look. The user agent header is trivially spoofable and scrapers impersonate these bots constantly. Real verification means the token matches and the source IP falls inside the published range for that agent. OpenAI, Anthropic and Perplexity all publish machine readable IP lists, linked from their respective crawler documentation. A request that passes the name check and fails the IP check is an impostor, and blocking it costs you nothing.
Without that second step your bot numbers will look healthier than they are.
Check 4: Load the page without JavaScript
Robots.txt permits the fetch. It does not guarantee the fetcher sees anything.
Disable JavaScript in your browser and reload a key service page. Whatever remains visible is roughly what a non-rendering fetch receives.
If the page goes blank, or your prices, service descriptions and location details
vanish, you have an access problem that no robots.txt change will fix. The crawler
arrived, got a 200, and received an empty shell.
This one catches a lot of modern sites. Client side rendering is the norm in several popular frameworks, and it is entirely possible to have a fast, well built site that returns nothing useful to a crawler that does not execute scripts.
The fix is server side rendering or static pre-rendering for the pages that matter. You do not need it everywhere. You need it on the pages you want quoted.
What to do with the results
If checks 1 and 2 fail, you have a permissions problem. Fix robots.txt, then find the firewall rule. Cloudflare, most managed hosts and most WordPress security plugins have AI crawler controls somewhere in their settings, often enabled by default without announcement. Allow the agents you want and re-run check 2 to confirm.
If checks 1 and 2 pass but check 3 is empty, you have a discovery problem rather than an access problem. Submit a sitemap, make sure your pages are internally linked, and give it a few weeks. Crawlers visit sites they have a reason to visit.
If check 4 fails, you have a rendering problem, and it is the most expensive of the four to fix but also the one with the clearest payoff.
If all four pass and you are still not being cited, that is genuinely good news. The technical layer is done and the problem has moved to content and entity clarity, which is the part you control directly. That means writing pages that state their answer in the first sentence, putting facts in text rather than images, and making sure the description of your business is identical everywhere a model might check it. The notes on AEO and GEO go through both.
Run these four checks once a quarter. Firewall rules get reset during routine maintenance, plugins update their defaults, and a site that was reachable in March is not automatically reachable in September.