If an AI crawler is getting a 403 or 405 on your site, the cause is almost never robots.txt. Those two status codes happen after the request reaches your server or CDN, and something there is actively rejecting it, usually a firewall rule, a web application firewall, a hosting provider's bot blocker, or a security plugin that treats any unfamiliar automated visitor as a threat. This walks through what the two errors mean and how to find the actual rule causing them.
What 403 and 405 actually mean
Per MDN's documentation on 403 Forbidden, the server understood the request but refused to process it, and the failure is tied to application logic such as insufficient permissions. Critically, authenticating or re-authenticating makes no difference to a 403. That rules out login walls as the cause when a crawler hits one: a crawler was never trying to log in in the first place.
Per MDN's documentation on 405 Method Not Allowed,
the server knows the request method but the target resource does not support it, and the server must send an
Allow header listing the methods that do work. A normal crawl is a GET request, so a 405 on an
ordinary page fetch usually means a server or CDN rule is misconfigured for that route rather than the crawler asking for
something unusual.
Why AI crawlers hit these specifically
- Firewall or WAF rules. Many web application firewalls ship with rules that challenge or block traffic from data center IP ranges, which is exactly where AI crawlers run from.
- Hosting provider bot blockers. Some hosts and CDNs include an automatic bot management layer that is separate from anything in your own dashboard, and it can reject a crawler before your site's own settings are ever consulted.
- CMS security plugins. WordPress and similar platforms commonly run security plugins that maintain their own bot blocklists, independent of robots.txt.
- Method restrictions. A server rule that only allows certain HTTP methods on certain paths can return a 405 to a crawler that is making an entirely normal GET request, if the rule is scoped too broadly.
Step by step diagnosis
- Check robots.txt first, to rule it out. Open
yourdomain.com/robots.txtand confirm there is noDisallowrule for the crawler in question. If there is, that is the fix, and it is not a 403 or 405 issue at all. - Check your server logs for the actual status code. Search your access logs for the crawler's user agent string and see what status code your server itself is returning, rather than guessing from a bot checker tool.
- Reproduce it with curl, using the crawler's real user agent. This isolates whether the block happens before your application code runs.
- Check CDN and WAF rules. If you use Cloudflare, Fastly, Sucuri or a similar service, check its bot management or firewall rules for anything blocking data center traffic or unfamiliar user agents.
- Check security plugins last. If the above are clear, a CMS level plugin is the remaining suspect. Review its bot or firewall settings specifically, not just the robots.txt it may generate.
A curl request using a real, published crawler user agent string looks like this:
curl -I -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" https://yourdomain.com/
The -I flag requests headers only, which is enough to see the status code without downloading the whole page. If
this returns a 403 or 405 and a normal browser request to the same URL returns 200, the difference in user agent is the thing
triggering the block, which points you straight at a firewall or bot management rule rather than your application.
Status codes an AI crawler may receive
| Code | Meaning | What it means for being cited |
|---|---|---|
| 200 | The request succeeded. | The crawler can read the page. This is what you want. |
| 301 | The resource has permanently moved to the URL in the Location header. | Fine, as long as the crawler follows the redirect to a working page. A redirect chain that never resolves is not. |
| 403 | The server understood the request but refused to process it, and authenticating makes no difference. | The page is unreachable. Usually a firewall or WAF rule, not your content. |
| 405 | The server knows the method but the resource does not support it; an Allow header is required. | The page is unreachable for that request type. Usually a server or CDN method restriction. |
| 429 | The client has sent too many requests in a given amount of time, commonly called rate limiting. | Temporary. A well behaved crawler should retry later, especially if a Retry-After header is present. |
| 503 | The server is not ready to handle the request, typically from maintenance or resource overload. | Temporary and should not be cached. Frequent 503s to crawlers point at a capacity problem worth fixing. |
Frequently asked questions
What does a 403 error mean for an AI crawler?
The server understood the request but refused it. Authenticating would not change the outcome. For an AI crawler this almost always means a firewall, WAF or bot management rule decided the request looked unwanted, not that the page itself requires a login.
What does a 405 error mean?
The server knows the request method but the target resource does not support it. The server is required to send an Allow header listing the methods that do work. A 405 on a normal page fetch usually points to a server or CDN configuration problem rather than the crawler doing anything wrong.
Is a 403 the same thing as robots.txt blocking a crawler?
No. robots.txt is a request the crawler can choose to honor before it ever asks for the page. A 403 happens after the request reaches your server or CDN, and something there refused it. Fixing robots.txt will not touch a firewall rule.
Will a security plugin block AI crawlers by default?
Many WordPress and CMS security plugins ship with bot blocking turned on, and some classify any automated user agent that is not a known search engine as suspicious. Check the plugin's bot or firewall settings specifically, not just its robots.txt output.
Should I worry about a 429 or 503 from an AI crawler?
A 429 means you are rate limiting the crawler, which is recoverable if the crawler retries later. A 503 means your server was not ready to handle the request, usually from maintenance or overload. Neither is as final as a 403, but both are worth checking in your logs if they happen often.
Checked which crawlers can actually read your pages?
Not sure whether AI crawlers can reach your site? The $49 audit checks crawler access, firewall rules and rendering, alongside 25 real buyer prompts. Get the $49 audit.
Sources: MDN, 403 Forbidden MDN, 405 Method Not Allowed MDN, 429 Too Many Requests MDN, 503 Service Unavailable MDN, 301 Moved Permanently Perplexity, Bots documentation