If your site sits behind Cloudflare, there is a real chance its AI bot settings are blocking the exact crawlers you want citing you. Cloudflare does not treat AI bots as one category. It splits them into three behaviors, and depending on how your zone is configured, one or more of them may already be blocked without anyone having chosen that on purpose.
The three categories, and the options for each
At the time of writing (October 2026), Cloudflare's dashboard groups AI bot behavior under Security Settings, Configure AI bot policies, into three categories, each with the same three choices:
| Category | What it covers |
|---|---|
| Search | Indexing content to answer questions later, which includes the crawlers that let an AI engine cite your site. |
| Agent | Real time automated activity carried out on behalf of a user, such as a bot fetching a page because someone asked a direct question. |
| Training | Content used for model training, including crawlers with a mixed purpose that combines training with another behavior. |
Each category can independently be set to:
- Block (on all pages), a zone wide block of verified bots in that category, plus similar unverified bots.
- Block on pages with ads, which blocks only on pages Cloudflare's automated detection identifies as carrying ads.
- Allow (do not block), which applies no blocking for that category.
A crawler that mixes Search and Training purposes is blocked under any configuration that blocks AI training, even if Search itself is set to Allow. That is worth knowing if a crawler you expected to pass through turns out blocked anyway.
What changed on September 15, 2026
For domains added to Cloudflare from September 15, 2026 onward, Training and Agent classified bots are blocked by default on pages that display ads, while Search stays allowed by default. Cloudflare's documentation does not state what the default configuration is for domains that existed before that date, so if your site has been on Cloudflare longer than that, check your own dashboard rather than assuming either way.
There is also a legacy, simpler toggle. At the time of writing (October 2026), it still appears in the dashboard under Security Settings, Block AI bots, and Cloudflare's own documentation marks it as deprecating. If you see that older setting instead of the three category view, your zone has not moved to the newer configuration yet.
The setup for a business that wants to be cited
If being named inside ChatGPT, Perplexity or similar answers matters to you, the practical setup is to set Search to Allow, decide separately on Agent depending on whether you want user triggered fetches to work, and block Training if you specifically do not want your content used to train a model. This mirrors the same split this site recommends for OpenAI's own three crawlers: training and citation are different decisions, and Cloudflare's categories let you make them separately rather than as one blanket block.
How to confirm it with curl
Changing a dashboard setting is not the same as confirming it worked. Request a page using a real crawler user agent and read the status code back:
curl -I -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" https://yourdomain.com/A 200 means the request got through. If you have set Search to Allow and still see a 403, the AI bot policy is probably not the cause: check your other firewall rules, since Cloudflare's own bot categories are only one layer among several that can reject a request.
Frequently asked questions
How do I check Cloudflare's AI bot settings?
In the Cloudflare dashboard, go to Security Settings and look for Configure AI bot policies, at the time of writing (October 2026). It shows three categories, Search, Agent and Training, each with its own Block, Block on pages with ads, or Allow setting.
Should I block all three Cloudflare AI bot categories?
Not if you want to be cited. Training covers crawlers gathering data for model training, which you may want to block. Search covers crawlers that answer questions by citing your site, which you generally want to allow. Agent covers real time automated activity on a user's behalf.
What changed on Cloudflare on September 15, 2026?
For new domains from that date, Cloudflare began blocking Training and Agent classified bots by default on pages that display ads, while Search stays allowed. Cloudflare's documentation does not state what the default is for domains created before that date.
Does the old Block AI bots toggle still exist?
At the time of writing (October 2026), Cloudflare's own documentation still shows it under Security Settings, Block AI bots, marked as deprecating. Check your own dashboard rather than assuming, since this is a setting in transition.
How do I confirm a crawler can actually reach my site through Cloudflare?
Request a page with curl, setting the crawler's published user agent, and check the status code. A 200 means it got through. A 403 after you have set Search to Allow usually means a different firewall rule, not the AI bot policy, is the one blocking it.
Not sure what your own Cloudflare zone is set to?
Not sure whether AI crawlers can reach your site? The $49 audit checks crawler access, firewall rules and rendering, alongside 25 real buyer prompts. Get the $49 audit.
Sources: Cloudflare, Configure AI bot policies Cloudflare, Bot concepts