Marketing & Sales · Tip
Cloudflare 'Block AI bots' SEO Troubleshooting Tip
Warning on how Cloudflare's AI bot blocking breaks Google search indexing
This note explains how enabling Cloudflare's 'Block AI bots' setting alongside Bot Fight Mode can unintentionally return 4XX errors to Googlebot. Because mixed-purpose crawlers that handle both search indexing and model training are affected, the feature risks silently breaking search performance. Disabling the setting or adjusting crawl scopes prevents these crawl errors and protects indexing.
How it works
The warning: If you use Cloudflare, turn off the 'Block AI bots' setting — it appears to be terrible for your SEO.
How the author found it: turned on 'Block AI bots' and Bot Fight Mode on Cloudflare four days earlier; then noticed an unusual client error for crawl requests in Google Search Console crawl stats for moaijobs.com. The Cloudflare 'Block AI bots [Deprecating on September 15]' dialog includes a warning that from September 15, mixed-purpose crawlers (used both for search indexing and AI training) will be included in AI-training blocking — Googlebot being one. Claude also confirmed Cloudflare is sending this 4XX response to Google.
Action items: check your Cloudflare 'Block AI bots' scope setting (options: block on all pages / block only on pages with ads / do not block) and set it to 'Do not block', and check your GSC crawl stats — this may have hit sites very recently.
What it's for
- Troubleshoot sudden 4XX client crawl errors reported in Google Search Console
- Prevent accidental Googlebot de-indexing when configuring Cloudflare security rules
- Audit Cloudflare bot management settings to protect search engine rankings
Key points
- Identifies 4XX response errors triggered against Googlebot by Cloudflare's AI scraper rules
- Highlights Cloudflare's policy change grouping mixed-purpose crawlers into AI blocks
- Provides immediate fix by changing 'Block AI bots' scope to 'Do not block'
- Outlines monitoring steps within Google Search Console crawl stats
Pricing
Free
Details
- Type
- Tip
- Category
- Marketing & Sales
- Topic
- Search Engine Optimization
- Open source
- No
- Platforms
- Web
- For
- Webmasters, SEO specialists, and developers managing sites on Cloudflare
- Tags
- seo, cloudflare, googlebot, crawlers, web performance, indexing
Why it was saved
- @NithurM: Cloudflare's Block AI bots feature (with bot fight mode) was returning 4XX errors to Google's crawler, hurting SEO; anyone using Cloudflare should check the setting and their Search Console crawl stats.
Related
Empirical SEO findings on how AI models and Google AI Overviews cite content
Checklist of 18 trust signals to improve website About pages for SEO
On-page SEO practice for targeting primary keywords in homepage headings
Turn Claude Code into an automated end-to-end SEO pipeline using external APIs
Find competitor search queries your site misses to guide new content.
Host reviews on an exact-match domain to control search results and AI citations
Updated