# Cloudflare 'Block AI bots' SEO Troubleshooting Tip

> Warning on how Cloudflare's AI bot blocking breaks Google search indexing

This note explains how enabling Cloudflare's 'Block AI bots' setting alongside Bot Fight Mode can unintentionally return 4XX errors to Googlebot. Because mixed-purpose crawlers that handle both search indexing and model training are affected, the feature risks silently breaking search performance. Disabling the setting or adjusting crawl scopes prevents these crawl errors and protects indexing.

## How it works

The warning: If you use Cloudflare, turn off the 'Block AI bots' setting — it appears to be terrible for your SEO.

How the author found it: turned on 'Block AI bots' and Bot Fight Mode on Cloudflare four days earlier; then noticed an unusual client error for crawl requests in Google Search Console crawl stats for moaijobs.com. The Cloudflare 'Block AI bots \[Deprecating on September 15\]' dialog includes a warning that from September 15, mixed-purpose crawlers (used both for search indexing and AI training) will be included in AI-training blocking — Googlebot being one. Claude also confirmed Cloudflare is sending this 4XX response to Google.

Action items: check your Cloudflare 'Block AI bots' scope setting (options: block on all pages / block only on pages with ads / do not block) and set it to 'Do not block', and check your GSC crawl stats — this may have hit sites very recently.

## What it is for

- Troubleshoot sudden 4XX client crawl errors reported in Google Search Console
- Prevent accidental Googlebot de-indexing when configuring Cloudflare security rules
- Audit Cloudflare bot management settings to protect search engine rankings

## Key points

- Identifies 4XX response errors triggered against Googlebot by Cloudflare's AI scraper rules
- Highlights Cloudflare's policy change grouping mixed-purpose crawlers into AI blocks
- Provides immediate fix by changing 'Block AI bots' scope to 'Do not block'
- Outlines monitoring steps within Google Search Console crawl stats

## Pricing

Free

## Details

- Type: Tip
- Category: [Marketing & Sales](https://xiry.app/topics/marketing-sales.md)
- Topic: [Search Engine Optimization](https://xiry.app/topics/marketing-sales/search-engine-optimization.md)
- Open source: no
- Platforms: Web
- For: Webmasters, SEO specialists, and developers managing sites on Cloudflare
- Tags: seo, cloudflare, googlebot, crawlers, web performance, indexing

## Why it was saved

- [@NithurM, 3 September 2026](https://x.com/NithurM/status/2095372965680672918): Cloudflare's Block AI bots feature (with bot fight mode) was returning 4XX errors to Google's crawler, hurting SEO; anyone using Cloudflare should check the setting and their Search Console crawl stats.

## Related

- [10 AI Search Optimization Findings from 1 Billion Data Points](https://xiry.app/resources/10-ai-search-optimization-findings-from-1-billion-data-point): Empirical SEO findings on how AI models and Google AI Overviews cite content (List, Free)
- [18 About Us Page Elements for E-E-A-T](https://xiry.app/resources/18-about-us-page-elements-for-e-e-a-t): Checklist of 18 trust signals to improve website About pages for SEO (Tip, Free)
- [Above-the-Fold H1 Heading Optimization for Homepage SEO](https://xiry.app/resources/above-the-fold-h1-heading-optimization-for-homepage-seo): On-page SEO practice for targeting primary keywords in homepage headings (Tip, Free)
- [Claude Code for SEO Playbook](https://xiry.app/resources/claude-code-for-seo-playbook): Turn Claude Code into an automated end-to-end SEO pipeline using external APIs (Tip, Free)
- [Competitor Keyword Gap Analysis Workflow](https://xiry.app/resources/competitor-keyword-gap-analysis-workflow): Find competitor search queries your site misses to guide new content. (Tip, Freemium)
- [Exact-Match Review Domain SEO & AI Citation Strategy](https://xiry.app/resources/exact-match-review-domain-seo-and-ai-citation-strategy): Host reviews on an exact-match domain to control search results and AI citations (Tip, Free)

Updated 11 October 2026. Page: https://xiry.app/resources/cloudflare-block-ai-bots-seo-troubleshooting-tip

Make your own library: Xiry takes your saved reels, tweets, Reddit bookmarks and Watch Later videos, makes them searchable and pulls the useful stuff out, like this library. It's invite-only for now; join the waitlist at https://xiry.app/#waitlist.