Blocking AI Crawlers Without Blocking Google

Sometime this quarter, someone on your team is going to open Cloudflare and turn off AI training. It's a reasonable thing to want. But if they reach for the obvious toggle, they can take Googlebot down with it, and nobody will notice until traffic slides three weeks later.

That's not a hypothetical risk. It's written into how the new controls work, in Cloudflare's own documentation.

What Cloudflare changed on July 1

On July 1, 2026, Cloudflare replaced its single "Block AI bots" switch with three separate controls, split by what a crawler is actually doing with your content:

• Search: "any behavior that collects or indexes your content, so it can answer questions about it later."
• Agent: "automated behavior that is acting, usually in real time, on a person's behalf, to get something done right now."
• Training: "a crawler taking your content to train or fine-tune a model."

For each one you can allow it, block it everywhere, or block it only on pages that show ads. It's live for every Cloudflare customer, including the free plan.

The split is genuinely useful. Most publishers don't feel the same way about a bot that sends them referral traffic as they do about a bot that quietly absorbs their archive into a model. The old toggle couldn't tell those apart. This one can.

But there's a catch in how it resolves, and it's the reason we're writing this.

The trap: crawlers that do more than one job

Googlebot doesn't do just one thing. It crawls to build the search index, and Google uses that same index to power its AI features. So what happens when one crawler lands in two of Cloudflare's three buckets?

Cloudflare answers this directly. Multi-purpose crawlers "will be allowed/blocked according to all of their behaviors." The most restrictive rule wins. And Cloudflare names names:

Since the defaults will be enforced by the most restrictive applicable rules, multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked

Read that twice. If you block Training, you also block Googlebot. And Bingbot. And Applebot.

So the intuitive move, "leave Search on, switch Training off, best of both worlds," does not do what it looks like it does. You don't get search traffic minus the training. You get neither.

This is the kind of thing that's invisible from a dashboard. The toggle says Search is still allowed. Your rankings say otherwise, eventually.

A date worth putting in the calendar

There's a second half to this. Starting September 15, 2026, new Cloudflare domains get new defaults: Training and Agent blocked on ad-displaying pages, Search still allowed. Existing customers can opt out of those defaults any time before that date.

Combine that with the most-restrictive rule and you can see why it matters. If you run ads and you're on Cloudflare, check what your defaults will be well before mid-September rather than after.

What Google actually gives you

Here's the good news: Google already built the thing people want. It's a separate robots.txt token called Google-Extended, and Google's documentation is unambiguous about its blast radius:

Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.

Google-Extended governs whether your content trains future Gemini models and grounds some of Google's other systems. Because it's a robots.txt token, you use it like any other user agent:

User-agent: Google-Extended
Disallow: /

That gets you exactly the trade most teams are after: out of Gemini training, still in Google Search. No collateral damage to Googlebot, because you're talking to Google in Google's own vocabulary instead of asking a proxy to guess.

The same pattern works per vendor. Cloudflare's managed robots.txt already recognizes the usual tokens, including GPTBot, ClaudeBot, CCBot, and Google-Extended. Each one is a specific crawler with a specific purpose, so blocking one doesn't ripple into your search visibility.

What Google-Extended does not do is pull you out of AI Overviews or AI Mode. Google is clear that those run off Search, and that "robots.txt directives for Googlebot is the control for site owners." If you want to limit what surfaces there, the levers are the preview controls: nosnippet, data-nosnippet, max-snippet, or noindex. Google also notes there are "no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary."

So the honest summary, and it's worth being blunt about it: you can opt out of training, and you can limit snippets. You can't stay in Google Search and stay out of AI Overviews. Not through robots.txt, anyway.

Content signals: a preference, not a block

Cloudflare's other piece here is Content Signals, machine-readable directives in robots.txt that describe how your content may be used rather than whether it can be fetched at all. There are three:

• search: "building a search index and providing search results (e.g., returning hyperlinks and short excerpts from your website's contents). Search does not include providing AI-generated search summaries."
• ai-input: "inputting content into one or more AI models (e.g., retrieval augmented generation, grounding, or other real-time taking of content for generative AI search answers)."
• ai-train: "training or fine-tuning AI models."

For sites without an existing robots.txt, Cloudflare's managed version writes this:

User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /

Look closely at what that is, and what it isn't. Allow: / means everything still gets fetched. The content signal is a stated preference sitting next to an open door. It's a "please don't," not a 403.

That's a feature, not a weakness. A preference costs you nothing in crawlability. A block can cost you your index. If your goal is to put your position on the record without gambling your search traffic, signals are the cheap move.

What we'd actually do

1. Name what you're protecting. "Block AI" is three different decisions wearing one coat. Training is the one most teams actually object to. Say that out loud before anyone touches a switch.
2. Do training opt-outs in robots.txt, not at the edge. Per vendor, per token. Google-Extended and friends won't drag Googlebot along with them.
3. Leave Cloudflare's Search control on. Obvious, but somebody has to say it.
4. Treat the Training toggle as a search decision. Given the most-restrictive rule, that's what it is. Price it accordingly.
5. Add content signals if you want to state a preference without a block.
6. Check your defaults before September 15 if you're on Cloudflare and you monetize with ads.

Verify it, don't assume it

Every change here is invisible from the outside until rankings move, which is the worst possible feedback loop. So close it yourself:

• Read the robots.txt you're actually serving. If Cloudflare manages the file, what you serve may not be what's in your repo. Our robots.txt tester shows the live file and which user agents it blocks.
• Crawl your key templates and confirm they still return a 200 to a search user agent.
• Watch your logs. If Googlebot's hit rate falls off a cliff right after somebody flipped a toggle, that's your answer, and log-file analysis surfaces it in days instead of quarters.
• Check Search Console for a jump in "Blocked by robots.txt." Here's how we work through crawl errors when they show up.

If you want the wider view on declaring an AI crawler policy, we covered llms.txt and controlling AI crawlers separately.

FAQ

Does blocking Google-Extended hurt my rankings?

No. Google's documentation says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

Will blocking AI training keep me out of AI Overviews?

No. AI Overviews and AI Mode run off Google Search itself. Google-Extended governs training and grounding in other Google systems. What appears in Search and its AI features is controlled by Googlebot's robots.txt directives plus preview controls like nosnippet and max-snippet.

Is Cloudflare's Training toggle the same as a robots.txt training opt-out?

Not even close, and this is the whole point. A robots.txt token talks to one crawler. Cloudflare's Training control applies the most restrictive matching rule, so multi-purpose crawlers like Googlebot, Applebot, and Bingbot get caught in it.

Do content signals actually stop anyone?

They're a stated preference, not enforcement. The managed file pairs ai-train=no with Allow: /, so the content is still served. Treat signals as policy and blocks as enforcement, and don't confuse the two.