When your firewall blocks Googlebot and AI crawlers

You can allow every crawler in robots.txt and still be invisible to them. If a firewall, CDN, or bot filter sits in front of your site, it decides who gets a page before robots.txt is ever read. Many sites lose crawl access this way and never notice.

This guide covers how those blocks happen, how to find them, and how to fix them without opening the door to scrapers.

Why good crawlers get blocked

Security tools are built to stop automated traffic. A search crawler is automated traffic. Unless the tool knows the crawler is legitimate, it may treat it like any other bot.

Common triggers include:

• Rate limits that count a crawler's fast, steady requests as an attack.
• Managed bot rules that challenge or block unknown user agents, including newer AI crawlers.
• Country or datacenter IP blocks. Most crawlers fetch from datacenter ranges, often in the US.
• JavaScript or CAPTCHA challenges. Crawlers cannot solve them.
• A default setting in a new plugin or CDN feature that someone switched on without a review.

Some CDN providers now offer one-click settings to block AI crawlers across a whole account. That can be a reasonable choice. It becomes a problem when nobody on the SEO side knows it was turned on.

What a block looks like from the outside

Blocked crawlers rarely get a clear error page. The signs are usually indirect.

In Search Console

Look at the Crawl stats report under Settings. A rise in 403, 429, or 5xx responses is the clearest signal. A sudden drop in total crawl requests with no site change is another. Pages may also show as Crawled but not indexed, or fail the live URL test.

In your server or CDN logs

Filter by user agent and check the status codes. If GPTBot, ClaudeBot, or PerplexityBot only ever get 403 or a challenge page, they are being refused. Our log file analysis guide covers how to pull this data.

In AI answers

If a page never shows up as a source for questions it answers well, the crawler may never have fetched it. This is a weaker signal, since many things affect citations. Treat it as a reason to check the logs, not as proof.

How to test your own site

1. Request a key page with a crawler user agent string using curl and compare the status code to a normal browser request.
2. Remember that this test only proves whether a rule matches the user agent. Real crawlers also come from known IP ranges, and some firewalls check both.
3. Use the URL Inspection live test in Search Console. It fetches from Google's real infrastructure, so it shows what Googlebot actually gets.
4. Check the firewall or bot management dashboard for blocked or challenged requests. Filter for crawler names and look at which rule fired.
5. Repeat the check after any change to security settings, a CDN migration, or a plugin update.

Do not assume a green result from one test means every crawler is fine. Each crawler has its own user agent and IP ranges, so test the ones you care about.

How to fix it without opening the gate to scrapers

The goal is to allow verified crawlers and keep filtering the rest. Do not switch bot protection off.

• Use your provider's verified bot list where one exists. Most large CDNs maintain one for the major search engines.
• Allow by verified identity, not by user agent alone. A user agent string is trivial to fake. Our guide on verifying AI crawlers explains the DNS and IP checks.
• Add allow rules for the specific crawlers you want, and place them above the general block rules so they match first.
• Exempt crawlers from JavaScript challenges and CAPTCHAs. They cannot pass them.
• Raise rate limits for verified crawlers, or exclude them from the limit.

AI crawlers are the harder case. Their IP lists and verification methods vary by company, and some are new and change often. Check each vendor's current documentation before writing a rule, and expect to revisit it.

Decide on purpose, then write it down

Blocking some AI crawlers is a valid business call. The mistake is blocking them by accident, or believing robots.txt says one thing while the firewall does another. Keep the two in agreement. Our robots.txt guide covers the policy side.

If robots.txt and your firewall disagree, the firewall wins.

What to do this week

Open your firewall or bot management dashboard and read the rules that block or challenge traffic. Then open the Crawl stats report and check for 403 and 429 spikes. If a crawler you want is being refused, add an allow rule based on verified identity, retest with the live URL test, and note the change so the next security update does not undo it.