User-Agent and WAF-Based Access Restrictions
User-agent and WAF-based access restrictions are technical rules or protection mechanisms that allow, block, challenge or limit requests depending on the user agent or on a web application firewall or bot-management system.
Such rules can target specific crawlers, but can also affect automated traffic in general or, in misconfigurations, even search engines and real users.
What does UA mean?
UA = user agent. The user agent is an identifier a client sends with an HTTP request. Examples:
- Browser
- Googlebot
- GPTBot
- ClaudeBot
- CCBot
The user agent is a self-declaration by the client and can be changed easily.
What does WAF mean?
WAF = web application firewall. A WAF typically sits in front of the web application and evaluates incoming requests. Among other things, it can:
- allow
- block
- challenge
- throttle
- log
Not every WAF behaves identically. Rule sets, signals and default behaviour differ by system and configuration.
How does user-agent-based blocking work?
A simple example: if the user agent is CCBot, then 403. That is a relatively simple rule.
It can be implemented at different layers, at the web server, in the CDN, in the WAF or in a reverse proxy.
How does WAF-based blocking work?
WAF systems can consider more signals than just the user agent, depending on the system, for example:
- IP address
- network
- bot verification
- request patterns
- request rate
- browser behaviour
- cookies
- JavaScript execution
- reputation
- known bot signatures
Deliberately always „can consider“, not „always considers“. Which signals actually apply depends on the given system and its configuration.
Why is a user-agent test not enough?
A request with Googlebot as its user agent is not a real Googlebot. Likewise, a request with CCBot as its user agent is not necessarily representative of a real Common Crawl request.
Because WAFs can use additional signals, a test server might return the following, for example:
…while the real Googlebot is treated differently thanks to verified bot detection. The test only measures the reaction to the reported string, not the real crawler.
A challenge is not the same as a block
Cloudflare example: the response header cf-mitigated: challenge signals a challenge response, not a permanent deny rule.
A real browser may pass such a check. A simple HTTP client will not. So a 403 can technically be a challenge rather than a permanent block.
CCBot (Common Crawl) checks robots.txt, uses HTTP GET and follows redirects, but runs no JavaScript and uses no cookies. A protection mechanism that requires JavaScript or cookie support can therefore effectively exclude CCBot, even if robots.txt allows it.
How a protection layer computes each fingerprint internally cannot be determined from the outside. Rely only on documented signals such as status code and response headers, not on assumptions about internal scoring logic.
Which status codes are relevant?
The status code alone is often not enough to determine the cause. Headers, response body, history and logs complete the picture.
Can real users be affected too?
Yes. Aggressive bot-protection systems can also challenge or block real users. Possible factors:
- VPN
- unusual IP
- certain regions
- privacy tools
- disabled JavaScript
- unusual browser behaviour
- high request frequency
Can search engines be affected?
Yes, especially in misconfigurations, but this needs careful wording. Googlebot can be treated differently through verified-bot mechanisms than an arbitrary client carrying a Googlebot string.
An external Googlebot-UA test is therefore not proof. More reliable are:
- Google Search Console
- real server logs
- verified IP / DNS signals
- WAF logs
Why is this relevant for Generative Engine Optimization (GEO)?
Generative Engine Optimization (GEO) also considers technical accessibility. The access of a single crawler is, however, only one part of the overall picture: AI systems can obtain content via various sources and retrieval paths.
WAF blocking of a single crawler does not automatically mean no AI visibility.
How do you recognise user-agent or WAF-based access restrictions?
1. Compare responses
Compare requests with browser and crawler user agents (both remain the same test client against the same infrastructure).
2. Check status codes
Watch for 403, 429 and others.
3. Check response headers
For example server, Cloudflare-specific headers or challenge signals.
4. Check the response body
Watch for human-verification, challenge pages or „access denied“ content.
5. Check historical crawl data
Did Common Crawl access the site successfully in the past?
6. Check owner data
Consult WAF/CDN logs, Google Search Console and server logs.
Examples
Example 1, user-agent-based
A hint of a user-agent-dependent access restriction. It does not follow automatically that the real CCBot is treated identically.
Example 2. WAF/challenge-based
cf-mitigated: challengeThe restriction is evidently not solely user-agent dependent. The response points to an upstream challenge.
Example 3, historical change
A hint of changed technical reachability for Common Crawl, no causality without further evidence.
Placing it within crawler accessibility
User-agent and WAF-based rules are only one part of the broader concept of crawler accessibility. It also includes robots.txt, rate limits, authentication, technical errors and further access layers.
Hands-on analysis
The Common Crawl Decoder shows how Common Crawl's actual access has developed over time. This lets you examine changes in successful retrievals, 403 responses or other HTTP status codes, for example.
Open the Common Crawl Decoder →The decoder measures Common Crawl signals. It does not directly measure Googlebot, GPTBot, ClaudeBot or other crawlers.
Common misinterpretations
- „All 403s mean bot blocking.“
- Wrong. A 403 can be a challenge, a misconfiguration or a general protection rule.
- „A Chrome user agent simulates a real browser.“
- Not fully. A WAF can check many additional signals that a mere UA string does not represent.
- „A Googlebot UA tests Google.“
- Wrong. It only tests the reaction to the string, not the real, verified Googlebot.
- „A browser 200 means Googlebot also gets a 200.“
- Not necessarily. Browsers and crawlers can be treated differently.
- „A CCBot 403 means ChatGPT does not know the page.“
- Wrong. Common Crawl is only one possible source; AI systems can obtain content via other paths.
Sources and further documentation
- Google Search Central. Verify Google crawlers and fetchers
- Cloudflare Docs. Detect a challenge response (cf-mitigated)
- Cloudflare Docs. Web Application Firewall (WAF)
- Cloudflare Docs. Verified Bots
- AWS WAF. Developer Guide
- MDN. User-Agent header
- MDN. HTTP response status codes
- Common Crawl. FAQ (CCBot: robots.txt, no JavaScript, no cookies)
Last reviewed: August 7, 2026