- Checking AI crawler access is a three-layer verification (robots.txt, server logs, and the CDN/WAF layer) because any one of them can silently override the other two, and reading only one gives you a false sense of certainty.
- A July 2026 study of 10,894 domains (HasData) found 39.5% of GPTBot Disallow rules in robots.txt are ignored by GPTBot itself, and in a separate finding from the same study, Google AI Mode cited domains that block at least one AI crawler roughly 3x more often than the sample's baseline blocking rate (51.9% of cited domains blocked something, against a 15% baseline).
- The most common real failure isn't a missing robots.txt rule. It's a CDN or WAF managed-bot-block silently overriding a permissive robots.txt, invisible unless you check server logs directly. This exact failure has hit rkarthik.in itself.
- User-agent strings can be spoofed. Verifying a crawler's identity means matching the source IP against the operator's published ranges, not trusting the UA string alone, a gap HasData's study found most sites don't check at all.
- Even a clean pass on all three layers doesn't guarantee visibility. Most AI crawlers (GPTBot, ClaudeBot, PerplexityBot) don't execute JavaScript at all, so JS-rendered content can be invisible to them regardless of what robots.txt, logs, and the CDN/WAF layer say.
- The full three-layer check takes under 15 minutes and needs no paid tool.
I've hit this exact failure on my own domain: a Cloudflare managed-rules bot-block was silently overriding a permissive robots.txt. The site looked completely normal to a human visitor. To GPTBot, ClaudeBot, and PerplexityBot, it was invisible, and nothing in my analytics said so directly. I had to go read server logs to find it.
That's Stage 1 of the five-stage sequence I run with clients: confirm the gate is actually open before spending anything on what's downstream of it. Below is the actual three-layer check: not just "read robots.txt," which turns out to answer a much narrower question than most guidance implies.
Why isn't "check robots.txt" actually checking access?
Robots.txt is a request, not a technical control. Crawlers that behave honestly read it and comply; crawlers that don't, ignore it; and a CDN or WAF sitting in front of your server can block or allow traffic regardless of what the file says, because that decision happens before the request ever reaches the application serving robots.txt.
HasData's July 2026 study (10,894 domains, a mix of the Tranco top-web list and news publishers) measured this directly and found 39.5% of GPTBot Disallow rules in robots.txt are not honored by GPTBot. The same study found something that complicates the simple "block them and you're safe" instinct even further: in a sample of 10 queries run against Google AI Mode, 52 cited domains matched the study's sample set, and 27 of those 52 (51.9%) blocked at least one AI crawler in their own robots.txt, against a 15% blocking rate in the sample as a whole. Sites that block crawlers got cited roughly 3x more often than the base rate, not less.
None of that means blocking is pointless or that compliance is universal: it means a robots.txt file, read on its own, tells you what you asked for, not what's actually happening in either direction.
What's the three-layer check?
1. Read robots.txt for your own domain. Specifically the named AI bot groups: OAI-SearchBot, ChatGPT-User, ClaudeBot, GPTBot, PerplexityBot, Claude-SearchBot, Claude-User, and Google-Extended are the ones worth checking by name, not just the wildcard block.
2. Grep your server access logs for real hits. This is the layer robots.txt can't show you. On a standard Apache or Nginx setup:
grep -i "GPTBot" access.log | wc -l grep -i "ClaudeBot" access.log | wc -l grep -i "PerplexityBot" access.log | wc -l
If those come back at zero for 30 days straight on a site that isn't brand new, that's a signal worth investigating: either something upstream is blocking them, or (less alarmingly) the specific bot you're checking simply hasn't prioritized your site yet. Distinguishing those two is exactly what layer 3 is for.
3. Check for a CDN/WAF-layer override. This is the layer a robots.txt read alone can never reveal, and it's the one that actually broke access on my own site. If you're on Cloudflare, check whether "Block AI Bots" or a managed bot-fighting rule is enabled at the zone level. These can apply regardless of what your robots.txt says, and won't show up anywhere in your application code. The practical test: curl -A "GPTBot" against your own domain and check the response code directly, rather than trusting the dashboard toggle state alone.
What blind spot does even a clean three-layer check miss?
Passing all three layers above (a permissive robots.txt, real 200s in the logs, no CDN/WAF override) still doesn't guarantee a crawler actually sees your content, because most AI crawlers don't execute JavaScript. Vercel's analysis of over 500 million GPTBot fetches found zero evidence of JS execution: GPTBot requests .js files as plain text roughly 11.5% of the time but never runs them, and independent checks from MERJ, Passionfruit, and Oncrawl confirm the same pattern for ClaudeBot and PerplexityBot. Google's own crawlers are the one notable exception: Googlebot and Gemini-related fetches do render JavaScript.
The practical implication: if the content that actually matters (pricing, product details, the specific answer someone's asking for) only appears after client-side JavaScript runs, a crawler can return a 200, log a real hit, and clear every CDN/WAF check, and still receive an effectively empty page. Check this the same direct way as the rest of this list: fetch your own key pages with JavaScript disabled, or curl the raw HTML directly, and confirm the content you actually care about is present in what comes back, not injected afterward by a script that never ran for the bot.
How do you verify it's actually the bot, not someone spoofing the UA?
A user-agent string is just a text header: anyone can set their request to claim to be GPTBot. HasData's study flagged this directly: most sites enforce AI-crawler access purely by matching the User-Agent string, and any actual crawler that omits or lies about its identity walks straight through that check. HUMAN Security's own research puts a number on how often that actually happens: roughly 1 in 18 requests carrying a known AI-crawler user-agent string is spoofed overall, and ChatGPT-User specifically (the worst rate of any AI-crawler identity HUMAN measured) is spoofed on roughly 1 in 6 requests.
The reliable fix is matching source IP against the operator's published ranges, not the UA string alone. OpenAI publishes machine-readable IP-range files per bot (at openai.com/gptbot.json, /searchbot.json, and /chatgpt-user.json). Anthropic publishes less in the way of a machine-readable feed for ClaudeBot specifically, which makes forward-confirmed reverse DNS (FCrDNS) the more dependable fallback: reverse-DNS the source IP to a hostname, confirm that hostname ends in the operator's domain, then forward-DNS that hostname and confirm it resolves back to the original IP. The forward step is what actually defeats spoofing: an attacker can fake a PTR record pointing at their own IP, but can't forge the operator's authoritative forward DNS record.
I go deeper on this specific verification step, the exact commands, and a reference table of every major AI/LLM crawler's user-agent string, in GPTBot, ClaudeBot, PerplexityBot: how to audit your access logs.
What does "no hits in the logs" actually mean?
Not all AI bots behave the same way, so a quiet log for one doesn't mean a quiet log for all of them:
| Bot category | Example | When it fires |
|---|---|---|
| Training crawler | GPTBot, ClaudeBot, Google-Extended | Periodic, broad crawling to build training datasets, infrequent on a small site |
| Search/retrieval crawler | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Crawls specifically to build a citable index for AI-generated answers |
| User-initiated fetch | ChatGPT-User, Claude-User, Perplexity-User | Fires only when a real person asks the assistant something that triggers a live fetch of your specific URL |
Zero hits from the user-initiated bots is normal until someone actually asks about you. Zero hits from the search/retrieval bots on an established, indexed site is the signal actually worth investigating: that's the category most directly tied to whether you can be cited at all. I cover the full log-analysis methodology (what to track over time, and how to build a repeatable report instead of a one-off check) in Server log analysis for AI visibility.
Frequently asked questions
How do I know if GPTBot or ClaudeBot can actually reach my site?
Check three layers, not one: robots.txt (what you're asking), your server access logs for real hits from that bot's user agent in the last 30 days (what's actually happening), and your CDN/WAF configuration (the layer that can silently override both without touching your robots.txt file at all).
Does robots.txt alone guarantee AI crawlers can access my site?
No. Robots.txt is a request, not a technical block, and the reverse is also true: an Allow rule doesn't guarantee access if something upstream (a CDN managed-bot-rule, a WAF policy) is intercepting the request before it ever reaches your server. A July 2026 study of 10,894 domains found 39.5% of GPTBot Disallow rules in robots.txt are ignored by GPTBot itself, which cuts both ways: the file is a weaker signal than most guidance assumes, in either direction.
Can a CDN or WAF block AI crawlers even if robots.txt allows them?
Yes, and this is the most common real failure mode: a managed bot-blocking rule at the CDN or WAF layer intercepts the request before your application (and its robots.txt) ever gets involved. This is invisible from reading robots.txt; the only way to catch it is checking server logs directly for whether the bot's requests are actually arriving and what status code they're getting.
Do AI crawlers respect blocks in robots.txt?
Inconsistently, and unevenly in both directions. Ahrefs' study of 137,210 domains found 97% of published llms.txt files get zero AI-crawler requests at all, while a separate HasData study found a meaningful share of robots.txt Disallow rules aimed at GPTBot specifically are not honored. The practical implication is the same either way: don't treat a text file as a technical guarantee in either direction; verify with logs.
Sources referenced
- HasData: The AI Crawler Block Index (Jul 2026, 10,894 domains)
- Ahrefs llms.txt server-log study (Jun 2026, 137,210 domains)
- Cloudflare: Your site, your rules — new AI traffic options
- Vercel: The rise of the AI crawler (500M+ GPTBot fetches analyzed)
- HUMAN Security: AI crawler spoofing research
For the full five-stage sequence this checks Stage 1 of, see the five stages for winning in the AI search era. For the deeper dives this piece points to, see auditing your access logs by user agent and server log analysis for AI visibility. For what to actually write in the robots.txt file itself and whether JavaScript rendering changes the check, see robots.txt for AI crawlers and JavaScript rendering and AI crawlers.
