- A user-agent string is a claim a request makes about itself, not proof. HUMAN Security found roughly 1 in 18 AI-crawler-UA requests are spoofed overall, and ChatGPT-User specifically, at roughly 1 in 6, has the worst rate of any AI-crawler identity measured.
- GPTBot, ClaudeBot, and PerplexityBot aren't one bot each. OpenAI alone runs at least three (GPTBot for training, OAI-SearchBot for citations, ChatGPT-User for live user-initiated fetches), each independently controllable in robots.txt.
- The reference table isn't just OpenAI/Anthropic/Perplexity. Common Crawl's CCBot feeds the shared dataset behind GPT-2/3/4-era models, LLaMA, and Mistral, and ByteDance's Bytespider has a documented history of ignoring robots.txt.
- The reliable verification method is matching source IP against the operator's published ranges, with forward-confirmed reverse DNS (FCrDNS) as the fallback, and cryptographic request-signing (Web Bot Auth, built on RFC 9421) as a newer, more direct option where a crawler supports it.
- Prioritize the search/retrieval bots in any audit. They're most directly tied to whether you can be cited, unlike training or user-fetch bots where a quiet log isn't automatically a red flag.
- A working log-audit command takes one line of grep; verifying the hit is real takes a few more.
Checking whether "GPTBot" hit your server tells you what a request claimed to be. It doesn't tell you the request was actually from OpenAI. That gap (matching a UA string instead of verifying an identity) is the specific weakness this piece covers, building on the three-layer check in how to check AI crawler access.
What's actually crawling you?
Major AI labs don't run one crawler each. They run several, split by purpose, and conflating them is a common mistake that leads to blocking (or allowing) the wrong thing:
| Operator | Training crawler | Search/retrieval crawler | User-initiated fetch |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | — | PerplexityBot | Perplexity-User |
| Google-Extended | (Googlebot, unified) | — | |
| Meta | Meta-ExternalAgent | — | Meta-ExternalFetcher |
| Apple | Applebot-Extended | — | — |
| Amazon | Amazonbot | — | — |
| Common Crawl Foundation | CCBot | — | — |
| ByteDance | Bytespider | — | — |
CCBot and Bytespider are worth naming explicitly, not just leaving to the broader reference list below. CCBot (run by the nonprofit Common Crawl Foundation, and respectful of robots.txt) feeds the shared, publicly available dataset that trained GPT-2, GPT-3, GPT-4-era models, LLaMA, Mistral, and others; because so many labs train on Common Crawl's data rather than running their own training crawler, CCBot's reach is arguably broader than any single lab's own bot. Bytespider, ByteDance's training crawler, has a documented history of not respecting robots.txt directives, worth flagging specifically in a piece about verifying what's actually crawling you versus what a site is asking to happen.
Blocking GPTBot does not block OAI-SearchBot. They're controlled independently in robots.txt, and a site that disallows GPTBot for training-data reasons while intending to stay citable in ChatGPT answers needs to leave OAI-SearchBot and ChatGPT-User untouched. Getting this distinction wrong in either direction is one of the more consequential mistakes in this category, because the two outcomes (opting out of training data vs. opting out of citations) are usually not the same decision.
This is a partial list of the crawlers worth naming explicitly; a comprehensive, actively-maintained reference covering well over a hundred AI-related user agents exists at the community-maintained ai.robots.txt project, worth checking directly rather than relying on any single static list, since new AI products (and their crawlers) launch continually.
Why isn't matching the UA string alone enough?
HasData's July 2026 study of 10,894 domains examined how sites actually enforce AI-crawler access rules, and found the dominant method is matching the literal User-Agent header string against a known list. The problem: nothing stops a request from lying. Any crawler (a bad actor, a scraper, a competitor's tool) can set its User-Agent header to say "GPTBot" and most enforcement checks will treat it as genuine, because the check never looks past the string itself. HUMAN Security's own research quantifies how often this happens in practice: roughly 1 in 18 requests carrying a known AI-crawler UA string is spoofed overall, and ChatGPT-User specifically (the worst rate of any AI-crawler identity HUMAN measured) is spoofed on roughly 1 in 6 requests.
This matters in both directions. If you're trying to allow only real AI crawlers, string-matching alone lets an impersonator in. If you're trying to block them, string-matching alone can be defeated by simply not sending a recognizable UA string at all, or by omitting it entirely, which several sources describe as a known evasion pattern for crawlers that don't want to identify themselves.
How do you verify identity properly?
Where the operator publishes IP ranges, use them. OpenAI publishes machine-readable IP-range files per bot, at openai.com/gptbot.json, openai.com/searchbot.json, and openai.com/chatgpt-user.json, so a request claiming to be GPTBot can be checked against OpenAI's own published range directly.
Where a machine-readable range isn't published, use forward-confirmed reverse DNS (FCrDNS). Anthropic, for instance, publishes less in the way of a structured IP feed for ClaudeBot specifically, which makes FCrDNS the more dependable check:
- Reverse-DNS the source IP to get a hostname.
- Confirm the hostname actually ends in the operator's real domain (e.g.,
.anthropic.com). - Forward-DNS that same hostname and confirm it resolves back to the original source IP.
The forward step in that sequence is what actually defeats spoofing. An attacker can set a fake PTR record pointing their own IP at a domain-looking hostname, but they can't forge the operator's authoritative forward DNS record to make that hostname resolve back to their IP. If steps 2 and 3 both hold, the identity is real; if either fails, treat the request as unverified regardless of what its UA string claims.
A newer option: cryptographic verification. Cloudflare, in partnership with the IETF (built on RFC 9421, HTTP Message Signatures), has been rolling out "Web Bot Auth": a bot signs each request with an Ed25519 key rather than relying on an IP range or reverse DNS to prove its identity. OpenAI has adopted it for Operator, and AWS WAF, Vercel, Shopify, and Akamai have added support on the verification side. It doesn't replace IP-range checks or FCrDNS as the fallback for bots that don't support it yet, but where a crawler does sign its requests, a valid signature is a more direct proof of identity than either.
What's a practical starting audit?
grep -i "GPTBot" access.log | wc -l grep -i "ClaudeBot" access.log | wc -l grep -i "OAI-SearchBot" access.log | wc -l grep -i "PerplexityBot" access.log | wc -l
Run that across 30 days of logs first. It's the fast pass. For any bot showing high-volume traffic that you intend to treat as trusted (feeding it different content, whitelisting it past a WAF rule, or anything else that assumes the identity is real), that's the traffic worth running through the IP-range or FCrDNS check above before trusting it. Low-volume, low-stakes traffic generally doesn't need the same scrutiny. Prioritize by what you'd actually do differently if the identity turned out to be false.
For the broader methodology (what to track over time, response codes, building this into a repeatable report instead of a one-off check), see server log analysis for AI visibility.
Frequently asked questions
Is checking the user-agent string enough to confirm a crawler is legitimate?
No. A July 2026 study (HasData, 10,894 domains) found most sites enforce AI-crawler access purely by matching the User-Agent header string, and any actual bad actor that copies a known AI crawler's UA string walks straight through that check. Matching the string tells you what a request claims to be, not what it is.
What's the difference between GPTBot and OAI-SearchBot?
They're different bots serving different purposes, even though both come from OpenAI. GPTBot collects content for training future models; OAI-SearchBot and ChatGPT-User are the ones tied to citations you'd actually see in a ChatGPT answer. Blocking one doesn't block the other, since they're controlled independently in robots.txt, and conflating them is a common, consequential mistake.
How do I verify a crawler is actually who it claims to be?
Match the source IP against the operator's published ranges where available (OpenAI publishes machine-readable IP-range files per bot), and fall back to forward-confirmed reverse DNS (FCrDNS) where it isn't. Reverse-DNS the IP to a hostname, confirm the hostname ends in the operator's real domain, then forward-DNS that hostname and confirm it resolves back to the original IP. The forward step is what defeats a spoofed PTR record.
Do I need to check every single AI crawler individually?
No. Prioritize by category. The search/retrieval bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are most directly tied to whether you can be cited at all, so verify those first. Training crawlers (GPTBot, ClaudeBot, Google-Extended) and user-initiated fetchers (ChatGPT-User, Claude-User, Perplexity-User) matter, but a quiet log from those isn't automatically a problem the way a quiet log from the search bots is.
Sources referenced
- HasData: The AI Crawler Block Index (Jul 2026, 10,894 domains)
- ai.robots.txt project (community-maintained AI crawler user-agent reference)
- HUMAN Security: AI crawler spoofing research
- Cloudflare: Web Bot Auth
This piece extends how to check AI crawler access. Start there for the full three-layer check. For turning this into a repeatable process rather than a one-off audit, see server log analysis for AI visibility. For whether a slow-responding site actually loses crawl volume to this traffic, see page speed and AI crawl budget.
