- "AI crawler" isn't one thing. OpenAI alone runs at least three separately-named crawlers with different jobs: GPTBot (training data), OAI-SearchBot (its ChatGPT Search index), and ChatGPT-User (on-demand fetches during a live chat session). A single wildcard rule can't tell them apart.
- Content-Signal is an emerging robots.txt extension that separates "may this be used for search," "may this be used as AI input/retrieval," and "may this be used for AI training" into three distinct signals, rather than one blanket allow or block.
- An Allow rule in robots.txt is a stated request, not a technical guarantee, in either direction: a non-compliant crawler can ignore a Disallow, and a CDN/WAF rule can silently override an Allow before the request ever reaches the file.
- This site's own robots.txt is a working example of a deliberate policy: named groups for AI answer-engine/citation crawlers, a separate named group for major training crawlers, and a long tail of smaller bots pulled from a maintained community list (github.com/ai-robots-txt/ai.robots.txt) rather than tracked by hand.
- Writing the file correctly is necessary but not sufficient. How to check AI crawler access covers the server-log and CDN/WAF verification that confirms the stated policy actually matches what's happening.
Most robots.txt guidance stops at "add Allow: / for AI bots," which treats a genuinely varied category of crawlers as one undifferentiated thing. Here's what a more deliberate file actually looks like, using this site's own as a worked example.
Why doesn't one wildcard rule cover "AI crawlers"?
Because "AI crawler" bundles together bots doing meaningfully different jobs. OpenAI alone operates at least three distinctly-named crawlers: GPTBot, which gathers data for model training; OAI-SearchBot, which builds the index ChatGPT Search draws on; and ChatGPT-User, which fetches a specific page on-demand when a live conversation needs it. A site might reasonably want to allow on-demand fetching and search indexing while declining to contribute training data, a distinction a single wildcard Allow or Disallow simply can't express. The same pattern holds across labs: Anthropic separates ClaudeBot (training) from Claude-SearchBot and Claude-User (citation and on-demand fetch), and Perplexity separates PerplexityBot from Perplexity-User.
What does a deliberate, named-group robots.txt actually look like?
This site's own robots.txt is structured around that distinction directly. A wildcard rule covers anything unnamed, but named groups above it handle the bots worth treating deliberately: one group for the answer-engine and citation crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User), a separate group for the major AI-lab training crawlers (GPTBot, ClaudeBot, Google-Extended, Meta-ExternalAgent, and others), and a long tail of smaller, less consequential bots pulled in from a community-maintained reference list (github.com/ai-robots-txt/ai.robots.txt) rather than tracked by hand one at a time, since that list moves fast and re-syncing periodically is more sustainable than manual upkeep.
What is the Content-Signal directive, and why does it matter?
Content-Signal is an emerging extension to the robots.txt format that goes beyond a binary allow/disallow. Instead of one yes-or-no decision per bot, it lets a site state three separate signals: whether content may be used for search, whether it may be used as AI input (meaning retrieval and citation in an AI-generated answer), and whether it may be used for AI training specifically. That's a meaningfully more granular statement of intent than Allow/Disallow alone, since a site might want to permit citation-driven retrieval while declining to contribute to model training, a distinction the older binary format has no way to express at all. It isn't yet a universally adopted standard the way robots.txt itself is, so treat it as an additional, forward-looking signal rather than something every crawler is guaranteed to read and respect.
Does writing the rule actually guarantee the outcome?
No, and this is worth repeating because it's the single most common misunderstanding in this space. Robots.txt is a stated request: a crawler built to behave honestly reads it and complies, one that isn't built that way ignores it, and this cuts both directions, not just toward over-permissiveness. A Disallow doesn't guarantee a bot stays away, and an Allow doesn't guarantee a bot actually gets through, because something upstream (a CDN managed-bot rule, a WAF policy) can override either outcome before the request ever reaches the file at all. How to check AI crawler access covers the full three-layer verification (the file itself, server logs, and the CDN/WAF layer) that confirms a written policy actually matches reality, and Cloudflare's AI bot blocking covers the specific, real failure mode of a permissive file getting silently overridden at the edge.
What's the practical starting point?
Decide, deliberately, what you actually want for each of the three categories (answer-engine/citation crawlers, training crawlers, and everything else), rather than defaulting to either a blanket allow or a blanket block without thinking about the distinction. Name the major bots in each category explicitly if you want different treatment between them, pull in a maintained community list for the long tail rather than tracking every new crawler by hand, and consider adding Content-Signal directives if the finer-grained search/input/training distinction matters to your situation. Then verify the stated policy actually holds, since writing the file correctly is the easy half of this problem.
Frequently asked questions
Is a wildcard "Allow: /" enough to let AI crawlers in?
It covers crawlers that don't match a more specific named rule, but it treats every AI bot identically, which may not be what you want. OpenAI alone runs at least three separately-named crawlers with different jobs (GPTBot for training, OAI-SearchBot for its search index, ChatGPT-User for on-demand fetches during a live chat), and a wildcard rule can't distinguish between them if you ever want to.
What's the Content-Signal directive, and is it standard yet?
An emerging robots.txt extension that lets a site declare, per bot group, whether its content may be used for search, AI input (retrieval/citation), and AI training, as three separate yes/no signals rather than one blanket allow/disallow. It's not yet a universally adopted standard the way robots.txt itself is, but it gives a more granular statement of intent than Allow/Disallow alone.
Does an Allow rule in robots.txt guarantee a crawler actually visits?
No, in either direction. It's a stated request, not a technical guarantee. A compliant crawler reads it and follows it; a non-compliant one ignores it; and a CDN or WAF rule sitting in front of your server can override it either way regardless of what the file says. Verifying actual access requires checking server logs directly, not just reading the file.
Should every AI crawler be named individually in robots.txt?
Naming the major, well-known bots by group (search/citation crawlers like OAI-SearchBot and PerplexityBot; training crawlers like GPTBot and ClaudeBot) is worth doing if you want different rules for different purposes. For the long tail of smaller, less consequential crawlers, a project like github.com/ai-robots-txt/ai.robots.txt maintains a synced list that can be pulled in wholesale rather than tracked by hand one at a time.
Sources referenced
For the full three-layer verification that confirms a written policy actually matches reality, see how to check AI crawler access. For the specific failure mode of a permissive file getting silently overridden upstream, see Cloudflare's AI bot blocking: what it silently breaks.
