- HasData's July 2026 study of 10,894 domains found 39.5% of robots.txt Disallow rules aimed specifically at GPTBot were not honored by the crawler, meaning a documented minority of intended blocks don't actually work as written.
- The same study found a surprising, counterintuitive correlation: sites blocking at least one AI crawler got cited in Google AI Mode roughly 3x more often than the sample's baseline blocking rate (51.9% of cited domains blocked something, against a 15% baseline).
- Neither finding means robots.txt is useless. The majority of Disallow rules in the study (60.5%) were respected, and the citation correlation is one sample's finding, not a demonstrated causal mechanism that blocking somehow improves citation odds.
- Compliance appears to vary by crawler and by whether a request originates from a batch-training crawl versus a live, on-demand fetch during an actual user query, a distinction worth more research than currently exists in public studies.
- The only reliable way to confirm a block is actually working on your own site is checking server access logs for real hits and their response codes, not trusting the stated rule, the same principle already covered in this site's three-layer access check.
Treating robots.txt as a binding technical control, in either direction, is the single most common mistake in AI-crawler guidance. One disclosed study gets specific about exactly how often that assumption breaks down.
How often is a GPTBot Disallow rule actually ignored?
More often than most guidance implies. HasData's July 2026 study, covering 10,894 domains drawn from a mix of the Tranco top-web list and news publishers, measured real crawler behavior against stated robots.txt rules and found 39.5% of Disallow rules specifically targeting GPTBot were not honored by the crawler itself. That's a meaningful minority, not a rounding error: roughly two out of every five intended GPTBot blocks in the study's sample didn't actually hold in practice.
It's worth being precise about what that figure doesn't say. It doesn't mean GPTBot ignores robots.txt as a rule; the majority of Disallow rules in the same study, 60.5%, were respected. It means a documented, substantial minority weren't, which is a different and more nuanced finding than either "AI crawlers always respect robots.txt" or "AI crawlers never respect robots.txt," both of which overstate the case in opposite directions.
Does blocking crawlers actually reduce AI citation, as intended?
This is where the study's findings get genuinely counterintuitive. In a smaller sub-sample (10 queries run against Google AI Mode, yielding 52 cited domains that matched the study's broader sample set), 27 of those 52 cited domains, 51.9%, blocked at least one AI crawler in their own robots.txt, against a 15% blocking rate across the sample as a whole. Sites that blocked something got cited roughly 3x more often than the baseline blocking rate would predict if blocking and citation were unrelated.
That's a real, measured correlation in this specific sample, but it's worth treating as exactly that, a correlation, not a demonstrated mechanism. A plausible, unconfirmed explanation: sites sophisticated enough to deliberately configure crawler-specific robots.txt rules in the first place may also be doing other things well (structured content, strong entity signals, genuine authority) that independently drive citation, making the blocking itself a marker of a well-run site rather than a cause of citation, the same pattern already established for schema markup's correlation with citation. Nobody has published a controlled study isolating whether blocking itself causes a citation lift, and it would be a strange mechanism if it did.
Does compliance vary by crawler, or by type of request?
The available data doesn't fully resolve this, and it's a real limitation worth stating plainly. The 39.5% non-compliance figure is specific to GPTBot; other crawlers may behave differently, and the study referenced here doesn't provide an equally detailed breakdown across every major AI crawler by name. There's also a plausible but unconfirmed distinction between a batch training crawl (which might be more likely to respect a blanket Disallow) and a live, on-demand fetch triggered by an actual user's query (where the crawler's operator might have a stronger incentive to fetch the page regardless, since a real user is waiting on an answer). No study reviewed in researching this piece directly tests that specific distinction.
What should you actually do with this information?
Don't treat a robots.txt rule, in either direction, as a guaranteed technical outcome. If a block matters to you, verify it's actually working by checking server access logs for the crawler's real requests and the response codes they're actually getting, exactly the verification already laid out in how to check AI crawler access. And don't assume blocking crawlers is a reliable way to either protect your content or improve your citation odds; the one disclosed correlation pointing toward the latter is a single sample's finding, not a demonstrated strategy.
Frequently asked questions
Does GPTBot actually respect a Disallow rule in robots.txt?
Not consistently. HasData's July 2026 study of 10,894 domains (a mix drawn from the Tranco top-web list and news publishers) found 39.5% of robots.txt Disallow rules specifically aimed at GPTBot were not honored by the crawler.
Does that mean blocking AI crawlers in robots.txt is pointless?
No, but it means the file alone isn't a guaranteed technical control. A meaningful majority of Disallow rules in the study were respected (60.5%), so robots.txt still works most of the time for compliant crawler behavior. It just isn't the airtight guarantee a lot of guidance implies, especially for GPTBot specifically, which showed the highest non-compliance rate measured.
Does blocking AI crawlers reduce your chances of being cited?
Counterintuitively, the same HasData study found the opposite correlation in its sample: domains that blocked at least one AI crawler in robots.txt got cited in Google AI Mode roughly 3x more often than the sample's overall baseline blocking rate (51.9% of cited domains in the sample blocked something, against a 15% baseline). That's a correlation in one specific sample, not a confirmed causal mechanism.
How can I verify whether a block is actually working on my own site?
Check server access logs directly for the specific bot's user-agent string and confirm what response code it's actually receiving, rather than trusting the robots.txt rule's stated intent. This is the same verification principle covered in this site's three-layer crawler-access check.
Sources referenced
- HasData: GPTBot robots.txt compliance study (July 2026, 10,894 domains)
For the full three-layer verification process, see how to check AI crawler access. For why a correlation like the citation-rate finding here shouldn't be read as causation, see does schema markup help AI citations, which found the same correlation-versus-causation gap in a different context.
