- Ahrefs' June 2026 study of 137,210 domains found 97% of published llms.txt files received zero AI-crawler requests in a full month, and where retrieval bots did request the file, they were just 1.1% of total requests to it. SEO audit tools checking whether the file existed were the largest single requester category.
- SE Ranking's separate ~300,000-domain study found no statistically significant correlation between having an llms.txt file and AI citation frequency, and only 10.13% of domains had the file at all, with 0% adoption among the top 1,000 sites by traffic.
- Google made this official in writing: a June 2026 update to Google Search Central's documentation states plainly that Google Search ignores llms.txt files. They neither help nor hurt visibility or rankings.
- The thin counter-evidence that exists (a small field study finding a handful of real OpenAI/Anthropic crawler hits) doesn't change the conclusion. It's a non-zero trickle against thousands of robots.txt requests over the same window, not meaningful engagement.
- llms.txt's genuine, better-supported use case isn't AI search citation at all. It's agent and IDE tooling reading developer documentation, a different problem the file was originally designed for.
This site has flagged the llms.txt finding twice already, in passing, as part of larger pieces. The five stages for winning in the AI search era calls it one of the two most oversold tactics in the category, and how to check AI crawler access cites the same study in a footnote. This is the full, standalone version: what the actual data says, what's changed since those studies were published, and the one place llms.txt does seem to earn its keep.
What two studies does this rest on?
Ahrefs, June 2026. 137,210 domains with server-log access, spanning a mix of the Tranco top-web list and news publishers. In the full month of May 2026, 97% of published llms.txt files received zero requests from any crawler. Where AI retrieval bots (the ones that actually generate citations, as distinct from training crawlers or coding agents) did request anything, they accounted for just 1.1% of total requests to the file. The single largest category of requester wasn't an AI system at all: it was SEO audit tools checking whether the file existed, not reading it for content.
SE Ranking, roughly 300,000 domains. No statistically significant correlation between having an llms.txt file and AI citation frequency, and removing the variable from their predictive model actually improved its accuracy, meaning the file wasn't just neutral, it was adding noise. Two additional details worth knowing: only 10.13% of the 300,000 domains had an llms.txt file at all, and adoption was 0% among the top 1,000 domains by traffic. Among the 50 most AI-cited domains in their sample, only one had the file.
Two different organizations, two different sample sizes, two different methodologies, the same conclusion: the file exists on a small minority of sites, almost nothing reads it, and having it shows no measurable relationship to being cited.
What changed since those studies?
The most significant update since the June 2026 Ahrefs study isn't a new dataset. It's Google closing the door explicitly. Google Search Central's documentation was updated on June 15, 2026 with a new section, "Clarifying guidance on llms.txt files," refined again on June 29. The reported language is direct: it's fine to create and maintain an llms.txt file for other services or systems that use it, but doing so "won't harm (nor help) your visibility or rankings in Google Search, as Google Search ignores them." That's a meaningfully stronger statement than the informal social-media comments Google staff had made earlier. This is a dated, written documentation change, not a forum reply.
No equivalent statement exists from OpenAI, Anthropic, or Perplexity about whether their retrieval crawlers act on the file. Anthropic does publish its own llms.txt and llms-full.txt for its developer documentation, but that's Anthropic acting as a publisher of the file for its own docs, not a statement about whether Claude's crawlers consume the file on other people's sites. The two are frequently conflated and shouldn't be.
Is there any counter-evidence?
The picture isn't a perfectly clean zero. A small independent field study placed llms.txt on 83 websites and tracked crawler behavior over several months in 2026, finding OpenAI's crawler fetched the file 7 times against roughly 3,990 robots.txt fetches over the same period, and Anthropic's fetched it 9 times against roughly 3,120 robots.txt fetches. PerplexityBot fetched it zero times.
That's real, non-zero engagement, not nothing, and not evidence that the file is being ignored outright by every system. But it's a trickle: a handful of hits against thousands of robots.txt requests from the same bots over the same window. It's consistent with Ahrefs' 1.1%-of-requests figure, not a contradiction of it. A separate 90-day before/after tracking study across 10 sites in different industries found 2 sites saw AI-referral traffic increases after adding the file (+12.5% and +25%), 8 saw no change, and 1 declined, and the study's own authors attributed even the two increases to other concurrent changes on those sites, not the file itself.
Which use case actually holds up?
The place llms.txt shows real, if narrower, utility isn't AI search citation. It's AI coding agents and IDE tools ingesting developer documentation quickly, which is closer to the file's original pitch than "getting cited in ChatGPT." That's a genuinely different problem from the one most llms.txt advice is sold against, and worth separating clearly: if you're optimizing documentation for an AI coding assistant to parse quickly, llms.txt has a real, if unmeasured-at-scale, rationale. If you're adding it hoping it improves your odds of being cited in a generated search answer, the two largest available studies say it doesn't, and Google has now said the same thing in writing for its own surfaces.
Where did the file actually come from?
Jeremy Howard (co-founder of Answer.AI and fast.ai) proposed llms.txt in September 2024, publishing the idea on Answer.AI's own site alongside a dedicated llmstxt.org. The original pitch: LLM context windows are too small to ingest a full website, and converting raw HTML (navigation, ads, script noise) into clean text a model can actually use is slow and error-prone, especially for developer documentation where fast, structured access matters most. The proposal was a plain-Markdown index file at the domain root, plus an optional full-text companion file.
Adoption stayed niche for a couple of months until Mintlify added automatic llms.txt support across every documentation site it hosts, around November 2024, after which thousands of docs sites, including Anthropic's own and Cursor's, picked up the file in a short window. That adoption story is itself a clue to what the file is actually good for: it spread fastest through developer-documentation tooling, not through general marketing or content sites hoping for AI-search citation.
Frequently asked questions
Does llms.txt actually help with AI search visibility?
On the current evidence, no. Ahrefs found 97% of published llms.txt files get zero AI-crawler requests in a full month, and SE Ranking's separate 300,000-domain study found no statistically significant correlation between having one and AI citation frequency. Removing the variable from their predictive model actually improved its accuracy. Google has now stated in its own documentation that Search ignores the file entirely.
Has any AI lab said whether their crawlers read llms.txt?
Google is the only one to make a clear, written statement, and it's a no: a June 2026 update to Google Search Central's documentation states plainly that Google Search ignores llms.txt files. No equivalent public statement exists from OpenAI, Anthropic, or Perplexity about whether their retrieval systems act on the file, though a small field study found non-zero but vanishingly small crawler engagement from OpenAI's and Anthropic's bots specifically.
Is there any real evidence llms.txt does something?
A small amount, and it's thin. One independent field study found OpenAI's and Anthropic's crawlers did fetch llms.txt a handful of times over several months, a real, non-zero signal, but a tiny fraction of the thousands of robots.txt requests the same crawlers made over the same period. The more credible use case for llms.txt isn't AI search citation at all. It's helping coding agents and IDE tools ingest developer documentation, a genuinely different problem from being cited in a generated answer.
Where did llms.txt come from?
Jeremy Howard (co-founder of Answer.AI and fast.ai) proposed it in September 2024, pitched as a plain-Markdown index to help LLMs work around small context windows and the noise (navigation, ads, scripts) in raw HTML. Adoption stayed niche until Mintlify added automatic llms.txt support across the documentation sites it hosts around November 2024, after which thousands of docs sites picked it up quickly.
Sources referenced
- Ahrefs llms.txt server-log study (Jun 2026, 137,210 domains)
- SE Ranking llms.txt citation-correlation study (2026, ~300,000 domains)
- llmstxt.org (original proposal, Jeremy Howard, Sep 2024)
This piece is the full standalone version of the llms.txt point already made in the five stages for winning in the AI search era and how to check AI crawler access.
