- No standard or major AI lab requires a separate, dedicated sitemap for AI crawlers. The existing sitemap.xml format already does the discovery job for any AI crawler that uses sitemaps at all.
- llms.txt, the genuinely new file in this space, is explicitly documented as complementing sitemap.xml and robots.txt, not replacing either; sample implementations often link directly back to the sitemap from within the llms.txt file itself.
- llms.txt's real-world impact remains unconfirmed. Google's John Mueller has reportedly stated no AI crawler has officially confirmed extracting information via it, and this site's own existing coverage already cites Ahrefs' finding that 97% of published llms.txt files received zero AI-crawler requests across 137,210 domains.
- Access control for different AI crawlers (training versus citation versus on-demand fetch bots) is a robots.txt job, not a sitemap job; a sitemap only aids discovery of URLs that already exist, it doesn't grant or restrict any crawler's access to them.
- The practical file set for AI discoverability is the same three files most sites already maintain or should: a standard sitemap.xml, a deliberate robots.txt, and optionally an llms.txt, rather than a fourth, AI-specific format.
"Do I need a separate sitemap for AI search?" has a short, confident answer once you separate what a sitemap actually does from what people assume a new AI-specific file might do instead.
What does a sitemap actually do, mechanically?
A sitemap.xml lists a site's canonical URLs in a machine-readable format, primarily to aid discovery: helping a crawler find pages it might otherwise miss, especially ones with few internal links pointing to them, and signaling which version of a URL is canonical when duplicates exist. That's the entire job. It says nothing about access permissions (that's robots.txt's job) and nothing about content quality, extractability, or citation worthiness (that's a function of the content itself). Any crawler, AI-affiliated or not, that reads sitemaps at all reads the same sitemap.xml format everyone else does; there's no separate, AI-specific sitemap schema that any major lab has published or required.
Then what is llms.txt, if it's not a replacement sitemap?
This is the genuinely new file in this space, and it's worth being precise about what it's actually designed to do. llms.txt functions as a curated pointer file: a short, prioritized list of a site's most important resources, meant to guide an AI system toward what the site itself considers its highest-value content, rather than an exhaustive listing of every URL the way a sitemap is. Its own documentation and the sample implementations reviewed in researching this piece are explicit that it doesn't replace robots.txt or sitemap.xml; it's designed to complement both, and a well-formed llms.txt file commonly includes a direct link back to the site's actual sitemap, treating the two as working together rather than as competitors.
Does llms.txt actually work, though?
That's a genuinely open question, and this site already has a dedicated piece on it worth reading in full: does llms.txt actually work?. The short version: Google's own Search Liaison, John Mueller, has reportedly stated that no AI crawler has officially confirmed extracting information through llms.txt, and Ahrefs' own server-log study, across 137,210 domains, found 97% of published llms.txt files received zero AI-crawler requests in a month. Having the file isn't harmful, but treating it as a confirmed, working mechanism for AI discoverability overstates the current evidence considerably.
Does access control require a separate AI sitemap?
No, and this is worth separating clearly from the discovery question. If you want to allow or restrict specific AI crawlers differently from each other (training crawlers versus citation/search crawlers versus on-demand fetch bots), that's a robots.txt job, covered in detail in robots.txt for AI crawlers: what to actually allow. A sitemap only helps a crawler discover URLs that already exist; it has no mechanism for granting or restricting any crawler's permission to actually fetch them. Conflating "listing my URLs for AI discovery" with "controlling which AI bots can access them" is a common source of confusion, and the two questions have two entirely different, already-standard answers.
So what should you actually maintain?
The same small set of files most well-run sites already have reason to maintain regardless of AI search specifically: a standard, accurate sitemap.xml for discovery, a deliberate, per-crawler-group robots.txt for access control, and optionally an llms.txt pointing to your most important pages, understanding clearly that its practical impact on AI citation remains unconfirmed by the evidence available as of this writing. No fourth, AI-specific sitemap format is a recognized requirement layered on top of those three.
Frequently asked questions
Do I need a separate sitemap specifically for AI crawlers?
No. There's no recognized standard or requirement for a dedicated AI-only sitemap format. Your existing sitemap.xml, listing canonical URLs for discovery, is the same file AI crawlers that use sitemaps at all would use, identical to how search engines use it.
What does llms.txt actually do, then, if not replace the sitemap?
llms.txt is meant to guide AI systems toward a site's preferred, most important resources, functioning as a curated pointer file rather than a full URL listing. Its own documentation and sample implementations explicitly state it doesn't replace robots.txt or sitemap.xml; it's designed to complement both, often including a direct link back to the sitemap within it.
Has any AI lab confirmed it actually uses llms.txt?
No, as of the most recent check in researching this piece. Google's own John Mueller has reportedly stated that no AI crawler has officially confirmed it extracts information via llms.txt. Separately, Ahrefs' own study found 97% of published llms.txt files received zero AI-crawler requests across a 137,210-domain sample, consistent with uncertain real-world uptake.
So what should you actually maintain for AI discoverability?
A standard, well-maintained sitemap.xml for canonical URL discovery, deliberate robots.txt rules per AI crawler group for access control, and optionally an llms.txt pointing to your most important pages, understanding that its practical impact is unconfirmed. No fourth, AI-specific sitemap file is a recognized requirement on top of those three.
Sources referenced
- Ahrefs: llms.txt server-log study (Jun 2026, 137,210 domains)
- texta.ai: llms.txt optimization
For the fuller treatment of whether llms.txt actually works, see does llms.txt actually work?. For access-control specifics by crawler, see robots.txt for AI crawlers: what to actually allow.
