- llms.txt is a plain-Markdown file at a site's root, proposed by Jeremy Howard (Answer.AI, fast.ai) in September 2024, an index of what a site is and which pages matter, written for a language model to read directly.
- The spec (llmstxt.org) requires exactly two things: an H1 with the project/site name, and a blockquote directly beneath it with a short summary. Everything else (free-form context, H2-grouped link sections) is optional structure on top of that minimum.
- llms-full.txt is a separate, optional companion file: the full text of a site's documentation in one file, rather than an index of links to follow.
- It's not a permissions file (that's robots.txt) and it's not a structured URL list for search indexing (that's sitemap.xml). It's closer to a curated, prose summary written for a model's context window than to either.
- This piece covers the format only. For whether publishing one actually changes anything about AI citation (the adoption and effectiveness data), see does llms.txt actually get read by anyone?
Most llms.txt coverage jumps straight to the argument (does it work, is it worth doing) without ever pinning down what the file actually is, structurally. Does llms.txt actually get read by anyone? already covers that argument in full, including where the file came from and the adoption data. This piece is the other half: the actual format, section by section, so you know exactly what you'd be publishing before deciding whether to bother.
What are the two required elements?
The specification, maintained at llmstxt.org, is deliberately minimal. Exactly two things are structurally required:
- An H1 heading with the name of the project or site. This is the only heading the spec treats as mandatory.
- A blockquote directly beneath the H1 containing a short, one-paragraph summary of what the site or project is.
That's the entire required structure: a name and a one-line description, in Markdown, at a predictable URL. Everything past that point is optional, layered structure.
What optional sections does the spec allow?
After the required H1 and blockquote, a fuller llms.txt file can include, in this order:
- Free-form Markdown context. Additional paragraphs or lists giving more detail about the project, but no headings are allowed in this section specifically, since a heading here would be ambiguous with the structured link sections that follow.
- H2-headed sections grouping links. Each H2 acts as a category label (commonly something like "Docs," "Guides," "API Reference," or an "Optional" section for lower-priority material), and under each H2 sits a Markdown-formatted list of links: a link text, a URL, and often a short inline description of what that page covers.
The format choice itself is deliberate: Markdown rather than a machine-only schema like XML or JSON, specifically because the target reader is a language model working inside a limited context window, not a parser executing a rigid schema. A model can read Markdown prose the same way it reads any other text; an XML sitemap is structured for programmatic parsing, not for being read and reasoned about directly.
What is llms-full.txt, and how is it different?
Some sites that publish llms.txt also publish a second file, llms-full.txt, at the same root level. Where llms.txt is an index (a summary plus a curated set of links a model or agent would need to follow to get the actual content), llms-full.txt is the destination: the full text of a site's documentation, concatenated into a single file.
The two solve different problems. llms.txt assumes a system that can follow links and fetch pages as needed, and optimizes for a small, fast-to-parse index. llms-full.txt assumes a system that would rather ingest everything in one pass and skip the link-following step entirely, useful for a smaller documentation set, less practical once a site's docs grow large enough that the full-text file itself becomes unwieldy to hand to a model in one shot.
How does this differ from robots.txt and sitemap.xml?
All three files live at a predictable root-level location, which is part of why they get casually lumped together. They do genuinely different jobs:
| File | What it actually does | Format |
|---|---|---|
robots.txt | A permissions file: tells crawlers what they're allowed or disallowed to access. Says nothing about what the content is. | Plain text, a fixed directive syntax (User-agent, Disallow, Allow) |
sitemap.xml | A structured, machine-only list of a site's URLs, built for search-engine indexing coverage. | XML, following a fixed schema |
llms.txt | A curated, prose summary of what a site is and which pages matter most, written to be read directly by a language model. | Markdown: headings, a blockquote, and link lists in natural prose |
robots.txt answers "can you access this." sitemap.xml answers "here is a complete, structured list of every URL." llms.txt answers a different question entirely: "here's what this is, and here's what actually matters, in a form you can read like text." None of the three substitutes for either of the others.
Where should you actually use this?
If you're building or updating documentation for a site that wants to publish one, the format above is the whole spec: an H1, a blockquote summary, optional free-form context with no headings, then H2-grouped link sections. That's a small, cheap thing to get technically right. Whether getting it right changes anything measurable about AI search citation is a separate question. On the evidence currently available, the honest answer is largely no; see does llms.txt actually get read by anyone? for the adoption numbers, Google's written statement that Search ignores the file, and the one use case (AI coding agents and IDE tooling reading documentation) where it does seem to earn its keep.
Frequently asked questions
What is llms.txt?
A plain-Markdown file published at a site's root (yoursite.com/llms.txt) that lists a project or site's name, a short summary, and a set of links to its documentation or key pages, formatted so a language model can parse it more easily than raw HTML. It was proposed by Jeremy Howard of Answer.AI in September 2024.
What's the required format of an llms.txt file?
Per the spec at llmstxt.org, only two elements are structurally required: an H1 heading with the project or site's name, and directly beneath it, a blockquote containing a short one-paragraph summary. Everything after that is optional: free-form Markdown context (no headings allowed in that section), followed by H2-headed sections that group Markdown links to further pages.
What's the difference between llms.txt and llms-full.txt?
llms.txt is an index: a short summary plus a curated set of links, designed to fit easily in a model's context window. llms-full.txt is the companion file some sites also publish: the actual full text of the documentation concatenated into one file, meant for a case where a tool can handle ingesting everything at once rather than following links from an index.
How is llms.txt different from robots.txt or sitemap.xml?
All three sit at a site's root, but they do different jobs. robots.txt tells crawlers what they're allowed to access, a permissions file, not a content file. sitemap.xml lists a site's URLs in a structured XML format for search engine indexing. llms.txt is neither a permissions file nor a raw URL list. It's a curated, human-and-model-readable Markdown summary of what a site is and which pages matter most, written in prose a model can use directly rather than a machine-only schema.
Sources referenced
This piece covers the file format only. For the adoption data, Google's written statement that Search ignores the file, and the one use case that does hold up, see does llms.txt actually get read by anyone?. That piece also covers where the file originally came from. For how llms.txt fits alongside your actual sitemap, see do you need a separate sitemap for AI engines?
