llms.txt explained: what it does and doesn't do

A file called llms.txt has been showing up at the root of documentation sites, SaaS product pages, and developer tools over the past two years. It borrows the format of robots.txt and sitemap.xml: a plain text file at a predictable URL, meant to help software understand a site. The pitch is that it gives AI models a clean, curated map of your best content instead of forcing them to crawl and parse full HTML pages. The reality is more uncertain than the pitch.

What llms.txt actually is

llms.txt is a markdown file placed at yoursite.com/llms.txt. It typically opens with the site or product name and a one-line summary, then lists links to key pages grouped under headings, each with a short description. A related convention, llms-full.txt, bundles full page content into one file so a model can read everything in a single fetch. The format was proposed in 2024 by Jeremy Howard and the team at Answer.AI, aimed at documentation-heavy sites where a model benefits from a short, hand-picked reading list instead of an entire site crawl.

How it differs from robots.txt and sitemaps

robots.txt is a permission file. It tells crawlers which paths they may or may not fetch. sitemap.xml is a discovery file. It lists every URL you want indexed, with no ranking of importance. llms.txt is a curation file. It says, out of everything on this site, these are the pages that best explain who we are and what we do. It does not block or allow anything, and it carries no technical enforcement. A model, or a human, can ignore it completely.

Does any AI engine actually read it

This is the honest gap. No major AI vendor, including OpenAI, Anthropic, Google, or Perplexity, has published a commitment to fetch llms.txt or give it special weight during a model's training or a live retrieval step. Some AI coding tools and browser agents do reference it when a user points them at a specific site, since it is a fast way to get an overview without pulling in full pages. But there is no confirmed link yet between having an llms.txt file and getting cited more often by an AI Overview or a ChatGPT answer. Treat it as a plausible bet, not a proven ranking factor.

Where it can still help

Even without a confirmed model behavior tied to it, llms.txt has a few uses that hold up on their own. It is cheap to write and does not require touching your CMS or site architecture. It works as a compact brief for AI agents and internal tools that a developer or support team points at your docs. And drafting it forces a useful exercise: picking the handful of pages that best represent your product, in your own words, without SEO padding.

What to include

• A one or two sentence summary of the site or product near the top
• Links to your most important docs, guides, or product pages, grouped under short headings
• A one-line description next to each link explaining what a reader will find there
• Only pages you would want an AI system to actually quote

How to write one

1. List the 10 to 20 pages that best explain your product or content, skipping thin or duplicate pages
2. Write a one-line summary for each link in plain language, not marketing copy
3. Group the links under a few clear headings, such as Docs, Guides, and Pricing
4. Save the file as llms.txt at your site root; it does not need to be linked from anywhere else on the site
5. Revisit it when you add or retire major pages, the same way you would maintain a sitemap

llms.txt does not replace good content or clean HTML. It just says, if you are only going to read a few pages, read these.

The takeaway

llms.txt is not a hard rule, and it will not fix a site that AI engines otherwise ignore. But it costs an afternoon to build, it carries essentially no downside, and it forces you to name your best pages in plain language. Add it as a small, low-risk piece of an AEO strategy, not a replacement for the structured data, clean HTML, and answerable content that actually get a site cited.