llms.txt is a proposed file that sits at the root of your site and gives AI models a curated Markdown index of your most important pages. It is a community convention rather than a standard, and as of 2026 no major AI company has publicly committed to using it in production. Google has confirmed on the record that it does not.
That makes this an unusual recommendation to write. The file is cheap, harmless and occasionally useful, and almost everything claimed about it in marketing content is unsupported. Both of those things are true at once.
What the file is
The proposal came from Jeremy Howard at Answer.AI in September 2024. The reasoning behind it is sound. Models have limited context windows, HTML is cluttered with navigation and scripts, and a model trying to understand a site burns most of its budget on markup rather than meaning.
llms.txt offers a cleaner entry point: a plain Markdown file listing your key URLs with a one line description of each, so a model can find your canonical pages without parsing your entire template.
A minimal version looks like this:
# Your Business Name
> One sentence describing what the business does and who it serves.
## Core pages
- [Services](https://yourdomain.com/services): What you sell and what it costs
- [About](https://yourdomain.com/about): Who runs the business and their background
- [Contact](https://yourdomain.com/contact): How to get in touch
## Documentation
- [Getting started](https://yourdomain.com/docs/start): Setup guide
- [API reference](https://yourdomain.com/docs/api): Endpoint documentation
Save it as llms.txt in your site root. There is no registration and no
gatekeeper. You publish it and hope the systems that matter choose to read it. The
specification lives at llmstxt.org.
A companion file, llms-full.txt, contains the full text of your key pages
in one document so a model can ingest everything in a single request. Most sites do not
need it. Documentation heavy products sometimes do.
llms.txt versus robots.txt
These get compared constantly and they solve opposite problems.
robots.txt is a permission file. It tells crawlers which paths they may and may not access. It is a recognized standard, every major crawler respects it, and getting it wrong has immediate consequences.
llms.txt is a recommendation file. It suggests which pages matter. It grants no permissions, blocks nothing, and reading it is entirely voluntary for each AI system.
If you have to choose where to spend an hour, spend it on robots.txt. One of these files decides whether AI can reach your site at all. The other offers a reading list nobody has agreed to follow.
What the data actually shows
This is the part most articles about llms.txt leave out.
Adoption is around one site in ten. An SE Ranking study covering 300,000 domains found an adoption rate of roughly 10%. More tellingly, among the fifty most AI-cited domains, only one had the file at all. The sites winning citations overwhelmingly are not doing this.
Crawlers barely request it. A monitoring analysis by Limy.ai covering
more than 500 million AI bot events over a ninety day window found only a few hundred
requests targeting /llms.txt directly. GPTBot, ClaudeBot, PerplexityBot,
OAI-SearchBot and Google-Extended overwhelmingly crawl ordinary HTML instead.
Google has said no on the record. Gary Illyes confirmed in July 2025 that Google does not support llms.txt and is not planning to. John Mueller compared it to the discredited keywords meta tag and made a further point worth sitting with: a model that has already downloaded your real content has no reason to trust a separate file describing that content. Google's own generative AI documentation lists llms.txt among tactics you do not need.
No provider has committed. As of 2026, none of OpenAI, Google, Anthropic, Meta or Mistral has publicly committed to reading or acting on the file in their production answer systems. GPTBot fetches it occasionally, which is not the same thing as using it.
Mueller raised one more objection that deserves attention. Because llms.txt describes your content rather than being your content, nothing stops a site listing one set of pages in the file while serving something else to users. That is cloaking, and it is a structural reason a serious provider would be cautious about trusting the file at all.
Where it genuinely does get used
The picture is not uniformly negative, and the useful part sits in a different place than the marketing suggests.
AI coding assistants read it. Cursor, GitHub Copilot, Windsurf, Cline and Aider fetch llms.txt to locate accurate documentation with less token waste. This is the file's designed use case and it works.
Anthropic recommends it in its guidance on writing for agents, and OpenAI uses it for its Agents SDK and Agentic Commerce Protocol. Both are agent contexts rather than answer contexts.
Browser agent tooling now checks for it. Chrome's Lighthouse added an agentic browsing audit category in May 2026 that looks for the file.
Perplexity retrieves it, according to several reports, though the extent to which it influences source selection is not documented.
So the honest summary is that llms.txt is real infrastructure for agents and unproven infrastructure for answers. If your buyers are developers using AI coding tools, or your category has a plausible agent driven use case like comparison shopping, vendor research or booking, the file is doing something. If you are a local service business hoping it will get you into ChatGPT's recommendations, the evidence does not support that.
Should you publish one?
Yes, if you run documentation, an API product or a developer tool, or if agent driven traffic is plausible in your category. It takes under an hour and the tooling that reads it is already in production.
Yes, as a cheap option, if your team has spare capacity and you want to be early on agent traffic. It cannot hurt. Nothing on your site changes when you add it.
No, if your team is stretched. The opportunity cost is the problem, not the file. Half a day spent on llms.txt is half a day not spent on entity consistency, structured data or writing a page that states its answer plainly, and those three have measurable effects that this does not.
The mistake worth avoiding is treating llms.txt as a substitute for the work that actually drives AI citation. If you publish one, publish it as a small experiment alongside the fundamentals, not instead of them.
If you already have one
Keep it, and keep it accurate. A stale llms.txt pointing at pages that no longer exist is worse than none, because the small number of systems that do read it will get a wrong picture of your site.
Then check whether anything is reading it. Filter your access logs for requests to
/llms.txt and /llms-full.txt from known AI user agents. That
query gives you a real answer for your site in ten minutes, rather than an industry
average that may not apply to you.
That habit generalizes. Most claims in this field are testable against your own server logs, and the logs are more honest than any blog post, including this one.