llms.txt: What This File Can Really Do for Your AI Visibility (2026 Guide)

The short answer: a near-zero-cost bet, with real uses today

Do you need an llms.txt file? Our answer: yes, create it, provided you know exactly what to expect from it. It is not a magic citation lever, and the data shows it; but it is a file that costs an hour, that already has real uses (technical documentation, coding assistants, AI agents loading a page on demand), that Google's own tooling has started checking, and that positions you early on a convention the agentic web may yet consecrate. This guide gives both sides honestly: what llms.txt can bring, what the studies measure today, and how to build a good one if you go for it.

What is the llms.txt file?

llms.txt is a proposed standard launched in September 2024 by Jeremy Howard (Answer.AI) (original proposal): a markdown file placed at the root of the site (yoursite.com/llms.txt) that summarizes the content and lists the important pages with a short description, to offer language models a clean table of contents, free of HTML noise, menus and scripts. A variant, llms-full.txt, goes further by gathering the full content of key pages into a single document.

The intention answers a very real problem: model context windows are limited, modern HTML is cluttered, and a model reading your site on demand works better on an accurate digest than on a script-heavy page. The question is who consumes this menu today, and who might consume it tomorrow.

What llms.txt can already bring

The strongest case: technical documentation. Coding assistants (Claude Code, Cursor and equivalents) can load an llms.txt on demand when a developer points them at a documentation, and this is a use that John Mueller (Google) himself acknowledges. If you are a SaaS with developer documentation, a well-built llms.txt concretely improves the experience of your users working with these tools, and therefore the probability that your product is well understood, well implemented and well recommended in those contexts.

The agentic web is coming, and it changes the math. The key distinction: training crawlers sweep the web in bulk, but AI agents load your pages on demand, at the moment a user asks a question. For those live reads, a clean digest at the root is exactly the format they need. A telling signal: Lighthouse, Chrome's audit tool, added an llms.txt check in May 2026 (version 13.3.0) in its "Agentic Browsing" category, dedicated precisely to these agents (TechWyse report). The same Google that dismisses the file for search has it checked for agents: the door is not closed, it is elsewhere.

Reads observed in the field. Log analyses report GPTBot fetching the file on some sites (Search Engine Land), and OpenAI's bots appear in the llms.txt requests measured by Ahrefs. It is an observed signal, not an official commitment, but it shows the file is not a dead letter everywhere.

You choose what the AIs read first. This is the structural argument: on a site, the architecture guides the reading; llms.txt is the one place where you explicitly tell a machine "here is who we are, here are our authoritative pages". The day an engine consumes it, it reads your version of the site first: up-to-date pages, accurate facts, a well-described offer. Conversely, a site without the file leaves the AI to compose with whatever it finds.

And adoption creates its own momentum. According to the Ahrefs study of May 2026, 28% of observed domains already published a valid llms.txt (Ahrefs study). A convention adopted by more than a quarter of the tech-savvy web becomes hard for engines to ignore: that is exactly how sitemap.xml established itself before being officially consecrated. Nothing guarantees history repeats, but being ready costs an hour; catching up with a consecrated standard always costs more.

What the data measures today: the limits to know

Honesty requires laying out the other side, because it is what calibrates your expectations. The same Ahrefs study shows that 97% of published llms.txt files received no request in the observed month, that 96% of requests came from bots (mostly SEO audit tools), with AI engines' retrieval bots (OAI-SearchBot, PerplexityBot) accounting for only about 1%, and that AI bots do not probe for the file on sites that lack one. SE Ranking, across roughly 300,000 domains, finds no significant correlation between the file's presence and citations in AI answers (SE Ranking study). And Google is explicit: its official guide of May 2026 states that no AI-specific file is needed for its generative features, with Gary Illyes having confirmed as early as July 2025 that Google does not support llms.txt for search.

The conclusion is not "useless"; it is: llms.txt is an asymmetric bet, one hour of cost, zero risk, and real potential if the agentic web consecrates it. What it is not, as of October 2026: a proven citation lever. Anyone selling it to you as such is selling you numbers that do not exist. It is also what we see in our audits: well-cited brands are cited for their extractable content and their corroboration; the file accompanies, it does not replace.

How to build a good llms.txt

Since the cost is low, you might as well do it properly: a bad llms.txt (outdated, contradicting the site) would be worse than no file at all, since it is precisely the document a machine will read as your official version.

1. A markdown file at the root (/llms.txt): an H1 title with the site's name, an accurate summary paragraph of your activity, then sections of links to your authoritative pages, each with a one-line description.
2. Pick the pages as you would for a pitch: offer, pricing, "who it is for", documentation, comparison pages, FAQ. The pages you would want an AI to read before talking about you.
3. Align every description with the site's facts: same numbers, same wording. The file is a promise of consistency, not a second marketing pitch.
4. Add an llms-full.txt if you have documentation: the full content of key pages in a single document is the format coding assistants consume best.
5. Update it with every change to your offer, and date it. An eight-month-old file announcing old prices works against you.
6. Leave robots.txt alone in the process: the two have nothing to do with each other. robots.txt controls access and is genuinely respected by AI bots; it decides whether GPTBot or ClaudeBot can read your site, and therefore whether you can be cited. Check it before anything else.

And if you would rather not do it by hand: the Sapian Metrics platform (transparency: it is our product) generates the llms.txt file directly from your site, aligned with your pages, and you can regenerate it with every change to your offer so it stays accurate.

What remains the priority, with or without the file

llms.txt guides the reading; it does not replace what is read. Engines build their answers on your HTML pages and on what the web says about you: direct answers at the top of each section, facts in plain text (prices, characteristics, "who it is for"), headings phrased like users' questions, the fundamentals of the GEO checklist, and external corroboration, because errors and absences get paid for in the answers. And the only proof that counts remains the repeated measurement of your citations, engine by engine: it is what will tell you whether your setup, file included, produces results.

Frequently asked questions

What is the llms.txt file? A proposed standard launched in September 2024 by Jeremy Howard (Answer.AI): a markdown file at the site's root that summarizes the content and lists the important pages, to make reading easier for language models and AI agents. It is not yet an official standard.

Should you create an llms.txt? Yes, the math is favorable: an hour of work, no risk, real uses today (technical documentation, coding assistants, agents reading on demand) and an option on the future of the agentic web. High priority if you have developer documentation, low priority otherwise, but in both cases after your content fundamentals.

Does Google read llms.txt? Not for search: Google has confirmed (official guide of May 2026, statements by Gary Illyes) that it does not use llms.txt for its results or its AI features. On the other hand, Lighthouse, Chrome's audit tool, now checks the file in its category dedicated to AI agents.

Does the llms.txt file help you get cited by ChatGPT? It is not proven to date: log and correlation studies (Ahrefs, SE Ranking) measure no effect on citations, even though GPTBot is observed fetching the file on some sites. Treat it as a preparation complement, not a citation lever.

Does llms.txt replace robots.txt? No. robots.txt controls crawler access and is genuinely respected by AI bots: it decides whether your site can be read, and therefore cited. llms.txt offers a reading summary. First check that robots.txt does not exclude the AI bots you want citing you.

How do you create a good llms.txt? A markdown file at the root: title with the site's name, accurate summary, sections of links to the authoritative pages with a one-line description each, plus an llms-full.txt if you have documentation. Short, accurate, consistent with the site, and updated with every change to the offer. The Sapian Metrics platform can also generate it automatically from your site.

Conclusion

llms.txt deserves better than the two caricatures that dominate the subject: neither a magic citation lever nor a gimmick to discard. It is an hour of work for a file that already serves coding assistants and agents, that Google's tooling checks on the agentic browsing side, and that puts you in control of what a machine will read about you first the day it consults it. Do it, do it well, and keep expectations calibrated: AI visibility is earned in your pages and verified in the engines' answers. The subject moves fast, and we will update this guide as engines and agents clarify their positions. To know where you really stand, the free Sapian Metrics audit queries 8 engines in minutes and shows you whether your content gets cited; and if you decide to create your llms.txt, the platform generates it for you.