llms.txt Explained: The New robots.txt for AI
Every website has had a small text file at its root for decades: robots.txt, a note to search engine crawlers saying which parts of the site they may and may not visit. In 2024 a proposal appeared for a sibling file aimed at a different reader. llms.txt is a plain text file, also at the site root, written for large language models. Where robots.txt says where a crawler may go, llms.txt says what the site is about and where the most useful content lives, in a form a model can read in one pass.
The problem it tries to solve
A language model with a limited context window cannot read a whole website. Pointed at a site, it has to choose a few pages, and the pages are full of navigation, scripts, banners and layout that mean nothing to it. The proposal's idea is simple: give the model a short, clean map. Here is what we are. Here are the pages that matter, with one line each on what they contain. Here, optionally, are plain text versions of those pages with the clutter removed.
What the file contains
The format is deliberately minimal and uses Markdown, which models read fluently. A top heading with the site or business name. A short paragraph describing it. Then one or more sections, each a list of links with a brief description of what the linked page is for. A hotel's file might point to its facts page, its outlets and hours, its policies, its FAQ and its booking information, each described in a sentence.
A companion convention suggests a fuller file, llms-full.txt, that contains the actual content of the key pages concatenated into one document, so a model can take everything in a single read without following any links.
What it is not
It is not a ranking signal, and it is not a permission system. It does not tell a model what it may or may not use; robots.txt and the newer AI specific crawler directives handle that. It does not force any model to read it. It is a courtesy and a convenience: a well written index that makes a site easier to understand for any system that chooses to look.
Who reads it
Adoption is still emerging. Several developer documentation sites publish one, and some AI tools and coding assistants fetch it when pointed at a domain. The major consumer assistants have not announced that they rely on it, and it is wise to treat it as a low cost addition rather than a strategy. Its real value today is twofold: it costs almost nothing, and the exercise of writing it forces you to decide what your most important machine readable content is and whether it exists at all.
How it relates to the rest
Think of three files and one layer. robots.txt controls access. Sitemaps list every page for crawlers. llms.txt summarises the site and points to what matters, for models. Structured data on the pages themselves is the layer that states the facts in machine readable form. The last of these does the most work for AI visibility; llms.txt is a helpful front door to it.
Writing a good one
Keep it short. Describe the business in two or three plain sentences. Link to the pages where facts live, not to marketing pages. Describe each link in terms of what question it answers. Make sure the linked pages are themselves readable without scripts, and ideally carry structured data. If you publish the full version, generate it from the same source as the pages so it never drifts out of date.
The takeaway
llms.txt is a plain text map of your site written for language models: what you are and where your useful content lives. It is cheap to publish, not yet widely relied on, and most valuable as a prompt to make sure the content it points to is clear, factual and machine readable in the first place.