llms.txt: What It Is, How to Write It and Whether It Works

What does the proposal say?

The specification, published at llmstxt.org, describes a simple format, in this order:

1. An H1 with the project or site name. It is the only required element.
2. A blockquote with a short summary.
3. Optional paragraphs or lists with more context, without headings.
4. H2 sections with lists of Markdown links, each with an optional note.
5. An H2 section called “Optional” for secondary links that an agent can skip when it needs less context.

The page itself says the file should stay small enough to fit in an agent’s context, with the detail living behind the links. The page presents itself as version 2 of the proposal (modified on August 10, 2026) and says it remains open to community input. Treat it as a proposal, not a standard.

How is it different from robots.txt and the sitemap?

Three files, three different jobs:

The specification itself notes that a sitemap often covers more content than fits in a context window and does not point to LLM-readable versions, and that robots.txt indicates which access is acceptable, while llms.txt is used on demand during inference and can complement robots.txt.

Do AI crawlers use it?

Here the data is uncomfortable. According to PPC Land, which summarizes a tracking study by Originality.ai, the number of llms.txt files grew 8.8 times between June 2025 and May 2026, but 97% of them received no requests in May 2026, based on Ahrefs research using server logs from 137,000 domains.

What the companies that matter here say:

• Google Search: its guide to optimizing for generative AI features says you do not need new machine-readable files, AI text files, markup or Markdown to appear in Google Search, and that Google Search does not use them. Creating them is optional and neither helps nor hurts visibility.
• John Mueller (Google): in June 2025 he wrote on Bluesky that “no AI system currently uses llms.txt” and that chatbots fetch pages but not that file, which is obvious in server logs.
• OpenAI: its crawler documentation explains how to manage OAI-SearchBot and GPTBot with robots.txt and does not mention llms.txt as a control. Its own developer documentation does publish an llms.txt as an index, which shows a different use: helping coding agents read technical docs.
• Anthropic: its page on web crawling explains how to block ClaudeBot with robots.txt and does not mention llms.txt.
• Chrome Lighthouse: it has an experimental llms.txt audit in its agentic browsing category, where the file is optional and a missing file (a 404) makes the audit not applicable.

The pattern is consistent: for ranking in AI search there is no evidence that the file increases citations. Where it can help is in technical documentation read by coding agents or agentic browsers.

How do you write one?

A minimal example for a design portfolio:

# Site name

> Product design portfolio with case studies and an AI wiki for designers.

## Case studies

- [Case study A](https://example.com/work/a/): redesign of a payment flow.

## Optional

- [About](https://example.com/about/): background and contact.

Keep the file short, with links to pages that exist and answer a clear question. If you have Markdown versions of your pages, you can link to them, but that is a separate step.

Steps to decide whether to publish it

1. Check whether your site has a structure an agent can walk without help.
2. Write the file in 20 minutes, with no more than 10 links.
3. Link only pages that answer real questions from your users.
4. Check your server logs to see whether any agent requests it. If nobody requests it, do not count it as a gain.
5. Prioritize before the file: clear, dated content with named authorship, and no bot you care about blocked in robots.txt.

Example

Illustrative example: a wiki with 200 articles publishes an llms.txt with ten links to its most-visited definition pages, and an Optional section with the other categories. A month later, the team checks its logs and finds almost no requests to the file. It keeps the file because it costs little, but does not present it as a ranking improvement. The figures are invented to illustrate the method.

What should you check in your logs before deciding?

If you have already published the file, or want to know whether it is worth it, check your server logs for at least a month. Look for requests to the file’s path and separate those from known crawlers from those from browsers. Also compare how often the pages the file links to are requested against pages it does not link to. If the linked pages get no more agent visits, the file is not doing its job, even if it is well written.

Record the date of publication and of every change. That way, if agent traffic rises or falls, you can tell whether it matches the file or something else. Without that reference, any conclusion about llms.txt is a guess.

Common mistakes

• Selling llms.txt as a guaranteed ranking tool.
• Including hundreds of links, which is exactly what the proposal tries to avoid.
• Linking to pages that return errors or redirects.
• Leaving the file out of date when the site changes.
• Using it to hide content that belongs in robots.txt.

When NOT to publish it

When the site is very small and the file repeats what the homepage already says, or when you cannot keep it maintained. It also makes little sense if your team does not measure anything. To understand what changes when AI searches instead of a classic search engine, review what AEO is and how to write content that LLMs cite.

Sources

• llmstxt.org, The llms.txt file (proposal)
• PPC Land, llms.txt adoption rises 8.8x but 97% of files get zero AI requests
• Google Search Central, Optimizing your website for generative AI features on Google Search
• Search Engine Roundtable, Google: No AI System Currently Uses LLMs.txt
• Chrome for Developers, Lighthouse llms.txt audit
• OpenAI, crawler documentation (bots)
• Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?