llms.txt

What it is

llms.txt is a proposed Markdown file that gives language models and agents a curated map of a site’s most important content. Jeremy Howard published the proposal on 3 September 2024, and a second version on 10 August 2026. It addresses a narrow problem: an HTML page wraps its information in navigation, ads and scripts, and every wasted token costs time and money. The format is lean — an H1 naming the site or project is the only required section, usually followed by a short summary and lists of links. Those links should point to LLM-friendly content; the proposal’s own example of that is the Markdown version of a page, which it separately asks sites to serve at the same URL with .md appended or the extension replaced. Version 2 defines what a file at a subpath means, so that a file covers the pages beneath it and the most specific file applies; adds link relations so an agent can find a page’s Markdown version and its covering llms.txt without guessing; and drops the context-expansion tooling that v1 described.

Why it matters

The file is contested, and the disagreement is now on the record rather than a matter of opinion. Google Search states in its own guidance that it does not use llms.txt, and that publishing one will “neither harm nor help” a site’s ranking. Chrome, also Google, ships a Lighthouse audit under its agentic browsing checks that looks for the file, but it stops short of requiring one: a missing file is marked Not Applicable because providing it is optional, and the documentation names a server error as the only failure condition. It does advise creating one, and tells site owners to place it in the root directory. Its stated rationale is that without the file agents may spend more time crawling a site to work out its structure. That documented condition is incomplete. The audit applies three content checks the page does not mention, failing a file it fetched successfully if there is no H1, if no link is written in Markdown link syntax, or if the file is shorter than 50 characters — so the minimal file the spec permits, an H1 and nothing else, fails on two of the three. It allows the fetch two seconds, so a spec-compliant file a server is slow to produce fails as well; the P1 bug filed on exactly that case was closed the day it was raised, once a Lighthouse maintainer traced it to the site’s own response time rather than to the audit. And it requests the file only at the site root, so the subpath files version 2 blesses are invisible to it. Independent measurement sits below both positions: Ahrefs examined 137,210 domains and found that 97% of the llms.txt files it observed received no requests at all in May 2026, and attributes roughly 1 in 1,000 fetches to Chrome’s audit. The surviving use is narrower than the proposal’s ambition. Among the 3% of files that were fetched at all, named AI bots sent 19.5% of the requests, with GPTBot top and Claude-Code second; Ahrefs calls coding agents the closest thing to an intended audience in its data, but did not test whether those fetches land on documentation paths, and notes that a fetch only tells you the file was fetched. Knowledge Company publishes one at knowledgecompany.ai/llms.txt and treats it as an experiment rather than a commitment. The underlying need — a machine-readable statement of what a site actually asserts — is real whether or not this file becomes the way it is met.