llms.txt: What It Is, Why It Matters, and How to Implement It for AI Search Visibility

What llms.txt is and why it was created

llms.txt is a markdown file placed at the root of a website that provides curated, LLM-friendly information about the site’s content and structure. It was proposed by Jeremy Howard, co-founder of Answer.AI, in September 2024 and published under the Apache 2.0 license at llmstxt.org. The GitHub repository has 2,500 stars and 150 contributors as of mid-2026.

The motivation is straightforward. When a user asks ChatGPT, Claude, Perplexity, or another LLM about a specific website, product, or brand, the AI engine needs to understand what the site is about, what its key pages contain, and how it is structured. Without llms.txt, the engine must crawl the site’s HTML, parse navigation, advertisements, JavaScript, and marketing copy, and attempt to construct an accurate mental model from the noise. The results are frequently inaccurate, hallucinated, or incomplete.

llms.txt solves a different problem than the existing infrastructure. robots.txt controls what crawlers may access during training data collection. Sitemap.xml tells search engines which URLs exist. Schema.org markup provides structured data about entities and content types. llms.txt provides curated context at inference time, the moment a user asks about your site, so that the AI’s answer is grounded in accurate, structured information rather than imperfect inference from raw HTML.

The timing is not accidental. As AI search engines have grown from experimental features to platforms processing billions of daily queries, the gap between what brands intend to communicate and what AI engines infer about them has become a significant business problem. A brand’s website might accurately describe its pricing, features, and competitive positioning. But when ChatGPT summarizes that brand for a user evaluating a purchase, the summary might be wrong, not because the engine is malfunctioning but because the engine could not extract the right information from a page designed for human visual consumption, not machine text processing.

The specification: simpler than most people assume

The llms.txt specification is deliberately minimal. A valid file requires only a single H1 heading naming the project or site. Everything else is optional, but the recommended structure is specific.

The file must be placed at the root path, /llms.txt, and must use Markdown formatting. The structure, in order: an H1 heading with the project or site name, this is the only required element. A blockquote, denoted by > prefix, containing a brief summary with the key information necessary for understanding the rest of the file. Zero or more paragraphs of unstructured Markdown with additional details. Zero or more H2-delimited sections, each containing a list of links in the format [Link title](URL): Optional description. A special ## Optional H2 section whose links can be skipped when a shorter context is needed.

The companion convention, llms-full.txt, is not part of the formal specification but has become standard practice. It contains the expanded content of all pages linked from llms.txt, typically generated by tools like the llms_txt2ctx CLI. Svelte’s implementation is the most sophisticated in the ecosystem: four tiers including llms.txt as the index at 349 tokens, llms-small.txt as a compressed overview, llms-medium.txt as an abridged version, and llms-full.txt at 279,000 tokens containing the complete documentation.

The specification also recommends providing Markdown versions of key HTML pages. For URL paths without file names, append index.html.md. For pages with file extensions like .html, append .md. This convention is separate from llms.txt itself but part of the same ecosystem, providing LLMs with plain-text alternatives to rendered HTML.

A minimal working example:

# GeoStack

> Independent rankings and reviews of AI search visibility tools.
> We track 13+ GEO monitoring platforms across 6 AI search engines.

## Essential pages

- [How We Score](https://geostack.co/how-we-score): Our 7-factor tool evaluation methodology
- [What Is GEO](https://geostack.co/what-is-geo): Complete guide to Generative Engine Optimization

## Tool reviews

- [Profound Review](https://geostack.co/browse/profound): Full review, feature breakdown, pricing, and alternatives
- [Otterly Review](https://geostack.co/browse/otterly): Full review, feature breakdown, pricing, and alternatives
- [Trakkr Review](https://geostack.co/browse/trakkr): Full review, feature breakdown, pricing, and alternatives

## Optional

- [Compare Tools](https://geostack.co/compare): Head-to-head tool comparison pages

Who has adopted llms.txt and what they are doing with it

Adoption among AI and developer-tool companies has been fast and sophisticated. The public directories at llmstxt.site and directory.llmstxt.cloud collectively list over 8,000 implementations across websites, products, and developer tools as of mid-2026.

Anthropic’s claude.com has an llms.txt file structuring product information, pricing, the developer platform, solutions, and partnerships. Perplexity’s documentation at docs.perplexity.ai provides both llms.txt at 4,000 tokens and llms-full.txt at 177,000 tokens. Hugging Face maintains separate llms.txt files for each major project, Transformers at 813,000 tokens, Diffusers at 383,000 tokens, Accelerate at 82,000 tokens, and the Hub at 210,000 tokens.

Cloudflare has the most ambitious implementation. The root llms.txt acts as a directory to roughly 80 product-specific llms.txt files, and the combined llms-full.txt spans 11 million tokens. Cloudflare also offers a “Docs for Agents” feature that serves Markdown versions of documentation when the Accept: text/markdown header is present, effectively turning their entire documentation platform into an LLM-native resource.

Next.js provides a comprehensive llms.txt with over 200 documentation links organized by section, including a ## Optional section for the Pages Router, the legacy routing system. Vite provides 293,000 tokens of full documentation. Svelte’s four-tier approach sets the benchmark for content trimming. Pinecone provides one of the most thoroughly documented implementations with descriptions, not just links, for every entry.

The plugin ecosystem supports automated generation. VitePress and Docusaurus both have plugins that generate llms.txt and Markdown versions of documentation pages. Drupal has an LLM support module. nbdev projects auto-generate Markdown versions of all pages. This matters because manual maintenance of llms.txt is error-prone and frequently neglected. Automated generation from structured documentation ensures the file stays current.

Additional notable implementations include Cursor at 2,000 tokens, Coinbase with a 74,000-token index and a 2 million token llms-full.txt, Postman at 32,000 tokens, ElevenLabs at 22,000 tokens, Zapier at 14,000 tokens index with 301,000 token full version, Resend at 8,000 and 306,000 tokens, and CrewAI at 9,000 and 412,000 tokens.

How llms.txt fits into a broader GEO stack

The GEO infrastructure stack is coalescing around four layers. The crawl-control layer, robots.txt and sitemap.xml, governs what search and AI crawlers may access and what URLs exist. The structured-data layer, Schema.org JSON-LD markup, provides machine-readable entity and content metadata. The inference-time layer, llms.txt and llms-full.txt, provides curated context for when an LLM is asked about your site. The tool-integration layer, MCP servers and API endpoints, enables AI agents to interact with your platform programmatically.

These layers are complementary, not competitive. A brand that implements robots.txt, schema markup, and llms.txt is providing different types of information at different points in the AI interaction pipeline. The crawler uses robots.txt and sitemap.xml to discover and access content. The training pipeline uses schema markup to understand entity relationships. The inference process uses llms.txt to ground responses in curated context. The agent layer uses MCP and APIs to take action. Each layer addresses a different stage and a different failure mode.

The most common failure mode that llms.txt addresses is inaccurate representation. A user asks an AI about your brand, and the AI summarizes your website based on whatever content its crawler happened to parse. Without llms.txt, the AI’s summary depends on which pages the crawler visited, which sections of those pages it successfully parsed, and how it resolved conflicts between different pages on your site. With llms.txt, the AI receives a curated map of your content, written in the plain-text Markdown format that LLMs process most accurately, and presenting the information hierarchy you intend.

The second failure mode is omission. Without llms.txt, an AI engine asked about your documentation might crawl your homepage and your getting-started guide but miss your API reference entirely because the crawler followed only two levels of links. With llms.txt, every key section of your documentation is explicitly listed with descriptions, and the AI is guided to the most important pages regardless of how they happen to be linked in the site navigation.

The third failure mode is staleness. llms.txt files can carry “Last updated” dates, as Anthropic’s implementation does, signaling to AI engines that the information is current and maintained. A website whose content has been updated but whose llms.txt file still points to old pages creates a worse representation problem than having no llms.txt at all.

Implementation guide: creating an llms.txt that actually helps

Step one is to define what an AI needs to know about your site when a user asks a question about it. This is not your marketing pitch. It is a factual summary. What does your organization do? What are your primary products or service lines? What are the three to five pages someone should read to understand what you offer? What questions do users commonly ask about your brand, and which pages answer them?

Step two is to structure the file. Start with an H1 naming your organization or project. Follow with a blockquote summary paragraph. Use H2 sections to group links by functional area: documentation, products, guides, API reference, pricing, about, blog. Each link should include a colon-delimited description explaining what the linked page contains. A bare URL list with no descriptions is nearly useless. The description is the part an LLM uses to determine whether to fetch the linked page for a given question.

Step three is to decide what to include and exclude. The llms.txt file is an index, not an exhaustive catalog. It should contain the pages that would help an AI answer the most common questions about your site. Navigation pages, landing pages, marketing pages with heavy design but light content, and pages that duplicate information from linked pages should be excluded. The ## Optional section is for content that adds detail but is not necessary for answering basic questions. If the AI needs to shorten its context, it can drop Optional links.

Step four is to generate the llms-full.txt companion. This is the expanded version containing all the content from the linked pages. If your documentation is built with VitePress or Docusaurus, the respective plugins can generate it automatically. If not, the llms_txt2ctx CLI can parse your llms.txt and assemble the full context file. Without llms-full.txt, the AI still needs to crawl each linked page individually, which is slower and introduces more potential for parsing errors.

Step five is to test. The specification includes a Python testing script that parses your llms.txt, converts it to XML context, and passes it to a language model to verify that the model can answer questions about your site correctly using only the file. The manual approach is to upload your llms.txt, then ask ChatGPT with web search, Claude with web search, and Perplexity a series of questions about your site. Compare the answers to what you expect. If the AI makes errors, fabricates details, or omits important information, your llms.txt needs more specificity in its descriptions or clearer section organization.

Step six is to maintain the file. llms.txt should be regenerated whenever your site structure changes. Anthropic includes a “Last updated” date in their file, which is a good practice to adopt. An llms.txt that points to pages that have been removed or restructured is actively harmful, because AI engines will attempt to retrieve those pages, find nothing, and construct answers from incomplete information.

Common mistakes and how to avoid them

The most frequent implementation error is treating llms.txt like robots.txt. The syntax is Markdown, not allow and block directives. A file that reads like a crawl policy is unusable by the LLMs it is supposed to serve.

Omitting link descriptions is second. An LLM looking at a list of URLs with no descriptions has no basis for deciding which pages are relevant to the user’s question. It must either follow every link, burning context window, or guess based on the URL slugs alone. The colon-delimited description after each link is the difference between a useful file and a useless one.

Linking to HTML pages instead of Markdown versions is a missed optimization. LLMs process plain text more accurately than rendered HTML. If you provide .md versions of key pages at the same URL, link to those. If you do not, the LLM must parse your HTML, extracting content from among navigation, advertisements, scripts, and styling.

Publishing and then never updating is the most common maintenance failure. llms.txt files that reflect the site’s structure as it was six months ago are worse than having no llms.txt. AI engines will attempt to retrieve pages that have moved or been deleted and construct answers from the combination of available and unavailable links.

Providing too much content in the main file defeats the purpose. The file is an index. If it exceeds a few hundred tokens, you are including detail that belongs in the linked pages or in llms-full.txt. LLMs have context window limits, and the llms.txt file should consume as little of that window as possible while still providing a complete structural map.

Blocking AI crawlers in robots.txt while publishing an llms.txt creates a contradiction. The llms.txt file tells AI engines what to understand. The robots.txt file tells them not to access the pages containing that information. Both files need to be aligned.

The bottom line

llms.txt is the simplest technical optimization in the GEO toolkit and one of the most persistently overlooked. It takes under an hour to write for a typical website. It requires no server configuration beyond placing a file at a known path. The file format is Markdown, which every LLM natively processes. The major AI companies, Anthropic, Perplexity, OpenAI have infrastructure that can consume it, and those companies are among the standards earliest adopters.

The file does not guarantee citations. It reduces the error rate when an AI engine attempts to answer questions about your brand. For organizations investing in content, documentation, and authority building, that accuracy gap translates directly into whether a user who encounters your brand through an AI answer leaves that interaction with an accurate understanding or a confused impression.

The implementation decision is not whether to adopt llms.txt. The standard is too simple, too widely supported, and too useful to ignore. The decision is whether to implement it well, with structured sections, descriptive links, maintained currency, and tested accuracy, or to implement it poorly and let an outdated or incomplete file misrepresent your brand to every AI engine that references it.

Explore the GeoStack Wiki for more GEO implementation guides →