Recipe 0038 min read

llms.txt

A practical guide to `llms.txt`: what it is, what it is not, when it helps, and how to publish one without turning it into cargo cult AI SEO.

TL;DR

`llms.txt` is a community convention for publishing a compact, Markdown-friendly map of a site’s most useful pages for LLMs and agent systems.
It is not an official web standard, not a guarantee of crawling or citation, and not a substitute for indexable, trustworthy, well-structured pages.
The main benefit is clarity: it can give AI systems a cleaner entry point than raw navigation, especially for docs-heavy or concept-heavy sites.
The safest way to use it is as lightweight housekeeping for your best pages, not as a major content or ranking strategy.
If the underlying pages are weak, stale, or hard to crawl, `llms.txt` will not rescue them.
Keep it short, curated, and obviously maintained. A bloated directory dump defeats the point.
As of July 9, 2026, support is still voluntary and uneven, so treat it as low-cost optional infrastructure rather than a proven growth lever.

Quick view

Best for

Docs, blogs, knowledge bases, API sites

Core shift

Curated map instead of raw site sprawl

Do first

Curate ten important URLs

Primary risk

Cargo cult AI SEO

What & why

`llms.txt` is a plain-text file, typically published at `/llms.txt`, that gives language models a cleaner overview of the important content on a site.

The idea is simple: many sites are full of navigation chrome, duplicated paths, and pages that matter to browsers more than they matter to reasoning systems. A curated text file can act as a better front door.

The proposal is especially attractive for documentation, reference libraries, and concept-heavy sites where the best pages are a small subset of the full URL graph.

That said, `llms.txt` is a convention, not a standards-body-backed protocol. It may help some systems, but support is voluntary and inconsistent.

Mental model

Think of `llms.txt` as housekeeping, not magic. It is closer to a site-maintained reading list than to a ranking signal.

A sitemap says which URLs exist. A `robots.txt` file says where crawlers should or should not go. A `llms.txt` file tries to say which pages are most useful to understand first and how they relate.

That means the file is only as good as the judgment behind it. If you point AI systems at weak pages, you have simply created a cleaner path to weak pages.

What it can realistically help with

Giving LLM-powered tools a compact list of high-value pages without asking them to infer importance from full-site navigation.
Reducing noise for docs or knowledge sites where the important explanatory pages are buried inside menus, changelogs, or duplicated template routes.
Making your own intent about canonical learning resources more obvious to humans and machines.
Creating one easy-to-maintain place to list key guides, policies, and reference pages in Markdown-friendly form.

What it does not do

It does not force search engines or AI systems to crawl, index, or cite your content.
It does not replace rendered HTML, internal links, clear authorship, original evidence, or page quality.
It does not act like a permission or exclusion system in the way people sometimes imagine `robots.txt` does.
It does not turn a weak content strategy into an AI visibility strategy.

Start here

01
Pick the handful of pages that actually matter
Start with the pages you would want a smart human to read first: the best explainer, the best getting-started page, the key docs hub, the pricing or product overview, and the most important policies or contact routes.
If the file becomes a dump of everything, it loses its value as a filter.
02
Write for comprehension, not keyword coverage
Use plain labels and short descriptions that explain why each link matters.
This is one of the rare places where editorial restraint is a feature. You are trying to reduce ambiguity, not stuff phrases.
03
Publish it at a stable root path
Use `/llms.txt` unless you have a very specific structural reason not to.
Keep formatting simple and predictable so the file remains easy to parse and easy to maintain.
04
Treat maintenance like any other navigation surface
If a page becomes stale, redirected, or strategically unimportant, update the file.
A neglected `llms.txt` can quietly become a source of misinformation about your own site.
05
Measure with humility
Do not expect a direct analytics spike just because the file exists.
Instead, treat it as one small part of making your site easier for AI systems and humans to understand.

Build the full system

01
Curate pages with strong entity clarity
Lead with pages that clearly explain who you are, what the product or publication does, and which pages are the best references for the topic.
This helps more than listing dozens of thin articles or utility pages.
02
Prefer durable pages over news churn
Use pages that will remain useful over time, such as canonical guides, docs overviews, API references, comparison pages, and policies.
Rapidly expiring announcements usually belong elsewhere unless they are central to understanding the product right now.
03
Keep the file short enough to scan
If a human cannot understand the file quickly, an LLM probably is not getting the intended benefit either.
A short curated list is usually better than mirroring your entire sitemap.
04
Coordinate it with real site architecture
The pages in `llms.txt` should also be easy to reach through normal internal links and standard crawl paths.
A private map to pages that are otherwise structurally weak is not a robust content system.
05
Use it as a secondary layer after SEO and GEO basics
Clean rendering, crawlability, trust signals, and strong passages still matter more than the presence of this file.
The highest-value use of `llms.txt` is usually to reinforce an already coherent site, not to patch a messy one.

What stronger `llms.txt` usage looks like

01
Directory dump to curated map
Before: a huge list of every article, tag page, and utility route.
After: a concise list of the handful of pages that best explain the site, product, or knowledge base.
Why it works: the file becomes a signal of editorial priority instead of another index to sift through.
02
Vague labels to descriptive labels
Before: `Docs`, `Guide`, `About`.
After: `API Authentication Guide`, `Core Product Overview`, `Editorial Policy`, `Pricing and Plan Comparison`.
Why it works: clearer labels reduce guesswork for both models and people.
03
AI hype to honest positioning
Before: treating the file as a guaranteed way to rank in AI search.
After: treating it as optional infrastructure that may improve comprehension for systems that choose to read it.
Why it works: the team makes better tradeoffs when expectations are realistic.

Choose the next move

01
If your docs are sprawling, make `llms.txt` a curated entrypoint
Point to the pages that explain the product fastest and best, rather than trying to summarize every subsection.
02
If your content quality is weak, improve pages before adding this file
A clean pointer system matters less than the quality of the pages being pointed to.
03
If your site already has strong hubs, mirror that judgment here
Use `llms.txt` to reinforce the same canonical pages your internal linking system already treats as important.
04
If no one on the team will maintain it, keep it minimal
A tiny accurate file is better than an ambitious stale one.
05
If you need guaranteed behavior, do not depend on `llms.txt`
Because support is voluntary, it should not be the only mechanism behind any business-critical discovery flow.

Prove it is working

Check whether the file stays current as pages change, move, or are consolidated.
Look for better consistency in how agents or LLM-based tools identify your canonical pages during testing.
Use log review or qualitative retrieval checks instead of assuming a clean attribution lift will appear in analytics.
Compare the file against your actual top-value pages every time strategy or IA changes.
Treat any downstream visibility gains as supporting evidence, not proof of direct causality.

Final checklist

The file is published at a stable and discoverable path, ideally `/llms.txt`.

Only high-value canonical pages are included.

Labels and descriptions are plain, specific, and useful.

The file is short enough to scan quickly.

Every linked page is still live, current, and strategically important.

The linked pages are also well-linked through the normal site architecture.

The team understands that `llms.txt` is optional infrastructure, not a guaranteed ranking mechanism.

Ownership for future updates is clear.

Common ways this goes off track

Dumping an entire sitemap into `llms.txt` and calling it optimization.
Using vague labels that hide what each linked page is actually for.
Publishing the file while the underlying pages remain weak, duplicated, or stale.
Treating it as a replacement for crawlability, good HTML, strong internal links, or trustworthy content.
Assuming every major model or AI search product reads and uses it the same way.
Letting the file drift out of sync with the actual site.

Examples

llmstxt.org proposal — The original proposal and format overview for the `llms.txt` convention.
llms.txt reference explainer — A simple explainer and practical reference for how the convention is presented.
Software Cookbook `llms.txt` — A live example from this site showing a concise recipe index for language models.

References

[1] The /llms.txt file — The original public proposal describing the purpose and format of the `llms.txt` convention.
[2] llms.txt core format reference — Technical format details for file structure and sections.
[3] What Is llms.txt, and Should You Care About It? — A recent practical industry take that is useful partly because it shows how mixed expectations around `llms.txt` still are.

More recipes

Subscribe to all Software Cookbook updates

If this recipe was useful, subscribe for the next ones. New pages arrive with the same concise, technical format.