llms.txt — the emerging standard for AI crawlers

Like robots.txt for AI training. Sites adopting it now will shape how future LLMs understand the web.

What it is

The llms.txt proposal (2024 → quasi-standard 2026) is a single Markdown file at your domain root that tells AI training crawlers and LLM-powered search agents how to navigate your site.

It's structured like a sitemap, but oriented around content meaning rather than URLs. Major sections: an intro, a brief site overview, links to canonical content with one-line descriptions.

What's in it

# Acme Inc.

> One-paragraph summary of what we do.

## Docs
- [Getting started](/docs/start.md): 5-minute intro
- [API reference](/docs/api.md): full endpoint docs

## Blog
- [Why we built X](/blog/why-x.md): 2024-Q3 retrospective

Why it matters now

More frontier models are doing live retrieval (Claude Web Search, ChatGPT Search, Gemini Deep Research). These agents now consult llms.txt as their first step on a domain — both to scope what to look at and to follow your preferred organization. Sites that don't have one rely on heuristic crawling.

Honest assessment

Adoption is still maybe 5% of sites.
Different agents read it differently.
It's not yet load-bearing for SEO in classical search.

But adding one is 30 minutes of work and the cost of being late if it does become the standard is real. We added one for this site this week.