---
title: "What is llms.txt? How to write one, and does it help AI search?"
category: "guides"
url: https://keirolabs.cloud/blog/what-is-llms-txt
---

## The 60-second version

**llms.txt is a markdown file at `/llms.txt` that gives AI agents a short, curated map of your site: a title, a one-paragraph summary, and lists of links to the pages worth reading, ideally in markdown.** Jeremy Howard of Answer.AI proposed it in September 2024, and the spec was revised to version 2 in August 2026 ([llmstxt.org](https://llmstxt.org/)).

- **What it is for:** helping an agent that is already reading your site find the right page fast, without wading through navigation, scripts and cookie banners.
- **Who reads it:** coding agents and AI assistants fetching documentation. OpenAI, Anthropic and Google's Gemini API docs all publish one ([llmsrank](https://llmsrank.dev/learn/llms-txt)).
- **Who does not:** Google Search. Google's John Mueller called the file "purely speculative for now" in June 2026 ([source](https://tomrochette.com/agents/protocols/llms-txt/)). It is not a ranking factor.
- **How long it takes:** an afternoon for a docs site, less for a small product site.

**Write one if agents use your docs or product. Do not expect it to move your Google traffic.** It is a usability feature for AI readers, not an SEO trick.

## What is llms.txt?

**llms.txt is a plain markdown file that tells a language model what your site is and where its most useful content lives.** It sits at the root of your site, like `robots.txt`, but it does a different job.

`robots.txt` tells crawlers what they may access. A sitemap lists every indexable page for search engines. **llms.txt is a curated selection: the twenty pages an agent should read, not the two thousand it could.** It can even link to pages on other sites.

The idea behind it is simple. Web pages are built for people, with menus, ads, scripts and layout. An agent asked "how do I authenticate with this API?" has to dig through all of that to find two paragraphs. llms.txt hands it a short index, and the index points to clean markdown versions of the pages that matter.

## What goes in an llms.txt file?

The spec defines the sections in a fixed order. **Only the H1 is required; everything else is optional but strongly recommended.**

| # | Element | Required? | What it holds |
|---|---|---|---|
| 1 | `# Name` (H1) | Yes | The name of the project or site |
| 2 | `> Summary` (blockquote) | No | A short summary with the key facts needed to understand the rest |
| 3 | Body paragraphs and lists | No | More detail, and how to use the linked files; no headings |
| 4 | `## Section` (H2) with a file list | No | Markdown lists of links, each `[name](url): optional note` |
| 5 | `## Optional` | No | By convention, secondary links an agent can skip when context is short |

Here is a complete example for a fictional API company:

```markdown
# Acme Search API

> Acme is a web search API for AI agents. One key covers search, page
> extraction and cited answers. Base URL: https://api.acme.example

Authentication uses a Bearer token. All endpoints accept and return JSON.
Prices are per 1,000 requests and failed requests are not billed.

## Docs

- [Quickstart](https://acme.example/docs/quickstart.md): first request in two minutes
- [Search endpoint](https://acme.example/docs/search.md): parameters, response shape, limits
- [Extract endpoint](https://acme.example/docs/extract.md): turn a URL into clean markdown
- [Errors and rate limits](https://acme.example/docs/errors.md): status codes and retry rules

## Pricing

- [Pricing](https://acme.example/pricing.md): plans, free tier and per-endpoint costs

## Optional

- [Changelog](https://acme.example/changelog.md): release notes
- [Blog](https://acme.example/blog.md): guides and benchmarks
```

**Each link should point to a markdown version of the page, not the HTML page.** The spec suggests serving that version at the same URL with `.md` added, so `/docs/search` has a twin at `/docs/search.md`. That is the part that does the real work: the agent reads clean text instead of rendered HTML.

## What is llms-full.txt?

**llms-full.txt is a companion file that puts the full text of your docs into one markdown file, so an agent can load everything in one fetch.** Documentation platforms popularized it alongside llms.txt.

The two serve different context budgets. llms.txt stays small enough to fit in any context window and lets the agent fetch only what it needs. llms-full.txt is for agents that want the whole thing at once, which works for a small docs set and fails for a large one. **If your docs are longer than a few hundred thousand tokens, publish only llms.txt, with good links.**

## What changed in llms.txt version 2?

**Version 2 (August 2026) lets an llms.txt file live at any path, not just the root, and adds standard ways for agents to discover it.** A file at `/docs/llms.txt` covers the pages under `/docs/`, and when more than one file applies, agents should use the most specific one ([llmstxt.org](https://llmstxt.org/)).

That matters for sites with separate products or doc sets: each section can carry its own map. The `## Optional` section is still allowed as a convention for secondary links, but it no longer has special meaning in the spec.

## Does llms.txt help with ChatGPT, Perplexity or Google?

**There is no public evidence that publishing llms.txt raises your visibility in ChatGPT, Perplexity, Claude or Google answers.** Google has said plainly that it does not use it, and a 2025 industry analysis reported that major AI platforms did not even request the file during their crawls ([source](https://tomrochette.com/agents/protocols/llms-txt/)).

Where it does help is the agent that is already on your site. When a developer asks Claude Code or Cursor to "use the Acme API", the agent fetches your docs. If it finds llms.txt and markdown twins, it reads the right page in one or two requests and gets the API call right. Without them, it parses HTML, misses details and guesses.

| Who reads it | Does llms.txt help? |
|---|---|
| Coding agents reading your docs (Claude Code, Cursor, Copilot) | Yes, this is its main use |
| AI assistants browsing your site during a chat | Sometimes, if they look for it |
| Google Search ranking | No |
| AI search citations (ChatGPT, Perplexity, AI Overviews) | No evidence it does |
| Model training | Possibly, but that was never the goal |

**If your goal is to be cited in AI answers, the levers are content AI engines can extract and trust: direct answers, tables, dated facts and sources.** Those matter far more than any file. Track whether it is working by checking the prompts your buyers ask, which is what AEO monitoring tools do.

## How do you create an llms.txt file?

1. **List the 10 to 30 pages an agent most needs.** Start with quickstart, core concepts, the main API or product pages, pricing and limits. Leave out marketing pages, team pages and old posts.
2. **Write the H1 and a summary blockquote.** Put in the facts an agent should never get wrong: what the product does, the base URL, the auth method, the pricing model.
3. **Group the links under H2 sections**, each link with a short note saying what the page answers.
4. **Serve markdown twins** at `<page>.md`, or link to wherever clean markdown lives. Many documentation platforms and CMSs, such as Mintlify, and a Drupal recipe, can generate both automatically.
5. **Serve it as plain text** at `/llms.txt` and keep it current. A stale llms.txt that points to old endpoints is worse than none, because agents trust it.
6. **Test it with an agent.** Give an agent only your llms.txt and ask it real questions about your product. If it answers wrong, your file is missing a link or a fact.

```python
# A minimal check: does the file exist, start with an H1, and do its links resolve?
import re, requests

def check_llms_txt(site: str) -> None:
    r = requests.get(f"{site.rstrip('/')}/llms.txt", timeout=20)
    print("status:", r.status_code, "| type:", r.headers.get("content-type"))
    text = r.text.lstrip()
    print("starts with H1:", text.startswith("# "))
    for url in re.findall(r"\]\((https?://[^)\s]+)\)", text)[:30]:
        code = requests.head(url, allow_redirects=True, timeout=20).status_code
        print(code, url)

check_llms_txt("https://docs.anthropic.com")
```

## Common llms.txt mistakes

- **No H1.** The H1 is the only required element, and files that open with key-value pairs or YAML are not llms.txt files by the spec, whatever they are named.
- **Linking to HTML pages.** The point is clean content. Link to markdown versions where you can.
- **Listing every page.** That is a sitemap. Curate.
- **Stale facts.** Old endpoints, retired URLs or last year's prices in the summary will be repeated back to your users by every agent that reads it.
- **Expecting SEO results.** It will not move Google rankings. Measure it by whether agents use your product correctly.

## The short answer

llms.txt is a small, curated markdown map of your site for AI agents. Publish one, with markdown twins of your key pages, if developers or agents use your docs; it takes an afternoon and makes agents less likely to get your product wrong. Do not publish one expecting more Google traffic or more AI citations; for those, write pages that answer questions directly, and monitor the answers.