# What Is llms.txt? How the File Works and Who Uses It

**Definition:** llms.txt is a proposed standard for a markdown file at a website's root (example.com/llms.txt) that summarizes the site and links to its most useful pages for language models. Jeremy Howard of Answer.AI proposed it in September 2024. Google says its search ignores the file, and no other major AI search engine has said it uses one.

**Published:** September 30, 2026  
**Author:** Connor Lahey

---

## How llms.txt works

A web page built for people carries a lot a language model doesn't need, like navigation menus and cookie banners. llms.txt is meant to cut through that. The site owner writes one markdown file that says what the site is and lists the pages worth reading, with a short note on each.

Jeremy Howard, co-founder of Answer.AI, published the proposal on September 3, 2024, at [llmstxt.org](https://llmstxt.org/). It's a community convention. No standards body such as the W3C or IETF has adopted it.

The idea is that an AI tool can fetch one small file and know where to look, instead of crawling and cleaning dozens of pages to piece the same picture together.

## The llms.txt format

The format is deliberately simple. In order, the file contains:

1. **An H1 with the site or project name.** This is the only required section.
2. **A blockquote** with a short summary of what the site is.
3. **Any other markdown** a model needs for context, such as paragraphs or lists.
4. **H2 sections of links**, each a markdown list of `[name](url)` entries with an optional note after a colon.

By convention, a section titled "Optional" holds secondary links that a tool can skip when it's short on space.

## llms.txt example

Here's a short example for a fictional project management company:

```markdown
# Acme

> Acme is project management software for creative agencies,
> with built-in time tracking and client approvals.

Plans start at $12 per user per month. All plans include unlimited projects.

## Product

- [Features](https://acme.com/features): Overview of task boards, time tracking, and approvals
- [Pricing](https://acme.com/pricing): Plans, limits, and billing FAQs
- [Integrations](https://acme.com/integrations): Slack, Google Drive, and QuickBooks

## Docs

- [Getting started](https://acme.com/docs/start): Set up a workspace in 10 minutes
- [API reference](https://acme.com/docs/api): REST endpoints and authentication

## Optional

- [Company blog](https://acme.com/blog): Agency operations articles
```

## llms.txt vs llms-full.txt vs robots.txt

These files come up together, but they do different jobs.

| File | What it does | Controls access? |
|---|---|---|
| robots.txt | Tells crawlers which URLs they may fetch | Yes |
| llms.txt | Summarizes the site and links to key pages | No |
| llms-full.txt | Puts the full text of key pages in one file | No |
| XML sitemap | Lists every URL you want crawled | No |

You still block or allow AI crawlers in robots.txt. See [AI crawlers](https://www.searchable.com/glossary#ai-crawlers) and [GPTBot](https://www.searchable.com/glossary#gptbot) for the user agents involved.

## Do AI engines use llms.txt?

The evidence you can check is thin.

- **Google** says it doesn't. Its [guide to optimizing for generative AI features](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) says you don't need AI text files to appear in Google Search, including AI Overviews and AI Mode, and that keeping one "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."
- **OpenAI, Anthropic, Perplexity, and Microsoft** document their crawlers and robots.txt rules, but as of September 2026 we found no public statement from any of them that their AI search products read llms.txt when choosing sources.
- **Developer tools** are where the file gets real use. Documentation sites such as [Anthropic's](https://platform.claude.com/llms.txt) and [Stripe's](https://docs.stripe.com/llms.txt) publish one, so coding assistants (and the developers using them) can load clean docs on request.

So llms.txt helps the tools that go looking for it, and nothing suggests AI search engines treat it as a ranking or citation signal. When an AI engine answers a question it uses [retrieval-augmented generation](https://www.searchable.com/glossary/retrieval-augmented-generation): it searches the web and reads the pages it finds, which means those pages have to be crawlable and clear on their own.

## Should you add an llms.txt file?

It's a small job and it does no harm. Documentation sites and developer products have the most to gain, because their readers are the ones most likely to point an AI tool at the file.

For AI search visibility, though, the effort pays off more on the pages themselves. Serve your content in the HTML so crawlers that don't run JavaScript can read it (see [server-side rendering](https://www.searchable.com/glossary#server-side-rendering)), and put a clear answer near the top of each page in a passage that makes sense on its own. Mentions on the third-party sites AI engines cite matter as well, especially when they recommend brands in your category.

Searchable publishes its own file at [searchable.com/llms.txt](https://www.searchable.com/llms.txt). We also serve markdown versions of our pages directly to AI crawlers, so they get clean text wherever they land. [Our dual-served content system](https://www.searchable.com/blog/we-built-a-dual-served-content-system-for-ai-crawlers) explains how that works.

## Frequently asked questions

### Where does the llms.txt file go?

At the root of your domain, so it loads at example.com/llms.txt, the same place robots.txt lives. The proposal also allows one in a subpath, such as example.com/docs/llms.txt, to cover that section of a site.

### Does llms.txt help SEO?

Not in Google. Google's guidance on AI features says an llms.txt file will neither harm nor help a site's visibility or rankings in Google Search, because Google Search ignores it. Any value comes from other AI tools that choose to read it.

### Should I create an llms.txt file?

Yes, if you can keep it accurate, since it takes little effort. Documentation and developer products get the most from it. It won't replace crawlable HTML or clear page content.

### What is the difference between llms.txt and robots.txt?

robots.txt tells crawlers which URLs they may fetch. llms.txt doesn't grant or block access at all. It's a curated summary and reading list that helps a model find your best content.

### What is llms-full.txt?

A companion file some sites publish with the full text of their documentation in one markdown file, instead of links to it. Plenty of sites use the convention, but it isn't part of the llms.txt proposal.

### How do I see a website's llms.txt file?

Add /llms.txt to the end of the domain in your browser, for example searchable.com/llms.txt. If the site publishes one, the markdown loads as plain text.

### How do you create an llms.txt file?

Write a markdown file in the format shown above, starting with an H1 for your site name and followed by H2 sections that link your key pages. Save it as llms.txt at your site root and check that it loads in a browser. Tools like Mintlify and Yoast SEO can also generate one for you. An llms-full.txt is built the same way, with the full page text in place of links.

### Can you submit llms.txt to ChatGPT or Google?

No. The proposal has no submission or registration step, so the file only works if a tool goes looking for it at /llms.txt. Placing it there won't get your site indexed by any AI engine, and Google has said its search ignores the file.

## Related terms

- **AI crawlers**: Bots run by AI companies that fetch web pages, either to collect training data or to retrieve live sources for an answer.
- **GPTBot**: OpenAI's crawler for collecting model training data. ChatGPT search uses a separate crawler, OAI-SearchBot, so blocking GPTBot in robots.txt doesn't remove you from ChatGPT search results.
- **Server-side rendering (SSR)**: Building a page's HTML on the server before it is sent, so crawlers that don't run JavaScript, including most AI crawlers, can still read the content.
- **Structured data**: Code, usually schema.org JSON-LD, that labels what a page contains, such as a product or an FAQ, in a format machines can parse.
- [Retrieval-augmented generation (RAG)](https://www.searchable.com/glossary/retrieval-augmented-generation): Retrieval-augmented generation (RAG) is a technique where an AI model first searches a set of documents for passages that match a question, then writes its answer from those passages. Because it looks things up at answer time, the model can use up-to-date information and point to where it came from.
- [XML sitemap](https://www.searchable.com/blog/what-is-an-xml-sitemap): A file that lists the URLs on a site you want search engines to crawl, often with the date each was last changed.

---

[AI Search Glossary](https://www.searchable.com/glossary) | [Searchable Homepage](https://www.searchable.com)