The GEO Directory · 19 November 2026 · Published

Who actually reads your llms.txt?

Vendor blogs call llms.txt the sitemap for AI and the future of search visibility. Four independent measurements of server logs and crawl data found retrieval bots almost never fetching it in the traffic they studied. Both can be true at once, and the difference decides whether the file deserves an afternoon of your time.

ONE FILE / FOUR MEASUREMENTS

The same question,
measured four ways.

01Ahrefs137,210 sites, May 2026 logs. 97% of published llms.txt files received zero requests in the month. No AI bot requested a file that did not exist.Source 4 in the list below
02SE Ranking~300,000 domains vs LLM citations. 10.13% adoption and no measurable link to how often a domain is cited in LLM answers.Source 5 in the list below
03Weekerp68,759 verified AI bot requests, two sites. Zero requests to llms.txt. The only fetchers were audit tools. Robots.txt and sitemaps were fetched normally.Source 6 in the list below
04Common Crawl584,107 files, content analysis. 68% look plugin-generated, 22.56% contain no links at all, and about 10% of live llms.txt paths actually serve a robots.txt.Source 7 in the list below
Schematic summary of four studies read on 19 November 2026. Populations, windows and methods differ, so the numbers are not interchangeable. Section 02 gives each study its scope and caveats before any conclusion.

Since September 2024, a simple proposal has circulated: put a Markdown file at /llms.txt that tells a language model what your site is and which pages matter. [1] [2] Just over two years later the file is everywhere and the readers are not. This guide lays out what the format actually is, what four independent measurements found in real traffic, why Google documents the file in one product and dismisses it in another, and how to decide from your own logs instead of from vendor enthusiasm.

01 / The file

A Markdown reading list, not a control

The specification is deliberately small. [1] An llms.txt file is Markdown at your site root. It starts with one required H1 naming the site, usually followed by a blockquote summary and a few freeform paragraphs. Then come H2 sections, each holding a Markdown list of links with optional descriptions, and an Optional section for links an agent may skip when context is short. The August 2026 revision added path-scoped files for large properties, machine-readable .md versions of pages advertised with rel="alternate", and two ways to point agents at the file: a rel="describedby" link in the HTML head or an HTTP Link header. Generators now ship in platforms such as Wix, Mintlify, GitBook and Yoast.

Three things the file is not. It is not robots.txt: nothing in the format grants or denies access, and writing policy language in it controls nothing. It is not a sitemap: no measured AI bot goes looking for it the way crawlers fetch sitemaps. And it is not an AI-search input, at least according to Google Search documentation, which states that no new machine-readable files, AI text files or markup are needed to appear in AI Overviews or AI Mode. [3]

02 / The logs

Four measurements, one direction

Between June 2026 and September 2026, four teams looked for the file in real traffic data. Their populations differ and none is the whole web, but they point the same way. Read the scope column before quoting any number.

Four independent measurements of llms.txt adoption and traffic, with scope, findings and reading for each.
StudyScope and windowWhat it foundRead it as
Ahrefs log study137,210 sites using Ahrefs Web Analytics; requests during May 2026. Population skews technical.28% publish the file; 97% of those files saw zero requests. Of the files that were read, 96% of requests came from bots; AI bots combined were 19.5% of requests, with GPTBot the top single client and the Claude-Code agent out-fetching every retrieval and assistant bot. Zero AI requests for nonexistent files. Lighthouse accounted for about 1 in 1,000 fetches. [4]In these logs, retrieval bots barely fetched the file, and nothing went looking for one. Agents are the most plausible real readers. A logged fetch still does not prove the content was used.
SE Ranking citation analysis~300,000 domains; citation frequency across major LLM answers, reported November 2025.Adoption was 10.13%. Correlation tests and an XGBoost model found no link between having the file and being cited; removing the feature improved the model’s accuracy. [5]No link to citations in this dataset, which is not proof there is none. Do not sell the file internally as a visibility lever.
Weekerp log study68,759 UA- and ASN-verified AI bot requests across two production sites over roughly one month, published August 2026.Zero AI bot requests to llms.txt, while robots.txt (1,938 requests) and sitemaps (1,224) were fetched in the same window. The 61 requests the file did get came from audit tools, 53 of them Chrome-Lighthouse. [6]In this window, AI crawlers fetched the files that govern crawling. The llms.txt fetchers were mostly tools checking whether it exists. Two sites is a small sample.
Common Crawl content analysis584,107 llms.txt files found in Common Crawl data, reported September 2026.68% appear plugin- or template-generated (Wix alone accounts for 41%); 22.56% contain no links; 49% carry the full suggested structure; 6% carry policy language the format cannot enforce; 136,578 robots.txt files are served at llms.txt paths; four files contained genuine prompt-injection attempts. [7]Most files exist because a platform made them, and a large minority are not the format at all. Crawl samples have their own biases; treat counts as indicative, not exact.
  1. Ahrefs log study

    Scope and window
    137,210 sites using Ahrefs Web Analytics; requests during May 2026. Population skews technical.
    What it found
    28% publish the file; 97% of those files saw zero requests. Of the files that were read, 96% of requests came from bots; AI bots combined were 19.5% of requests, with GPTBot the top single client and the Claude-Code agent out-fetching every retrieval and assistant bot. Zero AI requests for nonexistent files. Lighthouse accounted for about 1 in 1,000 fetches.
    Read it as
    In these logs, retrieval bots barely fetched the file, and nothing went looking for one. Agents are the most plausible real readers. A logged fetch still does not prove the content was used.
  2. SE Ranking citation analysis

    Scope and window
    ~300,000 domains; citation frequency across major LLM answers, reported November 2025.
    What it found
    Adoption was 10.13%. Correlation tests and an XGBoost model found no link between having the file and being cited; removing the feature improved the model’s accuracy.
    Read it as
    No link to citations in this dataset, which is not proof there is none. Do not sell the file internally as a visibility lever.
  3. Weekerp log study

    Scope and window
    68,759 UA- and ASN-verified AI bot requests across two production sites over roughly one month, published August 2026.
    What it found
    Zero AI bot requests to llms.txt, while robots.txt (1,938 requests) and sitemaps (1,224) were fetched in the same window. The 61 requests the file did get came from audit tools, 53 of them Chrome-Lighthouse.
    Read it as
    In this window, AI crawlers fetched the files that govern crawling. The llms.txt fetchers were mostly tools checking whether it exists. Two sites is a small sample.
  4. Common Crawl content analysis

    Scope and window
    584,107 llms.txt files found in Common Crawl data, reported September 2026.
    What it found
    68% appear plugin- or template-generated (Wix alone accounts for 41%); 22.56% contain no links; 49% carry the full suggested structure; 6% carry policy language the format cannot enforce; 136,578 robots.txt files are served at llms.txt paths; four files contained genuine prompt-injection attempts.
    Read it as
    Most files exist because a platform made them, and a large minority are not the format at all. Crawl samples have their own biases; treat counts as indicative, not exact.

Four studies read on 19 November 2026, quoted with their own populations and windows. Different samples, so no pooled averages here.

A fifth data point comes from a vendor experiment. OtterlyAI watched one site’s logs for ninety days and counted 84 requests to /llms.txt out of more than 62,100 AI bot visits, about 0.1%, with the file performing roughly three times worse than the site’s average page and no change in crawl patterns. Otterly’s own conclusion frames the file as integration infrastructure rather than a GEO lever, and it removed the llms.txt check from its audit product. OtterlyAI is listed in this catalog; it is cited here for its published numbers, not as a recommendation. [8]

One caution applies to every row. A logged GET proves a client requested the file, not that a model read it, trusted it or acted on it. The Ahrefs authors make the same point about their own data. [4] When someone shows you fetch counts as proof the file works, that gap is where the argument leaks.

03 / The contradiction

Google documents it in one product and dismisses it in another

Google Search’s AI features documentation, last updated in December 2025, is explicit that AI Overviews and AI Mode need no new machine-readable files, and that the controls remain the existing ones: robots.txt, nosnippet, preview controls. [3] Google’s John Mueller, quoted in the Ahrefs study, called the file a temporary crutch and said it was not done for search. [4] In the same year, Chrome shipped a Lighthouse audit for agentic browsing that checks whether a site serves llms.txt, reasoning that without it agents may spend more time crawling to understand the site’s structure. A missing file is marked Not Applicable rather than failed; a server error is flagged. [9]

Google Search, documentation updated 10 December 2025

No AI text files neededAI features run on existing Search systems and signals. Robots.txt and snippet controls remain the levers. [3]

Chrome Lighthouse, agentic browsing audit

Checks for the fileA 404 is Not Applicable. Read it as an agent-readiness checkbox, not a Search ranking signal. [9]

The practical reconciliation is boring. Lighthouse audits qualities of a page; Search ranks pages. The Weekerp logs show what that looks like in traffic: the one client reliably fetching llms.txt was Chrome-Lighthouse itself, 53 of 61 requests. [6] If your Lighthouse report wants the file, that is a reason to let your platform generate one. It is not evidence that Search uses it.

04 / The actual readers

Agents you invite, in the logs studied

One audience does show up in the data. In the Ahrefs logs the Claude-Code coding agent out-fetched every retrieval and assistant bot, second only to GPTBot overall, and agents as a class accounted for about a tenth of all requests to the file. [4] The specification site lists OpenAI, Anthropic and Google’s Gemini API documentation among the properties publishing their own llms.txt files. [1] In the logs studied, the pattern is consistent: documentation that agents are pointed at gets read; files waiting to be discovered do not, because no AI bot in those logs probed for the file speculatively. [4]

OpenAI

Publishes an llms.txt for its own developer documentation, per the specification site. [1]

Anthropic

Publishes an llms.txt for its docs, and its Claude-Code agent was the second most active single client in the Ahrefs logs. [1]

Google

Publishes an llms.txt for the Gemini API docs, while its Search documentation says no AI text file is needed for AI features. [1]

As listed by the specification site, read on 19 November 2026. Publishing a file for one’s own documentation is a documentation decision, not proof the format drives citations. Logos belong to their respective companies.

That discovery problem cuts both ways. In the traffic measured so far, nothing finds the file by accident. The specification gives you the mechanisms: a rel="describedby" link in your HTML head, an HTTP Link header, or an ordinary visible link. [1] If your customers work through coding agents, telling those agents the file exists, in your docs quickstart or integration guide, is worth more than the file itself.

05 / The failure modes

Most files fail before an agent ever sees them

The Common Crawl analysis is the unflattering mirror. [7] More than two thirds of sampled files look like plugin output, and over a fifth contain no links at all, which makes them placeholders, not indexes. Around 136,000 robots.txt files are served at llms.txt paths, roughly one in ten live URLs checked: servers configured to answer the path with the wrong file. Six percent of files carry policy language the format cannot enforce, and of the 1,570 files that name a specific crawler, 32 deny CCBot without blocking it in robots.txt, which is the only place such a block works. If your concern is crawler access, the enforcement point is robots.txt, and our access audit covers how to check it. [7]

The rarer failure is hostile. The same sample found four files containing genuine prompt-injection attempts, text crafted to steer an agent that reads it. [7] The Ahrefs team logged a crawler whose user agent belongs to a prompt-injection survey probing these files. [4] The lesson is not fear; it is hygiene. The file is content an agent will trust, so treat it like code: review what a generator writes, keep it in version control, and never paste text into it that you would not want a model to follow.

06 / Decide

Three cases, then a 30-day check in your own logs

The evidence supports a decision tree, not a verdict. The file is cheap, harmless when maintained, and unread in the studied logs by the retrieval bots that vendor blogs say it serves. Whether it earns its place depends on who integrates with your product.

Marketing site, no agent workflow

Skip it

Your buyers do not use coding agents against your documentation, and no platform generates the file for you. In the studies above, skipping cost nothing measurable: none found an AI-search effect in its sample, which is not proof there is none. Revisit when your logs or your customers say otherwise.

Wix, Mintlify, GitBook, Yoast or similar

Keep what the platform made

Many platforms now generate the file automatically; 68% of files in the Common Crawl sample look templated. Check that yours lists real links (22.56% of sampled files have none), fix template leftovers, and keep it as documentation hygiene, not as a growth channel.

Docs, API or integration product with agent users

Write a lean one by hand

If your customers integrate through coding agents, a curated file is a cheap, honest index: ten to twenty current links to your docs index, quickstart, API reference, pricing and changelog. Maintain it like code, in version control, reviewed when docs change.

If you publish one, make it verifiable. A lean file for a hypothetical analytics product looks like this: one H1, a one-line summary, and a dozen current links. The example is invented for teaching, with placeholder URLs.

# Northwind Metrics

> Product analytics for B2B SaaS. This file lists the pages an agent should read first.

## Docs
- [Documentation index](https://docs.example.com/): concepts, guides, glossary
- [Quickstart](https://docs.example.com/quickstart): first event in 15 minutes
- [API reference](https://docs.example.com/api): endpoints, auth, rate limits

## Product
- [Pricing](https://example.com/pricing): plans, limits, billing terms
- [Changelog](https://example.com/changelog): shipped changes by date
- [Security](https://example.com/security): data handling and subprocessors

## Optional
- [Migration guide](https://docs.example.com/migration): importing historical data

Hypothetical teaching example. Placeholder URLs, invented product, no claim that any agent has read it.

Then run the only check that answers the question for your site. One, publish the file at the root. Two, reference it, through rel="describedby", a Link header or a visible footer link, because nothing in the studied logs probed for it on its own. Three, after thirty days, count verified requests in your server logs by client type: coding agents, AI assistants, audit tools, humans. Four, decide. Agent requests mean the file has an audience, so maintain it. Only audit tools means it is checklist hygiene, so keep it if it costs nothing and drop it if it rots. Five, whatever the logs say, do not grade the file on AI citations: the studies cited here measured none, so that target would be unfalsifiable either way. [5] This site publishes its own llms.txt as a curated index of the catalog; the same log check applies to it, and if the day comes when only audit tools fetch it, this paragraph will be the reason it stays lean.

  1. Publish /llms.txt at the site root: one H1, a one-line summary, ten to twenty current links.
  2. Reference it from your HTML head (rel="describedby"), an HTTP Link header, or a visible link, so agents can find it.
  3. Wait thirty days, then count requests to the file in server logs, grouped by verified client type.
  4. Coding-agent or assistant traffic: keep and maintain. Only audit tools: keep if free, remove if it goes stale.
  5. Score success by whether agents use your docs, never by AI citation counts, which the studies cited here do not link to the file.

Method and sources

What is established, and what is not

This is a documentation-and-logs synthesis, not a controlled experiment and not a prediction. Four independent measurements, one analytics customer base, one domain-level citation analysis, two production sites and one Common Crawl sample, plus a ninety-day vendor experiment, agree in direction: in the samples studied, retrieval bots rarely fetched llms.txt, nothing probed for a missing one, and no citation lift was measured. None of them is the whole web, all of them carry the biases of their samples, and a logged fetch never proves a model used the content. The specification itself is younger than the hype around it and may keep evolving; the v2 mechanics described here were current on 19 November 2026.

Disclosures. Ahrefs, OtterlyAI and Semrush appear in this catalog; they are cited for their published data and documentation only, and nothing here recommends or ranks any tool. Google, OpenAI and Anthropic are mentioned for their public documentation and their own published files. No search-volume claims are made for llms.txt topics: suggestion checks from one English-language client on 19 November 2026 showed sustained how-to interest, and suggestions are not volumes.

Sources were read on 19 November 2026. Specifications, platform behavior and bot populations change; check the current pages before repeating any claim here.

  1. llms.txt: The /llms.txt file, v2 specification

    The format proposal itself, first published 3 September 2024 and updated to v2 on 10 August 2026. Read 19 November 2026.

  2. Answer AI: /llms.txt, the original proposal

    Jeremy Howard’s 3 September 2024 post proposing the convention: help LLMs use a website at inference time. Read 19 November 2026.

  3. Google Search Central: AI features and your website

    Eligibility and controls for AI Overviews and AI Mode, last updated 10 December 2025. States no new machine-readable files are needed. Read 19 November 2026.

  4. Ahrefs: We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read

    15 June 2026 study of 137,210 sites using Ahrefs Web Analytics, a population that skews technical. Server-log requests for May 2026. Read 19 November 2026.

  5. Search Engine Journal: LLMs.txt Shows No Clear Effect On AI Citations, Based On 300k Domains

    20 November 2025 write-up of an SE Ranking analysis of roughly 300,000 domains and their LLM citation frequency. Read 19 November 2026.

  6. Weekerp: Does llms.txt Work? We Analyzed 68,759 AI Bot Requests

    16 August 2026 analysis of roughly one month of verified AI bot traffic across two production sites. Small sample, independently run. Read 19 November 2026.

  7. Search Engine Journal: Analysis of 584k llms.txt files in Common Crawl

    8 September 2026 write-up of Malte Ostendorff’s Common Crawl content analysis: what the files actually contain. Crawl samples are not the whole web. Read 19 November 2026.

  8. OtterlyAI: The llms.txt experiment

    Ninety-day single-site log watch, updated 5 February 2026. OtterlyAI is listed in this catalog; cited here for its published data only. Read 19 November 2026.

  9. Chrome for Developers: Lighthouse agentic browsing audits, llms.txt

    Documentation for the Lighthouse audit that checks for the file. A 404 is Not Applicable; a server error is flagged. Read 19 November 2026.

The GEO Directory. The only llms.txt question your logs cannot answer is why you would write one before checking them.