
Does an llms.txt file get your site into AI answers?
Unity Horizon7 min read
No. Google's guide says Search ignores it, and 97% of the llms.txt files in Ahrefs' May 2026 log study of 137,210 domains got zero requests.
No. Google's AI optimization guide says Search ignores the file, and in a log study of 137,210 domains published by Ahrefs in June 2026, 97% of the llms.txt files it found received no requests in a month. We fetched /llms.txt from eight hostnames on 11 November 2026, including those of three companies that run answer engines. Their developer documentation sites serve the file. The sites a customer lands on return 404.
What does Google say about llms.txt?
That it is ignored. The mythbusting section of Google's AI optimization guide is titled "LLMS.txt files and other 'special' markup" and reads: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search", because "Google Search itself doesn't use them." The note under it goes further. Keeping one "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." The same guide dismisses two other items off the standard GEO checklist: "There's no requirement to break your content into tiny pieces for AI to better understand it", and "Structured data isn't required for generative AI search, and there's no special schema.org markup you need to add." Google's separate AI features page repeats the first half and names robots.txt directives for Googlebot as the access control that does work. Checked 11 November 2026.
Then why does Chrome grade you on the file?
The browser team is building for a different reader. Lighthouse now ships an experimental Agentic Browsing category that "evaluates how well your site is constructed for machine interaction", scored as a ratio of checks passed rather than out of 100. Its Stability and Discoverability group sits next to WebMCP integration and an agent-centric accessibility subset, and it contains an llms.txt audit. That audit "flags the pages if a server error occurs when attempting to retrieve the llms.txt file", while a 404 is marked Not Applicable, "as providing the file is optional at the moment". Running the category needs Chrome 150 or later, and the WebMCP audits need an origin trial registration.
So one company publishes two positions, and both are current. Search ignores the file. Chrome measures it.
Who requests an llms.txt file?
Tools, mostly. Ahrefs checked the root of 137,210 domains in its own analytics product for an llms.txt returning HTTP 200, screened out the soft 404s, then read every request to those paths in its bot analytics for May 2026 and published the result on 15 June 2026. Of those domains, 28% published a file, about 38,360 of them. Then the number that matters. 97% of those files received no requests in the month. The 3% that did, roughly 1,100 domains, drew about 22,000 requests between them, and 96% of the 22,000 came from bots rather than people.
The breakdown is where the GEO case dissolves. SEO audit tools 21.7%, general web crawlers 13.1%, tech profiling tools 11.6%, AI training crawlers 5.3%, AI assistants 2.5%, AI retrieval bots 1.1%. The top named AI bot was GPTBot at 4.51%, which is the training crawler. OAI-SearchBot, the token that decides whether ChatGPT can cite you, accounted for 0.74%. ClaudeBot 0.8%. Slackbot fetched llms.txt more often than PerplexityBot did. Chrome Lighthouse made 22 requests in the whole dataset, about one in a thousand. And requests to files that did not exist came 98% from humans, with zero from AI bots, so nothing is out there hunting for the file speculatively.
Read the caveats, because every one of them cuts the figure down and none of it up. Ahrefs sells the analytics that produced this, its customers skew technical, and it calls 28% an upper bound for the web. It also states that a fetch is not a read, which makes every figure above a ceiling on consumption. It measured the index file only.
One name in that data explains the rest. The second most active named AI bot fetching llms.txt was Claude-Code, a coding agent reading documentation.
Do the engines document reading your llms.txt?
Neither of the two that publish crawler pages does. OpenAI's crawler documentation gives each of its four tokens a job: OAI-SearchBot "is for search" and surfaces sites in ChatGPT's search features, GPTBot crawls "content that may be used in training our generative AI foundation models", OAI-AdsBot checks ad landing pages, ChatGPT-User fetches a page when a person asks for it. The only mention of llms.txt on the page is a note at the top telling the reader where OpenAI's own documentation index lives. Perplexity's page has the same shape. PerplexityBot "is designed to surface and link websites in search results on Perplexity", webmasters "can use the following robots.txt tags to manage how their sites and content interact with Perplexity", and the header offers "the complete documentation index at: https://docs.perplexity.ai/llms.txt".
Both companies publish an llms.txt. Both document robots.txt. When we read the robots.txt of 24 domains for Does blocking GPTBot keep your site out of ChatGPT's answers?, twelve of the fourteen software company sites in that sample named no AI crawler at all. The file that governs citation was sitting on whatever default shipped with the CMS. The checklist was pointing at a different file.
Who publishes llms.txt, and on which hostname?
We sent a plain HTTPS GET to eight hostnames on 11 November 2026, with no browser engine behind it, and recorded what came back.
| Hostname | /llms.txt |
What came back |
|---|---|---|
| platform.claude.com | 200 | "Anthropic Developer Documentation", roughly 750 links |
| docs.perplexity.ai | 200 | "Perplexity", roughly 190 links |
| developers.openai.com | 200 | "OpenAI Developers", roughly 40 links |
| www.anthropic.com | 404 | nothing |
| www.perplexity.ai | 404 | nothing |
| platform.openai.com | 404 | nothing, the docs moved to developers.openai.com |
| ai.google.dev | 404 | nothing, and the Gemini API subpath redirects to aistudio.google.com |
| developers.google.com/search | 404 | nothing |
openai.com returned 403 to our fetcher, so we cannot report that one either way. The pattern across the rest is hard to misread. Three companies that run answer engines serve an llms.txt where their API reference lives, and none of them serves one on the domain a buyer types. The file is a table of contents for an agent that had already decided to read the documentation. It is not a way in.
Which half of the llms.txt proposal works?
The Markdown mirror is the half with something in it. llmstxt.org, published by Jeremy Howard on 3 September 2024 and now at v2, last modified 10 August 2026, asks for two things. One is the index: an H1, a summary, H2 sections of links. The other is "clean markdown versions of pages at the same URL", with .md appended, discoverable through a rel="alternate" type="text/markdown" link. We appended .md to two documentation URLs on 11 November 2026. Both returned Markdown with a four-line YAML header and nothing else wrapped around the content. OpenAI documents the same convention for its own pages.
That half costs an engine nothing to use and saves it the HTML. The Ahrefs study did not measure it, by its own statement, so nobody has published a number on the mirrors yet. The index file makes a claim about your site. The mirror is a cheaper page to read.
Who does the llms.txt recommendation suit?
A company whose product is documentation. The index saves a coding agent two or three crawl steps, that agent arrived with a task, and Claude-Code's position in the Ahrefs data says the traffic is real. It suits anyone selling an AI visibility checklist even better, because the file takes an afternoon, it is verifiable on delivery, it looks substantial in a report, and no measurement exists that could ever show it failed.
It costs the five-person software firm that spent that afternoon generating a Markdown index while its robots.txt still disallowed the one crawler that decides whether ChatGPT can name it. The bill is the afternoon, and that firm had one of those this month.
What should you check before Monday?
Open /robots.txt first and read it for the retrieval tokens rather than the training ones. Then decide about the index on the basis of who your readers are: serve one where your developer documentation lives, and serve .md next to each page while you are there. If an llms.txt is generated at build time, keep it, the cost is a route. Delete it if a person maintains it by hand, because a stale index of your own site is worse than no index.
Our own site publishes /llms.txt, generated at build time with about 47 links, and /llms-full.txt with the text of every post in one file. We do not serve a .md mirror of individual pages, which on this evidence is the half worth having. We also cannot tell you whether anything read either file. We do not keep the crawl logs that would show it, and the study above is one vendor's panel over one month, which is thinner evidence than the conclusion usually gets quoted with. Our AI visibility check reads a site and asks the engines what they say about it.
Which crawler fetched your llms.txt last month, and if you cannot answer that from a log, what is the file doing?
Common questions
Does llms.txt help my site appear in ChatGPT, Google AI Mode or Perplexity?
No engine has documented reading it. Google states in its AI optimization guide that Google Search ignores llms.txt files. OpenAI and Perplexity document robots.txt tokens as the control over whether their engines can cite a page, and neither crawler page says anything about reading an llms.txt on your domain.
Does Google use llms.txt?
No. Google's AI optimization guide says you do not need to create machine readable files, AI text files, markup or Markdown to appear in Google Search, because Google Search itself does not use them. The same page adds that maintaining one will neither harm nor help visibility or rankings. Chrome's Lighthouse is the exception inside Google: its experimental Agentic Browsing category includes an llms.txt audit.
Who requests llms.txt files?
Mostly tools, not engines. In Ahrefs' May 2026 log study of 137,210 domains, 97% of the llms.txt files it found received no requests at all. Of the requests that did arrive, SEO audit tools were the largest category at 21.7% and AI retrieval bots the smallest at 1.1%. Slackbot fetched llms.txt more often than PerplexityBot.
Should I publish an llms.txt file at all?
Publish one if your product is developer documentation, because coding agents fetch documentation indexes and Claude-Code was the second most active named AI bot in the Ahrefs data. For a marketing site, the measured return is close to zero. The half of the proposal worth implementing either way is the Markdown mirror: the same page served as clean Markdown at the same URL with .md appended.


