Is llms.txt worth it? Skeptics are right it won't rank you, but that misses its real job: agent navigation and coding tools. An honest, by-site-type decision.
Quick answer
Is llms.txt worth it? For most sites with real content, yes — but only if you grade it correctly. The skeptics are right about one thing: llms.txt will not get you discovered, ranked, or cited (Ahrefs found ~97% of files got zero AI requests; Google says it is not a ranking signal). Where they go wrong is treating that as the whole verdict. The file's actual job is on-site agent navigation, feeding coding agents like Cursor and Claude Code, and serving as the read layer for a WebMCP handoff — none of which is search ranking. Ship it if you run a docs site, SaaS, developer tool, ecommerce store that agents might browse, or an on-site AI assistant. Skip it, or deprioritize it, only for a tiny brochure site with no agent or developer audience. Generation is nearly free, so the downside is tiny either way.
This is the question I get every time the "llms.txt is dead" takes make the rounds, and they make the rounds often. A major SEO vendor publishes a stat showing almost nobody fetches the file, the headline gets shared, and a few hundred site owners quietly decide not to bother. The stat is real. The conclusion most people draw from it is wrong, because they are grading llms.txt against a job it was never built to do.
So here is the honest version, with the case against laid out in full, the case for explained where it actually applies, the evidence on both sides, and a decision by site type. The goal is not to talk you into a file. It is to tell you whether your specific site benefits, because for some sites the honest answer is "don't bother yet."
The skeptics have one genuinely strong point: llms.txt does not get you discovered, ranked, or cited. If you publish the file expecting AI search traffic or better placement in ChatGPT answers, you will be disappointed, and the data backs that up.
Ahrefs studied 137,000 domains with an llms.txt file and found roughly 97% of them received zero AI bot requests to the file. ChatGPT and Perplexity together accounted for about 1% of the requests they did measure. That is not a hype number you can wave away — it is the strongest single piece of evidence in the whole debate, and any honest answer has to start with it. We unpack exactly what that study measured (and its blind spots) in the full breakdown of the 97% stat.
Google's John Mueller reinforced the ranking half of the critique. On Search Off the Record he said llms.txt cannot help AI systems differentiate sites during discovery — which is accurate. Google separately called the file "completely fine," which means no penalty, not a benefit. There is no confirmed path from llms.txt to rankings, AI Overviews, or crawl budget. The full nuance is in Google's llms.txt guidance and why Mueller's "can't differentiate" framing is correct.
And it is not just one skeptic. As of June 2026, Semrush has publicly argued llms.txt is ineffective, Peec AI calls it "a distraction without any upside," and Ahrefs' own blog is openly skeptical. When three serious vendors land in the same place, you should take the critique seriously rather than dismiss it. They are right, on the merits, about the thing they are actually testing: llms.txt is not an SEO tactic, and judged as one it fails.
Where the skeptics go wrong is the unstated assumption that ranking is the only job worth measuring. llms.txt was never a discovery file. It is a navigation and ingestion file, and it has four real audiences that have nothing to do with search placement.
On-site agent navigation. When an AI agent has already arrived on your domain — directed there by a user, or because it retrieved one of your pages — it needs to find the right content fast. A clean llms.txt map lets it reach your pricing page, your docs, or your booking flow in one lookup instead of crawling three pages and guessing. Mueller's own analogy was a store directory: useless to someone who hasn't walked in, genuinely useful to someone who has.
Coding agents. This is the audience that is already reading the file at scale today. Cursor, Windsurf, Continue, and Claude Code pull llms-full.txt to load your API reference or documentation set before generating code — one clean request instead of scraping 40 HTML pages. If you publish developer docs, a technical blog, or an SDK, your llms.txt has an active audience right now. The catch is that these fetches come through IDE and tooling pipelines, so they barely register in studies counting HTTP requests to the public URL — which is part of why the 97% number looks worse than the file's real usage.
On-site AI assistants. If you run a chatbot or AI search box on your own site, llms.txt (and the markdown behind it) is the clean corpus that assistant reads instead of parsing your rendered HTML. You control the input, so you get better answers and fewer hallucinations from your own tool.
The WebMCP handoff. llms.txt is the "read" layer of the emerging agent stack; WebMCP is the "do" layer. An agent reads your llms.txt to understand what your site offers, then invokes WebMCP tools to actually do something — search products, add to cart, request a quote. Worth being precise here: WebMCP is a draft spec, no browser ships it on by default as of June 2026, and the real invokers today are Perplexity Comet plus some extensions and custom agents. But the read-then-do direction is where this is heading, and llms.txt is the front half of it.
None of those four jobs is search ranking. That is the whole disconnect: the critics measure llms.txt as SEO and find it lacking, while the file quietly does navigation and ingestion work for a different audience.
Rather than re-litigate every study here, the honest read is that both camps are citing real evidence about different questions. Lined up side by side:
Both sets of facts are true at the same time. The mistake is using evidence about one question to answer the other. "Almost nobody fetched the file from a search bot" is a fair reason to stop expecting search traffic. It is not a reason to conclude the file is useless to a coding agent that fetched your llms-full.txt through Cursor yesterday and never showed up in that dataset.
The yes/no depends almost entirely on whether your site has an agent or coding-tool audience. Here is the honest split.
If you land in the "yes" or "optional" bucket and want help choosing a tool, we compared the options in the best llms.txt generators, and the what-is-llms.txt guide covers format and hosting from scratch.
Here is the part that reframes the whole debate. Most of the criticism implicitly assumes llms.txt is expensive — that you are trading real effort or money for uncertain payoff. You usually are not.
Generating the file is close to free. You can hand-write a basic one in under an hour, or auto-generate it from your sitemap in minutes. There is no recurring cost to having the file sit at your domain root, no ranking penalty (Google confirmed that), and no risk beyond the prompt-injection caveat above for auto-generated files. The downside of shipping a well-formed llms.txt is, in practice, close to zero.
So the expected-value math is lopsided. The cost is tiny. The upside — coding agents reading your docs cleanly, agents navigating your site in one lookup, a working read layer for WebMCP later — is real for the right site types and free optionality for the rest. The only genuinely wrong move is shipping it blind and assuming it works. The fix for that is measurement: server-side bot detection tells you which AI crawlers actually fetched your file and which pages they read next, because standard analytics filter bot traffic out by default.
Disclosure: Crawlytics is one of the tools that generates and serves llms.txt, and because it also reads your bot logs it can show you the coverage gap — pages you declared that no bot ever fetched. That closed loop is the honest answer to "is it working." A standalone generator can publish the file but can't tell you whether anything read it, which is exactly the question the 97% stat should make you ask.
Net: for a docs site, SaaS, developer tool, ecommerce store with an agent surface, or any site with an on-site assistant, generate it and measure it. For a tiny brochure site, skip it without guilt. The skeptics are right that llms.txt is not SEO. They are wrong that "not SEO" means "not worth it."
Written by Crawlytics Team. Crawlytics tracks AI bots, generates llms.txt, and powers WebMCP commerce, all from one snippet on any stack. See how it works →
For most sites with real content, yes — but for navigation and agent use, not for ranking. It is worth it if you publish documentation, run a SaaS or developer tool, sell products that AI shopping agents might browse, or run an on-site AI assistant. It is not worth prioritizing for a tiny brochure site with no developer or agent audience. The file costs almost nothing to generate, so the practical question is rarely "should I pay for this" and almost always "does my site have an agent or coding-tool audience that benefits."
It works for what it was designed to do — help an AI agent that is already on your site navigate your content — and does not work for what skeptics test it against, which is discovery and ranking. Ahrefs found about 97% of llms.txt files across 137,000 domains received zero AI bot requests to the file, which is real evidence that it is not a search-traffic driver. Coding agents like Cursor and Claude Code do read llms-full.txt regularly, but they pull it through developer tooling that those request counts mostly miss.
Coding agents read it consistently today; AI search bots read it rarely. Cursor, Windsurf, Continue, and Claude Code pull llms.txt and llms-full.txt to understand a codebase or documentation set, and that is a real, active use case. ChatGPT and Perplexity fetch the file at low rates — together roughly 1% of requests in the Ahrefs dataset. The only way to know whether anything is fetching your file is server-side bot detection, because analytics tools filter bot traffic out by default.
No. That is not its job. Google's John Mueller has said llms.txt cannot help AI systems differentiate sites during discovery, and Google has separately called the file "completely fine" — meaning no penalty, not a ranking benefit. There is no confirmed signal path from llms.txt to Google rankings, AI Overviews coverage, or faster indexing. If your goal is rankings, llms.txt is the wrong lever; links, sitemaps, server-side rendering, and content quality are the right ones.
Documentation sites, SaaS and developer tools, ecommerce sites that AI shopping agents might browse, and any site running an on-site AI assistant or planning a WebMCP integration. These all have an agent or coding-tool audience that benefits from a clean content map. Sites that can deprioritize it: tiny brochure sites, single-page local-service sites, and any site under about ten pages where an agent can infer the structure without help. Because generation is nearly free, "skip it" is a smaller club than the criticism implies.
This page is part of Crawlytics.app. View all pages: llms.txt · llms-full.txt