llms.txt: What It Really Does for AI Visibility
llms.txt is a Markdown file for language models. What Ahrefs data and Google say it achieves, how to create one, and what works better for AI search.
An llms.txt is a Markdown file in your website's root directory that gives language models a curated overview of your most important content, but on current evidence it does almost nothing for your visibility in ChatGPT, Perplexity, or Google's AI Overviews. An Ahrefs analysis of 137,210 domains found that 97% of existing files were not requested a single time in May 2026. Google states in its guide to AI features that Search ignores the file. It does not appear in the crawler documentation of OpenAI, Anthropic, or Perplexity (as of October 2026).
This article explains what the file is and how it is structured, who actually reads it, how to create one in six steps, and which measures work instead.
Key takeaways
- Jeremy Howard (Answer.AI) proposed llms.txt in September 2024, and version 2 has been on llmstxt.org since August 2026. It is not an official web standard.
- Ahrefs analyzed 137,210 domains: 28% have an llms.txt, and 97% of those received zero requests in May 2026.
- Of the requests that did happen, only 1.1% came from AI retrieval bots, the crawlers that feed AI search with live sources.
- Google's guide to AI features in Search (May 2026) is explicit: Google Search does not use llms.txt, and the file neither helps nor hurts there.
- Our decision: llms.txt is a nice-to-have, never a priority, and we do not sell it as a service.
- Server-side rendering, a robots.txt that admits AI crawlers, answers early on the page, and genuine mentions on other websites do more.
The short answer
llms.txt is a table of contents in Markdown that lives at yourdomain.com/llms.txt. It is meant to show language models (LLMs, AI systems like ChatGPT or Claude that read and generate text) what your website is about and which pages matter. It controls nothing: it blocks no sections and grants no permissions.
As of October 2026, it plays practically no role in your visibility in AI answers. According to an Ahrefs study from June 2026, 97% of files go unread, and the crawlers behind AI search barely request them. Google says in its guide to AI features that Search ignores the file. Among AI systems, it is mostly fetched by agents for developers.
Our recommendation: creating one is cheap and harmless, so go ahead if you like. Just do not expect citations in ChatGPT from it. Your time is better spent on server-side rendering, clear page structure, and mentions on other websites. How classic search and AI answers differ is covered in our comparison SEO vs. GEO.
What is llms.txt?
The idea comes from Jeremy Howard, co-founder of Answer.AI and fast.ai, who published it on llmstxt.org on September 3, 2024. The problem it addresses: a normal web page wraps its information in navigation, cookie banners, and JavaScript, and a language model can only process a limited amount of text at once. llms.txt offers a short, tidy overview with links to the details instead.
In August 2026, Howard released version 2. It allows the file in subfolders too, for example for a documentation section, and describes how a page can point to its Markdown version and to the llms.txt that covers it. Howard himself writes that the file is used most heavily for software documentation, where AI agents look up API references while coding. An API is a programming interface that lets one piece of software talk to another.
Worth keeping in mind: llms.txt is an open proposal. robots.txt, by contrast, has been codified since September 2022 as RFC 9309, an official internet standards document. The frequently mentioned llms-full.txt is not part of the specification either. It comes from documentation platforms such as Mintlify and packs the full text of every page into one file.
How is an llms.txt structured?
The file is written in Markdown, a simple text format: a hash sign makes a heading, a hyphen makes a list item. The specification sets a fixed order, and only the first heading is required. A typical file consists of these parts:
- An H1 heading with the name of the website or company, such as `# Example Ltd`. This is the only required element.
- A blockquote with a short summary, starting with `>`, for example `> Manufacturer of packaging machinery based in Bavaria`.
- Optionally a few paragraphs or lists with context, but no further headings.
- Sections with H2 headings like `## Services`, each containing a list of links.
- Every list entry follows the pattern `- Page title: short description`.
- An `## Optional` section for secondary material an agent can skip when context is tight.
Our own file at in-sync.io/llms.txt follows exactly this pattern: brand name as H1, one sentence on positioning, then sections on services, industries, content, and contact. Extra Markdown versions of every page, as the specification suggests, pay off for software documentation but rarely for a company website.
llms.txt vs. robots.txt vs. sitemap.xml: what is the difference?
All three files sit in the root directory, but their jobs have little in common. robots.txt tells crawlers (programs that fetch websites automatically) which areas they may visit. sitemap.xml lists every page that should be in the search index. llms.txt explains content without regulating anything.
llms.txt vs. robots.txt vs. sitemap.xml: what is the difference?
| Attribute | robots.txt | sitemap.xml | llms.txt |
|---|---|---|---|
| Purpose | Allow or block access | List all indexable pages | Curated overview with short descriptions |
| Audience | All crawlers | Search engines | Language models and AI agents |
| Format | Its own text rules | XML | Markdown |
| Status | Standard (RFC 9309, 2022) | Established, supported by Google and Microsoft | Proposal (2024, version 2 from 2026) |
| Controls crawling? | Yes | No, gives hints | No |
| Used by Google Search? | Yes | Yes | No, according to Google |
| Relevance for AI visibility | High: block AI crawlers and you are out | Medium: helps discovery via search indexes | Low on current evidence |
If you check only one of the three files, make it robots.txt. A single wrong line can lock out ChatGPT or Perplexity entirely. How to set up robots.txt and your sitemap properly is covered in our guide to technical SEO.
Who actually reads llms.txt?
The most solid figures come from an Ahrefs study published on June 15, 2026. Ahrefs analyzed request logs for 137,210 domains that had visitors via Ahrefs Web Analytics in May 2026. 28% of them, around 38,000 domains, had a valid llms.txt. Ahrefs itself treats that figure as an upper bound, because its customers are more technical than the average website.
The result is sobering. 97% of these files were not requested once during the whole of May, by bots or by humans. The remaining 1,100 or so domains received about 22,000 requests combined, 96% of them from bots. Here is how all requests broke down:
Who actually reads llms.txt?
| Who requested the file? | Share of all requests | Examples |
|---|---|---|
| SEO audit tools | 21.7% | SiteAuditBot, WebPageTest |
| Unidentified bots | 14.9% | Anonymous scripts |
| General web crawlers | 13.1% | Googlebot, Amazonbot |
| Tech profiling tools | 11.6% | BuiltWith, Dataprovider |
| AI agents and their infrastructure | 10.5% | Claude-Code |
| GEO and AEO scoring tools | 5.8% | Bots that rate AI readiness |
| AI training crawlers | 5.3% | GPTBot, ClaudeBot |
| AI assistants acting for a user | 2.5% | ChatGPT-User, Claude-User |
| AI retrieval bots of AI search | 1.1% | OAI-SearchBot, PerplexityBot |
Retrieval bots, which fetch live sources for ChatGPT search or Perplexity and which almost every guide is betting on, sit at the very bottom with 1.1%. According to Ahrefs, Slack's link preview bot loaded the files more often than PerplexityBot did. And requests from AI bots for llms.txt files that did not exist: zero. No AI system goes looking for the file; it gets fetched only when a link, a directory, or a user instruction points to it.
The real consumers are agents for developers. Claude-Code, Anthropic's coding agent, fetched the files more often than any search or assistant bot. And according to Ahrefs, a request does not mean the content was used.
One detail you should know: the largest research crawler in the dataset calls itself prompt-injection-survey. Prompt injection means slipping hidden instructions to an AI system. Because agents trust the file, Ahrefs advises treating it like code: version changes, restrict write access, include only links and descriptions, and nothing that reads like an instruction.
What do Google, OpenAI, Anthropic, and Perplexity say?
Google is the only major provider with a clear written statement. Its guide to optimizing for generative AI features in Google Search, published in May 2026, lists llms.txt in its mythbusting section. In essence: you do not need to create machine-readable files, AI text files, or Markdown versions to appear in Search or its AI features, because Google Search does not use them. If you maintain an llms.txt for other services, that is fine. It neither helps nor hurts your visibility in Google Search.
John Mueller, Search Advocate at Google, has held this view for a long time. In April 2025 he wrote on Reddit that, as far as he knew, none of the AI services had said they use llms.txt, and that server logs show they do not even check for it. He compared the file to the keywords meta tag, a long-ignored field in which site owners declared what their pages were about. According to Ahrefs, in May 2026 he called it a temporary crutch to save tokens for AI coding tools, tokens being the units in which language models process text. It is not meant for search, he said.
Confusion comes from another part of Google. Since May 2026, the Chrome team's analysis tool Lighthouse (from version 13.3) checks for an llms.txt under Agentic Browsing. The audit is informational only and feeds into no score, and the Chrome documentation explicitly calls the file optional. It targets AI agents in the browser, not search results.
OpenAI, Anthropic, and Perplexity have not committed to using llms.txt for their search. In their official crawler documentation, you control access through robots.txt alone, with their own user agents such as OAI-SearchBot, Claude-SearchBot, or PerplexityBot. At the same time, all three publish an llms.txt for their own developer docs. That fits the Ahrefs finding: the file is a tool for coding agents, not a signal for AI answers. Which AI search draws on which index is covered in our overview of AI search engines.
How do you create an llms.txt?
If you create one anyway, keep the effort small. Here is how:
Pick your pages:
Take the homepage, your service pages, your most important articles, and the contact page. Legal pages belong in the Optional section at most.
Write the file:
In a text editor, create a file with an H1, a short summary in a blockquote, and H2 sections with link lists. Each description is one factual sentence, no marketing and no instructions to the AI.
Save it as llms.txt:
Use exactly this file name and UTF-8 encoding so special characters come through correctly.
Publish it in the root directory:
The file must be reachable at yourdomain.com/llms.txt and served as text, not as an HTML page or download.
Check the response:
Open the address in your browser and confirm the server returns status code 200, meaning it found the file. A 404 means it is missing.
Assign maintenance:
Decide who may edit the file, and update it whenever new services or pages are added. An outdated llms.txt sends agents in the wrong direction.
Many systems do the work for you. According to its help center, Wix automatically generates an llms.txt for websites on a paid plan with a custom domain. The WordPress plugins Yoast SEO and AIOSEO have a built-in generator, and documentation platforms like Mintlify and GitBook ship the file too. There are also free online generators that read your sitemap. Always review the result by hand: auto-generated descriptions tend to be generic, and Ahrefs also recommends checking generated files.
With Next.js or Astro and a headless CMS, a content system that manages content separately from its presentation, the file can be generated automatically from CMS data at publish time.
Should you create an llms.txt?
The decision depends less on your industry than on who reads your content by machine. For most company websites, the file is a footnote.
Yes, if
The file makes sense if AI agents are realistic readers or it costs you nothing:
- You offer software, an API, or a platform with technical documentation that developers use with coding agents.
- Your system generates the file automatically anyway, and you only need to review it once.
- You want to give agents working for you or your customers a fixed entry point, and you actively link to the file for that purpose.
Skip it, if
Move the file to the bottom of your list if any of these applies to you:
- Your goal is to be cited in ChatGPT, Perplexity, or Google's AI Overviews. On current evidence, it has no effect there.
- Basics like server-side rendering, Bing indexing, or a clean robots.txt are still open.
- Nobody on your team can maintain the file over time and protect it against unwanted changes.
What works better for AI visibility?
The levers that actually move things are less new and take more work. In return, they also pay off in classic Google Search, which llms.txt does not, according to Google.
- Server-side rendering: Your pages should be built on the server as finished HTML. An analysis by Vercel and MERJ from December 2024 found that GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot do not execute JavaScript. Content assembled only in the browser stays invisible to them.
- robots.txt access: Admit crawlers such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot. OpenAI's help center names this as a requirement for appearing in ChatGPT search.
- Bing indexing: According to OpenAI's help center, ChatGPT search sometimes sends queries to external search providers and refers to Microsoft in that context. So register your website in Bing Webmaster Tools; details are in our article on LLM SEO.
- Answers early on the page: Answer the core question in the first sentences and use real questions as subheadings. That way a system finds the quotable passage quickly.
- Mentions on other websites: According to Google's guide, its AI features also show what is being said about products and providers across the web. Genuine mentions in trade media, directories, and reviews count; bought ones do not.
- Measure instead of guessing: Check regularly whether and where you appear in AI answers. How to do that is covered under measuring AI visibility.
How these levers add up to a strategy is explained in our primer on Generative Engine Optimization. For ChatGPT specifically, we collected the steps in getting into ChatGPT answers. What Google's AI Mode does differently is covered in our article on Google AI Mode.
Our take
We created an llms.txt for in-sync.io ourselves. The effort was small, and it does no harm. We give it no more weight than that. In client projects it is a nice-to-have on the checklist, never a priority, and we do not sell it as a service of its own.
Anyone selling you llms.txt as a GEO lever is selling hope. AI search crawlers barely ask for it, Google Search ignores it, and no major AI provider has committed to using it for answers. If an AI visibility offer starts with llms.txt, ask what comes next.
When we moved in-sync.io from Wix to Next.js with Sanity in 2026, the noticeable difference for AI systems came from elsewhere. Every page is built into readable HTML on the server, robots.txt admits all crawlers, and our articles carry FAQPage schema, structured data that marks up questions and answers in machine-readable form. How such a move works is described in migrating from Wix to Next.js. These are the foundations we put in place as a GEO agency and in ongoing SEO support. Should a major provider officially start using the file, we will update this assessment.
Frequently asked questions
llms.txt is a Markdown text file in a website's root directory that gives language models a curated overview of the most important content, with links. Jeremy Howard of Answer.AI proposed it in September 2024. It is not an official web standard and, unlike robots.txt, it does not regulate access in any way.
No. Google states in its May 2026 guide to AI features that Google Search ignores llms.txt files. So the file neither helps nor hurts your visibility there. Googlebot does fetch it occasionally, but treats it like any other file it finds on a website, with no special meaning attached.
There is no evidence that it does. OpenAI does not mention the file in its crawler documentation and controls access through robots.txt. In the Ahrefs study, only 1.1% of requests came from retrieval bots such as OAI-SearchBot and PerplexityBot. GPTBot, OpenAI's training crawler, fetches the files more often, but that says nothing about citations in answers.
In a text editor, write a Markdown file with your company name as H1, a short summary in a blockquote, and H2 sections with link lists. Save it as llms.txt in UTF-8 and upload it to the root directory. Wix, Yoast SEO, and AIOSEO generate the file automatically, but you should still review the result.
llms.txt is a table of contents with links and short descriptions, while llms-full.txt contains the full text of all pages in one file. Only llms.txt is part of the specification on llmstxt.org; platforms like Mintlify established the full-text variant. For company websites it is hardly worth it, for large software documentation more so.
A small one, as long as you maintain it. AI agents trust the file, and in its study Ahrefs found a research crawler called prompt-injection-survey that examines it specifically. So include only links and factual descriptions, link only to your own content, and limit who can edit the file.
Next step
If you want to know which measures will actually make your website more visible in AI answers, we will review it with you.
