llms.txt is a proposed Markdown file at the root of a website that lists its key pages for language models, according to llmstxt.org. Google's AI optimization guide, updated July 10, 2026, says Google Search does not use llms.txt or other AI text files, and creating one will neither harm nor help visibility or rankings in Google Search. This guide covers the file's definition and contents, its readers and Google's position, the data, its difference from robots.txt and sitemap.xml, and the decision on priority.
What is llms.txt?
llms.txt is a proposed Markdown file, placed at the root of a website, that gives language models a curated list of the site's key pages, according to the llmstxt.org proposal.
The file is written for language models to read, not as a crawl rule: the llmstxt.org text, as we read it on November 14, 2026, lists no allow or disallow element.
What does llms.txt stand for? The "llms" in the name refers to LLMs, large language models.
Two questions refine the definition: what the file is used for, and who proposed it.
What is llms.txt used for?
llms.txt is used for giving an agent information about a topic on demand, when the agent needs it while assisting a user, according to llmstxt.org.
The file sits at a known path, so an agent can fetch and read it, and the proposal says agents are "expected" to view or search it and then follow the links. That describes the author's intent, not the measured behavior of any engine. The origin of the proposal has an author and a date.
Who proposed llms.txt, and when?
Jeremy Howard proposed llms.txt on September 3, 2024, according to llmstxt.org. The llmstxt.org page, now version 2, shows August 10, 2026 as its modified date.
Is llms.txt an official standard? llmstxt.org presents llms.txt as a proposal and an informal overview open for community input on GitHub. In the ten pages we read on November 14, 2026, none names a standards body, in our reading. Chrome for Developers calls the file an "emerging convention", and Mintlify's documentation calls it "an industry standard". Vu du web, a French-language site, presents the file as an access control in an article dated February 10, 2025, whose aim is to "définir quelles pages de votre site ces IA peuvent consulter, indexer ou utiliser" (define which pages of your site these AIs may consult, index or use). In our reading, that role diverges from the one llmstxt.org gives the file. What the file contains comes next.
What does an llms.txt file contain?
An llms.txt file contains a title, an optional summary and sections of annotated links, in a fixed order that the specification defines, according to llmstxt.org.
Five parts make up the file, a worked example shows them in a small file, the companion llms-full.txt has its own job, and four steps write and place the file.
What format does an llms.txt file use?
An llms.txt file uses the Markdown format, with one required element, an H1 title, followed by optional parts in a fixed order, according to the llmstxt.org specification. The five parts appear in this order, and only the first is required.
- H1 title: an llms.txt file opens with an H1 that names the project or site, the only required part.
- Blockquote summary: an llms.txt file can continue with a short summary in a blockquote.
- Free Markdown sections: an llms.txt file can add paragraphs or lists without headings, with more detail about the project.
- H2 file lists: an llms.txt file groups its links in H2 sections, each item written as
[name](url)with an optional note after a colon. - Optional section: an llms.txt file can end with a section named Optional, which by the specification's convention holds links an agent can skip when a shorter context is needed.
The specification lists an optional byte order mark (BOM) ahead of the title. A worked example shows the five parts in a small file.
What does an llms.txt example look like?
An llms.txt example looks like a title, a short summary, two link lists and a last Optional section, for a small agency site.
# Example Studio
> Example Studio is a small agency that publishes guides on search and AI visibility.
## Guides
- [What llms.txt is](https://example.com/guides/llms-txt/): the file in plain words
## Services
- [SEO audit](https://example.com/services/audit/): scope and deliverables
## Optional
- [About](https://example.com/about/): team and methodIllustration built from the specification, not a measured file.
In this llms.txt example, the title names the site, the summary gives its context in one line, the Guides and Services lists point to pages with one note each, and the Optional list holds a page an agent can skip. A companion file, llms-full.txt, has a different job.
How is llms-full.txt different from llms.txt?
Mintlify's documentation describes llms-full.txt as a file that combines a whole documentation site into one file as context for AI tools, and Anthropic's llms.txt points to its own llms-full.txt.
In our reading of Mintlify's documentation and Anthropic's llms.txt, llms.txt lists the pages and llms-full.txt carries the text; llms-full.txt is a companion file that some platforms generate, and the llmstxt.org text we read does not describe it. A site serves one when it wants the whole text of a documentation site in a single file, the use Mintlify describes. Writing and placing the llms.txt file itself takes four steps.
How do you write and place an llms.txt file?
To write an llms.txt file, list the pages you want an AI system to read first, add a short summary and publish the Markdown at /llms.txt on the site's root, following the llmstxt.org specification. Four steps produce the file.
- Choose the pages: select the pages an AI system needs first for the llms.txt list, judged by the questions visitors ask most (our reading).
- Write the sections: write the title, a short summary and the link lists of the llms.txt file in concise, clear language, with a brief note per link, as llmstxt.org recommends.
- Publish the file: publish the llms.txt file at the root path /llms.txt of the site.
- Check the URL: check that the llms.txt URL answers with status 200 as plain text (our own check), then ask an agent questions with only the file as a starting point, as llmstxt.org suggests.
Where do you put an llms.txt file? At the root, /llms.txt; in the current version of the proposal, a file at a sub-path such as /docs/llms.txt covers the URLs under its path, and the most specific file applies. Can you generate the file automatically? Yes: the Integrations section of llmstxt.org, read on November 14, 2026, names five platforms and what each does. Yoast SEO, a WordPress plugin, generates and maintains the file; AIOSEO, a WordPress plugin, offers an llms.txt generator; Wix generates it for every Wix site; GitBook serves it for published documentation sites; and Mintlify generates it for every site it hosts. This page ranks and recommends none of them. Where the file sits in AI search optimization is the next question.
Where does llms.txt sit in AI search optimization?
llms.txt sits in AI search optimization as a file a site publishes for AI systems to read: a content index. That placement is our reading and not a claim that any system uses the file.
Is llms.txt SEO or GEO? Google's AI optimization guide places creating unnecessary AI text files, like llms.txt, among the tactics it says you can ignore for Google Search, under the advice to prioritize effective SEO strategies over "AEO/GEO hacks".
GEO (generative engine optimization) is optimization for the engines that write an answer from several sources, AEO (answer engine optimization) is optimization to be quoted in a direct answer, and ChatGPT is one of the chat assistants these terms cover. The wider map of engines, crawlers and methods is in the guide to GEO, AEO, ChatGPT and how to optimize for them. Which systems actually read the file is the next question.
Which systems actually read llms.txt?
None of the AI crawler documentation we read presents llms.txt as an input or a control, so the documented readers are few. The pages are the crawler pages of OpenAI, Perplexity, Anthropic and Google, read on November 14, 2026.
Two layers follow: the llms.txt files that AI companies publish for their own documentation, and what Chrome's Lighthouse checks about llms.txt.
Which AI companies publish their own llms.txt file?
OpenAI, Anthropic, Perplexity and Google, for its Gemini API and AI Studio documentation (a product separate from Google Search), each publish an llms.txt that indexes developer documentation. We fetched the four files on November 14, 2026. Their addresses are developers.openai.com/llms.txt, platform.claude.com/llms.txt, docs.perplexity.ai/llms.txt and aistudio.google.com/llms.txt; ai.google.dev/gemini-api/docs/llms.txt reaches the last one through a 301 redirect, which we measured on November 15, 2026. The llmstxt.org v2 specification names OpenAI, Anthropic and Gemini among the AI labs that publish their own files for developer docs.
Perplexity's file sits on a Mintlify-hosted documentation site, so we infer no intent from it. Publishing a file for one's own documentation does not show that these companies' crawlers read other sites' files, in our reading. Chrome's Lighthouse takes up the file from another side.
What does Chrome's Lighthouse check about llms.txt?
Chrome's Lighthouse includes an llms.txt audit among its "Agentic browsing audits" and flags a page only when fetching the file returns a server error, according to Chrome for Developers' documentation.
A 404 is marked Not Applicable because providing the file is optional "at the moment", and Chrome for Developers calls the file an "emerging convention" on a page last updated May 5, 2026. The audit is Chrome's and makes no statement about Google Search. Google Search has its own statement on the file.
What does Google say about llms.txt?
Google Search does not use llms.txt or other AI text files, and creating one will neither harm nor help visibility or rankings in Google Search, according to Google's AI optimization guide.
The guide, "Optimizing your website for generative AI features on Google Search", was last updated July 10, 2026, and speaks for Google Search only; other services can read the file, and this page makes no claim for them. Google says its generative AI features on Google Search are rooted in its core Search ranking and quality systems, so "the best practices for SEO continue to be relevant". Therefore, in our reading, the base of visibility in those features is the same SEO work that serves the rest of Google Search, and our comparison of GEO, SEO and AEO covers what to prioritize between them.
In its section "Mythbusting generative AI search: what you don't need to do", the guide says you do not need to create new machine-readable files, AI text files, markup or Markdown to appear in Google Search, generative AI capabilities included, because Google Search itself does not use them. Does Google use llms.txt for AI Overviews and AI Mode? The guide presents itself as the official best practices for generative AI features in Google Search, "such as AI Overviews and AI Mode", and its llms.txt sentence names neither feature; that neither uses the file is our reading.
As Search Engine Journal reported on June 2, 2026, Google's John Mueller wrote in a Reddit comment: "I don't think anyone knows – it's purely speculative for now (the file has existed for years, yet none of the AI systems use it — what does it mean?)". His comment is a personal remark, not Google documentation.
Is llms.txt a citation signal for other AI systems? No source we verified shows one (our reading); the studies of citation frequency and of server logs report what was measured. What the data shows comes next.
What does the data show about llms.txt?
SE Ranking's data shows no correlation between having an llms.txt file and how often a domain is cited by LLMs, in a vendor study of nearly 300,000 domains published October 15, 2026.
SE Ranking calls its findings context-dependent, and its scope is citation frequency in sampled LLM answers, with no claim about rankings, clicks or traffic. Four layers separate what the data counts: how many sites publish the file, what server logs show, what the file changes for a site, and how to check fetches on your own logs.
How many sites publish an llms.txt file?
Of the 20 domains ranked on US page one for "ai seo" and "ai search optimization" on November 11, 2026, 12 served a file opening with a Markdown H1 when we requested /llms.txt on November 14. This is the set of 20 domains that our guide to AI search optimization checked for robots.txt, a different file requested on a different date. What a domain returned at /llms.txt varies in four ways.
| What /llms.txt returned | Domains |
|---|---|
| A file that opens with a Markdown H1 | 12 |
| A text file that starts with a list and no H1 | 1 |
| An HTML page with status 200 | 1 |
| A 404 status | 6 |
One of the 12 domains serves an llms-full.txt next to its llms.txt. Search Engine Land, SEO.com and HubSpot's blog serve a file, and Ahrefs returns a 404. This is one observation of one set of SEO and GEO sites, not the web.
SE Ranking reports that 10.13 percent of nearly 300,000 domains had an llms.txt file, in a study published October 15, 2026.
Ahrefs, in its study "We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read" (June 15, 2026), studied the 137,210 domains in its Web Analytics that received traffic in May 2026, and reports that 28 percent publish an llms.txt, an upper bound because its customers skew technical and SEO-aware. What server logs show about requests comes next.
What do server logs show about llms.txt requests?
Server logs show no visits from GPTBot, ClaudeBot or PerplexityBot to the llms.txt page of Search Engine Land, from mid-August to late October 2025, according to Semrush's analysis of its sister site, in an article of September 28, 2026.
Googlebot and Bingbot fetched the page only a few times, and Semrush reads that as no special importance. Fetching a file is not using it, in our reading.
SeoMix, a French SEO agency, exported the server logs of 6 of its own sites over 60 days in April and May 2026, in its page "Llms.txt : le fichier inutile pour le GEO et les IA" (updated June 16, 2026): 33,092 visits from AI bots to 2,249 different URLs, and no bot request for llms.txt. That is one publisher's figure: "aucun appel" (no call) holds for the AI robots SeoMix identified, and the page does not say whether each of the 6 sites published a file (our reading of the page).
Abondance, a French SEO publication, counted 1,657 requests to its two files on one site, abondance.com, between June 1 and November 4, 2026, over 156 days of logs (the logs of October 22 are missing), in its article "Fichier llms.txt : ChatGPT ne l'a ouvert que 2 fois en 5 mois sur Abondance" (published November 5, 2026). Only 10 of them, 0.6 percent, came from identified AI robots, in a category that includes Common Crawl; Abondance details ChatGPT-User 2, ClaudeBot 2 (a single opening, on October 21) and Meta-ExternalAgent 4, and does not detail the rest. Its host's firewall blocked GPTBot from August 25 to November 5 and PerplexityBot in part, so their zeros are not evidence of non-use and are to be read with that limit; OAI-SearchBot shows 0 with no block reported. One request does not prove use, so Abondance reads the figures as a ceiling. In the same logs, 585 requests (35.3 percent) came from SEO and GEO tools and scanners, and 34 from the real Googlebot in five months.
Ahrefs reports that 97 percent of the roughly 38,000 domains with a valid file in its Web Analytics sample received no request in May 2026. The 19.5 percent of requests from named AI tools concern the 3 percent of files that received any request, not the 137,210 domains, and Ahrefs treats every figure as a ceiling. What these counts change for a site is the next question.
What does llms.txt change for visibility, citations and traffic?
No source we verified shows a change in visibility, citations or traffic from llms.txt, so the measured result so far is nothing. The absence is our reading of the pages of SE Ranking, Semrush, Ahrefs and Adobe, read on November 14, 2026.
Why does llms.txt matter if it changes nothing measured? Site owners ask for it: about 3,600 US searches a month use "llms.txt" (Google Ads monthly average, United States, measured November 14, 2026). The file costs little to publish, in our reading.
Adobe's Experience League video tutorial "LLMs.txt - is it relevant (June 2026)", on a page updated November 8, 2026, reports an audit of the server logs of 4,685 AEM-hosted sites: 0.4 percent had an llms.txt. According to the video, whose figures come from automatic captions, 92 percent of about 22,000 bot requests for these files over 30 days came from SEO tools, 6.4 percent from Googlebot and 1.1 percent from LLM bots, and Googlebot made 1,219 requests for the files. In our reading, these two Googlebot figures do not match exactly: 6.4 percent of about 22,000 requests is about 1,400, not 1,219. Adobe says Googlebot now reads (fetches) the file and that it cannot say for sure Google uses it.
A fetch is not a use, in our reading: Google's guide says Google may discover, crawl and index many kinds of files in addition to HTML, which does not mean a file is treated in a special way. A site can measure fetches on its own logs.
How do you check whether AI systems fetch your llms.txt file?
To check whether AI systems fetch your llms.txt file, read your server or CDN logs for requests to /llms.txt and group them by user agent. Three steps run the check.
- Find the requests: filter the logs for the path /llms.txt, including the llms.txt requests answered with a 404.
- Group the user agents: count the llms.txt requests per user-agent string and per day.
- Compare the agents: compare each llms.txt user agent with the names in the vendors' crawler documentation.
Google's documentation describes a reverse then forward DNS lookup to verify a request that claims to come from Google, so a user-agent string alone does not prove who sent it (our reading). The file's neighbors, robots.txt and sitemap.xml, do other jobs.
How does llms.txt differ from robots.txt and sitemap.xml?
llms.txt differs from robots.txt and sitemap.xml by its job: robots.txt sets crawl access rules, sitemap.xml lists URLs for discovery, and llms.txt offers curated, optional context for language models. The "Existing standards" section of llmstxt.org says that sitemaps list all pages for search engines while llms.txt offers a curated overview for LLMs, and that robots.txt and llms.txt have different purposes. The three files differ by who reads them and what they decide.
| File | Job | What it decides |
|---|---|---|
| robots.txt | robots.txt tells search engine crawlers which URLs they can access, per Google Search Central's "Introduction to robots.txt" | Crawl access |
| sitemap.xml | sitemap.xml tells search engines which pages and files the owner thinks are important, per Google Search Central | Discovery of important URLs |
| llms.txt | llms.txt offers a curated overview for language models, per llmstxt.org | Which curated pages an agent can use on demand |
Two questions detail the file's role: which file keeps AI crawlers out, and whether llms.txt replaces the other two.
Which file keeps AI crawlers out of a site: robots.txt or llms.txt?
robots.txt, not llms.txt, is the file named in the crawler documentation of OpenAI, Anthropic and Perplexity for managing crawler access: OAI-SearchBot and GPTBot at OpenAI, ClaudeBot at Anthropic, PerplexityBot at Perplexity.
Among the tokens documented on the OpenAI, Anthropic, Perplexity and Google pages we read on November 14, 2026, we retain these nine: OAI-SearchBot, GPTBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User and Google-Extended. OpenAI's page "Overview of OpenAI Crawlers" says OAI-SearchBot and GPTBot are set independently in robots.txt; Anthropic says ClaudeBot, Claude-User and Claude-SearchBot honor robots.txt directives; Perplexity gives a robots.txt tag for PerplexityBot; and Google presents Google-Extended as a standalone token for Gemini with no effect on Google Search.
OpenAI says ChatGPT-User actions are user-initiated, so robots.txt rules may not apply, and Perplexity says Perplexity-User generally ignores robots.txt rules. Anthropic's support page says its bots honor robots.txt directives and, as of October 30, 2026, a machine-readable AI-training preference signal.
For Google's generative AI features, the control sits in Search Console: according to Search Console Help, the "Search generative AI control" (Settings > Search generative AI) defaults to "Include my site's links and content in Search generative AI features", and a property with a parent property defaults to "Inherit control from parent".
Does a page need llms.txt to be eligible for Google's AI features? No, in our reading. According to Google's guide, updated July 10, 2026, a page is eligible to appear in generative AI features on Google Search, such as AI Overviews and AI Mode, when it meets three conditions: it is indexed, it can be shown in Google Search with a snippet, and its site is included in "Search generative AI features" in Search Console. Eligibility does not guarantee display, and a text file is not among these conditions, in our reading; our guide to Google AI Overviews covers these conditions.
Will llms.txt replace robots.txt or sitemap.xml?
No, llms.txt does not replace robots.txt or sitemap.xml, because llmstxt.org says llms.txt "is designed to coexist with current web standards", can complement robots.txt, and has a different purpose from robots.txt. The llmstxt.org "Existing standards" section adds that llms.txt can complement robots.txt by providing context for allowed content.
Is llms.txt worth the effort next to other GEO priorities?
No, not as a priority: you should treat llms.txt as an optional, low-cost addition after the SEO basics, because no source we verified shows an effect on citations or traffic.
The work that comes first is crawler access, indexing, content that answers the question and measurement of what AI systems fetch and cite, in our reading. A site with those four in place can add llms.txt for the cost of one Markdown file, with no result promised. In our reading, publishing it is not a priority, but it is not forbidden and not useless for everyone either, because the measures we read say neither: Google writes that it is "completely fine" to create such files for other services, and no source we read forbids the file or measures a gain. A site that wants that work measured and carried out can hand it to a generative engine optimization agency. Who should publish the file is the next question.
Who should publish an llms.txt file?
Sites that publish developer documentation should publish an llms.txt file, in our reading, as AI companies do for their own documentation. Three cases show the decision.
- Documentation site: a documentation site can publish an llms.txt file to list the pages of its guides for agents; the cost is a short list of links to maintain, and the caveat is that publishing shows no reader.
- E-commerce site: an e-commerce store can publish an llms.txt file to list its category and policy pages for agents; the cost is one Markdown file kept in step with the catalog, and the caveat is a catalog that changes often.
- Agency or publisher site: an agency or publisher can publish an llms.txt file to list its guides and services; the cost is the upkeep of the list, and the caveat is that no source we verified shows a change in citations or traffic.
Outside advice on that decision is the last question.
When does SEO consulting help with llms.txt and GEO priorities?
SEO consulting helps with llms.txt and GEO priorities when a site has more candidate fixes than time, since a session ranks the actions by impact and effort. An SEO consultant advises on priorities while the site's own team or an agency carries out the work. One-to-one SEO consulting covers the audit, the ranking of actions and the written summary.
