llms.txt Explained: Do You Actually Need One On Your Site
llms.txt is a plain markdown file at yoursite.com/llms.txt that gives AI tools a short summary of your site and a curated list of links to your most useful pages. It was proposed in 2024 as a way to help language models find the right content without wading through navigation, ads and scripts.
Do you need one? For most marketing, service and ecommerce sites, no. Google says its Search doesn’t use llms.txt, a study of about 300,000 domains found no link between having one and being cited by AI, and no major AI provider has said its crawlers read it. If you run a documentation site for developers, it can be useful. Either way, it takes minutes to add and won’t hurt. Here’s the evidence, a verdict by site type, and a two minute check you can run on your own logs.
What Is llms.txt And Where Did It Come From?
The llms.txt proposal was published by Jeremy Howard of Answer.AI, with its main version dated September 3, 2024. The idea is simple: AI models have limited context windows, and a typical web page wastes much of that space on menus, footers and scripts. The file gives them a clean, human written summary and a short list of the pages that matter, in markdown, which models read easily.
The spec suggests two companions. Sites can offer markdown versions of individual pages by adding .md to the URL, and some sites publish an llms full txt file that puts the full text of their docs in one place. Neither is required.
It’s worth being clear about what the file isn’t. It doesn’t block or allow any crawler, it isn’t a ranking signal anyone has confirmed, and it isn’t an official standard from Google, OpenAI or Anthropic. It’s a community proposal.
What Does An llms.txt File Look Like?
The spec requires only one thing, an H1 with the site or project name. By convention, files also include a one paragraph summary in a blockquote, optional notes, and H2 sections listing links with a short description each. Here’s an example for a fictional local plumbing company:
# Northside Plumbing
> Licensed plumbers serving Phoenix, Arizona. Emergency repairs,
> water heater installation and repiping for homes and small businesses.
## Services
– [Emergency plumbing](https://example.com/emergency/): 24/7 call outs and pricing
– [Water heater installation](https://example.com/water-heaters/): tank and tankless options
## Pricing And Areas
– [Pricing guide](https://example.com/pricing/): typical costs for common jobs
– [Service areas](https://example.com/areas/): cities and zip codes we cover
## Optional
– [About us](https://example.com/about/): licenses, team and history
The “Optional” section tells a tool which links it can skip if space is short. Keep the summary factual and the list short. A file that links to every page on the site defeats the point.
Who Actually Reads llms.txt?
This is where most guides get vague, so here’s what each side has actually said or shown.
Google: Search ignores it. Google’s guide to optimizing for generative AI features, published in May 2026, says you don’t need llms.txt or other “special” AI files to appear in Google Search, including AI Overviews and AI Mode, because Google Search doesn’t use them. It adds that creating one for other services is fine and will neither help nor harm your Google visibility. Earlier, John Mueller compared llms.txt to the keywords meta tag in April 2025, noting that server logs showed AI services weren’t even requesting the file, and in June 2025 said “no AI system currently uses llms.txt.”
The data: no measurable effect on AI citations. SE Ranking analyzed nearly 300,000 domains in November 2025. Only 10.13% had the file, and it showed no meaningful relationship with how often a domain was cited by AI. Removing that variable from their prediction model actually made it more accurate.
AI labs: they publish them, for their own docs. Anthropic publishes one for its developer documentation, and OpenAI publishes one for its developer hub. That’s the clearest real use case: developers point coding assistants at a docs index so the tool can find the right API page. But publishing your own docs index isn’t the same as reading everyone else’s. OpenAI’s crawler documentation describes how its bots respect robots.txt and links to its own llms.txt as a docs index; it doesn’t say its crawlers use llms.txt files on other sites to decide what to cite.
llms.txt vs Robots.txt vs Sitemap.xml
| robots.txt | sitemap.xml | llms.txt | |
| Purpose | Tells crawlers what they may access | Lists URLs for search engines to discover | Summarizes the site and lists key pages for AI tools |
| Format | Plain text directives | XML | Markdown |
| Officially supported by Google Search | Yes | Yes | No; Google says Search ignores it |
| Controls access? | Yes | No | No |
| What happens without one | Crawlers assume everything is allowed | Pages are still found through links | Nothing changes for most sites |
If you want to control what AI crawlers can access, robots.txt is the file that matters. For example, OpenAI says its search crawler and its training crawler follow separate robots.txt rules, which we cover in how to rank on ChatGPT. An llms file can’t do that job.
Do You Need An llms.txt File? A Verdict By Site Type
| Site type | Verdict | Why |
| Developer documentation, APIs, SDKs, open source projects | Worth adding | Developers paste docs links into coding assistants and agents; a clean index and markdown pages save them time. This is what the AI labs do themselves. |
| SaaS with substantial help docs | Optional, low effort | Useful for docs and support pages; little effect on marketing pages. |
| Service businesses, agencies, local businesses | Optional, low priority | No evidence it affects AI citations. Time is better spent on crawler access, indexing and answer quality. |
| Ecommerce stores | Low priority | Product discovery in AI depends on feeds, indexing and structured product data, not llms.txt. |
| News and publishers | Your call | Some publishers want to guide AI tools to licensed or canonical content; it has no effect on Google. |
If you’re in the “optional” rows, add one only if your SEO plugin does it in a few clicks, and don’t count it as AI search work. The things that actually move AI visibility are crawler access, indexing and pages that answer real questions better than the alternatives, which we cover in how to get your pages cited in Google AI Overviews.
How To Check Whether Anything Reads Your llms.txt
You don’t have to take anyone’s word for it, including ours. If your host gives you raw access logs, search them for requests to /llms.txt and note the user agent on each one. On a Linux server, a single command does it:
grep “llms.txt” access.log | awk -F\” ‘{print $6}’ | sort | uniq -c | sort -rn
That lists every user agent that requested the file, with a count. Compare it with requests to your main pages from the same bots. If GPTBot or OAI SearchBot fetch hundreds of your pages but never touch the file, that tells you more than any guide. Many managed WordPress hosts also show raw logs in their dashboard if you don’t have server access.
How To Add llms.txt In WordPress
If you decide to add one and you use Rank Math, it’s built in. Per Rank Math’s guide, enable the LLMS Txt module in the Rank Math dashboard, then go to Rank Math SEO > General Settings > Edit llms.txt. Choose which post types and taxonomies to include; it lists titles, URLs and descriptions, skips noindexed posts, and defaults to a maximum of 100 items. Preview it at yoursite.com/llms.txt.
On other platforms, write the file by hand in the format above and upload it to your site’s root folder. Check that it loads in a browser as plain text.
Common llms.txt Mistakes
• Treating it as access control. It doesn’t block anything. Use robots.txt for that.
• Dumping every URL into it. A long, unfiltered list removes the whole benefit of a curated summary.
• Letting it go stale. Auto generated files are easier to keep current than hand written ones.
• Listing pages that contradict your site. If prices or services in the file don’t match your pages, you’ve created conflicting facts about your own business.
• Counting it as AI SEO. If an agency’s AI search plan starts and ends with this file, ask what else is in it.
If you want the parts of AI search that do move the needle handled properly, that’s the AI search optimization work Rizqly does alongside SEO and paid media.
Free Strategy Session
Not Sure What AI Search Sees On Your Site?
Book a free strategy session with Rizqly. We’ll review how AI search tools reach and read your site today, which files and settings actually matter, and what we’d fix first.
- Which AI crawlers reach you
- Whether llms.txt is worth it
- Your first fixes, in order
No cost, no pressure.
Frequently Asked Questions About llms.txt
Q1:Does llms.txt Help SEO?
Not for Google. Google’s May 2026 guide says Google Search ignores the file, and it neither helps nor hurts your rankings.
Q2: Does ChatGPT Read llms.txt?
OpenAI hasn’t said its crawlers use llms.txt files on other sites. It publishes its own llms.txt for its developer docs, and controls crawler access through robots.txt.
Q3: What Is The Difference Between llms.txt And llms Full txt?
llms.txt is a short index of links with descriptions. An llms full txt file, used by some documentation sites, contains the full text of the content in one file.
Q4: Is llms.txt An Official Standard?
No. It’s a community proposal published at llmstxt.org. No search engine or AI company has adopted it as an official standard for crawling.
