Someone suggests adding an llms.txt file to your lawn-care website. They say it will help AI assistants understand your business and let you control whether AI companies use your content for training.

That sounds like a convenient solution. But it combines two different jobs, and the file doesn't do both.

Before adding another technical file to a small business website, it's worth knowing what llms.txt was designed for, what major platforms actually support, and which problem you're trying to solve.

An Information Guide, Not a Crawler Rulebook

The original llms.txt proposal describes a Markdown document that introduces a website or a section of it and points AI tools toward useful information. Think of it as an optional, curated guide to important pages. Because adoption varies by tool, publishing an llms.txt file does not guarantee that an AI assistant will read or use it.

For a service business, that might mean a short description of the company with links to its actual service details, coverage information, and contact process. The proposal also has applications for extensive documentation sites, where tools may need help navigating a large collection of reference material.

It is not a standard permission system for AI training. The proposed format organizes descriptions and links; it doesn't establish crawler-access rules through User-agent, Allow, and Disallow directives.

Those directives belong to a different system: robots.txt. Simply putting bot-blocking instructions into llms.txt does not apply the controls described in the major AI providers' crawler documentation.

Google Does Not Require llms.txt for AI Search

Google's guidance on generative AI search explicitly says that Google Search doesn't use llms.txt as a special signal. Creating one neither helps nor harms a site's visibility or rankings in Google Search, including its AI features.

For AI Overviews and AI Mode, Google's existing Search requirements and SEO fundamentals still apply. A page must meet the relevant indexing and snippet eligibility requirements to be considered as a supporting link, but eligibility doesn't guarantee an appearance.

This is specific to Google Search. It doesn't prove that no other software or AI agent can benefit from a curated llms.txt file. It does mean that a small business shouldn't treat the file as a required Google SEO task.

AI Crawlers Have Different Jobs

If your concern is how AI services access or use your site's content, start with the provider's actual crawler documentation.

OpenAI, for example, distinguishes between GPTBot, associated with crawling content that may be used for model training, and OAI-SearchBot, used for ChatGPT search. Its crawler guidance describes separate robots.txt preferences for these functions. Allowing search crawling while disallowing training crawling is a documented option; neither setting guarantees a particular search placement.

Google-Extended is a robots.txt product token, not a separate crawler. Google uses it to manage certain Gemini model training and grounding uses of crawled content without changing a site's inclusion or ranking in Google Search. It isn't a replacement for Googlebot settings.

Anthropic likewise documents separate bots for model development, search, and user-requested access, and gives site owners robots.txt instructions for managing them.

These distinctions matter because training, search discovery, and user-initiated retrieval are not interchangeable. A blanket promise to “control AI” with one new file oversimplifies the choices involved.

And robots.txt itself isn't a security barrier. It communicates preferences to cooperating crawlers; it doesn't make a public page private or prevent every form of access. Confidential customer information belongs behind appropriate access controls, not in a publicly accessible document.

When Might llms.txt Be Worth Maintaining?

If you operate a documentation-heavy website or use an AI tool that explicitly reads llms.txt, a carefully maintained guide may serve a purpose. Its value would come from helping that particular tool find accurate material, not from a guaranteed increase in AI mentions.

A small cleaning or handyman business with a handful of clear service pages may have little reason to create and maintain another file. An outdated summary could even conflict with the information customers see on the site.

If someone recommends the file, ask which tool will read it, what the file will contain, and how its usefulness will be checked. If those questions have no concrete answers, there's no urgent reason to add it.

Choose the Control That Matches the Goal

If you want customers to find correct service information, focus first on accurate, accessible pages and the search settings relevant to your website.

If you want to manage AI crawling or training preferences, review the relevant providers' current robots.txt guidance with whoever maintains your site. Changes should be checked carefully because different bots serve different purposes.

If you need to protect private materials, use proper access restrictions rather than assuming a crawler instruction will keep them hidden.

An llms.txt file can describe your website. It isn't a substitute for deciding who can access it, what may be used for training, or whether your existing pages serve customers well.

Sources and Further Reading

  1. Jeremy Howard / llms-txt. The /llms.txt File, v2 — Original proposal and intended Markdown format; revised August 10, 2026.
  2. Google Search Central. Optimizing Your Website for Generative AI Features on Google Search — Google's statements on llms.txt, AI features, and SEO fundamentals.
  3. OpenAI. Overview of OpenAI Crawlers — Separate crawler purposes and robots.txt controls.
  4. Google Crawling Infrastructure. List of Google's Common Crawlers — Google-Extended product token and scope.
  5. Anthropic / Claude Help Center. Does Anthropic Crawl Data From the Web, and How Can Site Owners Block the Crawler? — Published April 7, 2026; crawler purposes and robots.txt settings.
  6. Google Search Central. Introduction to robots.txt — Limits of crawler directives and why they aren't access controls.