Google's own products are sending contradictory messages about llms.txt. Google Search says you do not need the file for AI visibility. Google Lighthouse flags it as part of a new agentic browsing audit. Site owners who followed the Search guidance may now be failing audits they did not know existed.

The split became visible in May 2026. Google Search published a new optimization guide that listed llms.txt among tactics site owners do not need. The guide grouped it with content chunking, AI-specific rewriting, and special schema, saying none of these are required for generative AI features in Search.

Days earlier, Google Lighthouse shipped version 13.3 with a new Agentic Browsing category. The update includes an llms.txt audit that checks whether a site provides the file and flags server errors when retrieving it. The audit marks sites as Not Applicable only on 404 responses; any other error counts as a failure.

Table of Contents

Why Google Says You Do Not Need llms.txt

Google's Search team has maintained for over a year that llms.txt is not a Google initiative. John Mueller compared it to the keywords meta tag, noting no AI services were using it and bots were not requesting the file. Gary Illyes and Amir Taboul confirmed at Search Central Live Deep Dive Asia Pacific that Google was not pursuing llms.txt.

The logic is straightforward. Search uses its own crawling and indexing signals, which do not depend on llms.txt. The file was never part of Google's ranking algorithm or AI citation system in Search generative experiences. Following the Search team's guidance means allocating zero effort to something Google Search does not use.

Why Lighthouse Now Flags It

The Lighthouse audit takes a different view. It describes llms.txt as a way to provide a machine-readable summary of a website for LLMs and AI agents. The documentation at llmstxt.org calls it an emerging convention and advises site owners to place the file in their root directory.

Lighthouse version 13.3 treats llms.txt as a tool for agents that browse the web. Browser agents need a different kind of site description to navigate efficiently without crawling every page. The file serves that use case, even if Search does not care about it.

The Root Cause: Two Different Problems

The reason for the divergence is that Search visibility and agentic browser readiness are different problems. The Lighthouse agentic browsing category is labeled experimental. Google has a track record of launching audits that later get removed when adoption does not materialize.

The tension between the Search team, which evaluates new crawling standards against existing indexing infrastructure, and the Lighthouse team, which evaluates emerging browser standards, is a structural feature of how Google is organized rather than a bug.

What Site Owners Should Actually Do

The practical answer depends on your audience. If your primary concern is Search visibility and organic traffic from Google Search, the Search team's position is clear: llms.txt is not needed. You can safely ignore the Lighthouse audit for now.

If your site benefits from AI agents that browse it for research, product comparison, or data extraction, having a properly formatted llms.txt file is worth the minimal effort. For sites with deep content hierarchies or large archives, this can reduce crawl budget waste and improve how your content is represented in AI-generated outputs.

The safest approach for most site owners is to monitor the situation rather than act immediately. Keep the llms.txt conversation on your radar, but do not allocate significant resources until there are clearer signals. The more important questions are whether your structured data is complete, whether your content is accessible to crawlers and agents, and whether your site performs well on Core Web Vitals.

If you do decide to add llms.txt, the implementation is straightforward. The file lives in your root directory and follows the format documented at llmstxt.org. It should describe your site's purpose, main content areas, and any access requirements. Do not try to optimize it for SEO; it is a machine-readable summary, not a page for human visitors.

Conclusion

Google's conflicting signals on llms.txt come down to one thing: two different teams solving two different problems. The Search team does not need llms.txt for indexing or AI citation. The Lighthouse team built an audit for agentic browsing readiness. Neither is wrong.

For now, most site owners should watch and wait. If you already have an llms.txt file, there is no harm in keeping it. If you do not, there is no urgent need to create one unless AI agents are a significant traffic source for your site. The experimental Lighthouse audit is unlikely to affect Search rankings in either direction.

Sources: Search Engine Journal

For more insights on SEO technical changes, visit XerAds Blog.