A pattern is emerging in how some brands approach AI search optimisation, and it is worth examining carefully before it becomes entrenched practice. The argument goes roughly like this: AI crawlers are separate from Google's web crawlers, so you can feed them richer, more structured, more AI-friendly content than you serve to human visitors or standard bots. Give Googlebot your normal page. Give GPTBot your detailed, claim-heavy, citation-ready content. Job done.
The problem is that this logic has a name, and that name is cloaking. Chris Green's piece in Search Engine Journal draws the parallel directly. Serving materially different content to crawlers than to users is precisely what cloaking policies have targeted for years. The fact that the crawler in question belongs to an AI system rather than a traditional search engine does not change the fundamental mechanics of what is happening.
Why the Parallel to Cloaking Matters
Cloaking was always primarily a trust problem, not just a technical violation. The reason search engines penalised it was straightforward: if the content they indexed did not reflect what users actually saw, their index became unreliable. The same logic applies to AI systems. If ChatGPT or Perplexity surfaces a claim sourced from content that no human visitor to that page would ever read, the AI's output is built on a foundation that cannot be independently verified.
AI search providers are citation engines. Their value proposition rests on being trustworthy intermediaries - summarising real, accessible information and pointing users towards it. If the content being summarised exists only for the crawler and not for the visitor, the citation is effectively hollow. A user following a link from a Perplexity answer expects to find the substance behind the claim. If it is not there, that is a user experience failure that reflects on the AI platform, not just the brand.
AI systems will cross-reference. They already do. Google AI Overviews pulls from indexed web pages that Google has crawled under normal conditions. If your website does not contain the claim that your AI-targeted content asserted, the inconsistency will surface - either through algorithmic detection or through the simpler mechanism of the AI checking whether what it is citing actually appears on the destination page.
The llms.txt Misunderstanding
Part of what is driving this behaviour is a misreading of how tools like llms.txt are meant to work. The llms.txt specification was designed to help AI crawlers understand a site's structure and identify the most relevant content for training or retrieval. It is a signposting mechanism, not a delivery channel for content that exists nowhere else. Using it to direct AI bots to entirely separate content repositories - content hidden from your normal site - is a misuse of the concept.
The correct approach is to use llms.txt to surface content that is genuinely on your site and genuinely accessible. If your most useful structured content is buried three clicks deep or locked behind a login, llms.txt can legitimately help AI crawlers find it faster. That is fine. That is what the mechanism is for. But if the content it points to cannot be found by a human following a normal browser session, you are building on sand.
What AI Providers Stand to Lose - and How They Will Respond
It is easy to focus on the risks to brands, but the incentives of the AI platforms themselves matter here too. Google, OpenAI, and the teams behind Perplexity and Gemini all have strong reasons to detect and discount content that is presented only to their crawlers. Their reputations depend on the quality and accuracy of their outputs. If brands can game AI citations by serving manufactured content to bots, the quality of AI answers degrades. That is a serious commercial problem for these platforms.
We should therefore expect AI providers to develop detection mechanisms for this kind of content disparity - much as Google developed cloaking detection. The brands that have invested in building hidden content layers for AI crawlers will find that investment wiped out when those mechanisms kick in. The question is not whether this will happen, but when. Brands that have built AI visibility the right way - by making genuinely useful content accessible to everyone - will be fine. Those that have not will face a reckoning.
What AI-Attributable Revenue Actually Requires
From an attribution standpoint, this matters for a reason that often goes unspoken. When we track AI-referred traffic through GA4 and into a CRM, we are tracking visitors who clicked a citation link and arrived at a page. If that page does not contain what the AI claimed it did, conversion rates will be poor. The citation earns the click; the page has to earn the lead. A hidden-content strategy might generate citations in the short term, but the downstream metrics will be weak because the landing experience does not match the AI's summary.
Genuine AI search revenue attribution starts with the same content discipline that good citation earning requires. The page a user lands on from a Gemini or ChatGPT citation must contain clear, structured, accurate information that reflects what the AI said about it. That consistency - between what AI cites and what the page delivers - is what converts AI-referred visits into qualified leads. Optimise for the citation and ignore the landing experience, and your AI traffic contribution to the CRM pipeline will always disappoint.
The Sustainable Path for AEO Practitioners
The practical conclusion is straightforward, if demanding. Every piece of content you want AI systems to cite must be live, accessible, and consistent across all crawlers and all users. Structured data, clear entity signals, well-organised prose that answers specific questions directly - all of it needs to be on your actual website, not in a shadow layer visible only to bots. This is harder than building a separate AI content feed. It requires genuine editorial investment.
It also means auditing what your llms.txt file currently points to. If any of those destinations contain content that a standard browser session would not surface, fix that before an AI platform's quality systems do it for you. The brands that will hold AI search citations over the next two to three years are those that have made their real expertise genuinely crawlable, structured, and consistent - not those that built the smartest shortcut around it.
The shadow web is a tempting construct. It feels like precision - serving exactly what AI needs, without the constraints of your normal site. But AI search visibility is, at its core, a trust relationship. You cannot build a durable trust relationship on content that only one party in the exchange ever sees.