A Cloudflare pilot has helped OpenAI index hundreds of millions of additional pages for ChatGPT search. The mechanism: Cloudflare's network signals gave OpenAI a way to discover and crawl content at scale that it previously couldn't reach. The result is a meaningfully larger index underpinning what ChatGPT can draw on when generating search responses.
For most marketers, that sounds like good news. More pages indexed means more chances to appear. But the correct read is more conditional than that. Expanded indexing capacity only benefits brands whose content is actually accessible to AI crawlers. If you've been treating bot access as a technical afterthought, this development shifts it into strategic territory.
The Index Was Already Uneven
AI search engines don't index the web the way Google does. They rely on a combination of their own crawlers, licensed data partnerships, and network-level infrastructure to discover content. ChatGPT's OAI-SearchBot crawler is real and active, but its reach has historically been limited compared to what Googlebot can cover. That gap has had direct consequences for AI visibility.
The Cloudflare pilot is significant because it demonstrates a different route to scale: rather than OpenAI independently growing its crawler infrastructure, it used signals from a network that already sits in front of a large portion of the web's traffic. That's a meaningful architectural move. It also signals that AI search engines are actively working to close the indexing gap rather than accepting it as a permanent constraint.
For brands, this means the pool of content ChatGPT can draw citations from is growing. That changes the competitive context. Content that wasn't previously reachable by OAI-SearchBot may now be in scope. And content that is blocked - whether by accident or design - is falling further behind.
Crawler Access Is a First-Principles Problem
Before worrying about schema, entity optimisation, or citation strategy, the question is simpler: can AI bots actually read your pages? Robots.txt files that were configured years ago to disallow all non-Google crawlers will block OAI-SearchBot, PerplexityBot, and others. Many sites have exactly this configuration and have never revisited it.
This isn't an academic concern. If ChatGPT is drawing on hundreds of millions of additional pages and yours aren't among them, you're not in the index. You cannot be cited. No amount of well-structured content or authoritative writing changes that outcome if the crawler can't get in.
The practical audit is straightforward. Check your robots.txt for explicit disallow rules targeting AI crawlers. Review your server-side firewall and WAF configurations - Cloudflare's own bot management tools, for instance, can silently block bots that aren't on an approved list. Check your CDN settings. Then verify with log file analysis that OAI-SearchBot, PerplexityBot, and GPTBot are actually returning 200 responses on your key pages, not 403s or 429s.
What llms.txt Actually Does Here
There's been a lot of noise about llms.txt as a mechanism for AI indexing. To be precise about what it is: llms.txt is a proposed convention, not a standard, that allows site owners to provide a structured summary of their content for AI systems to consume. It's distinct from robots.txt, which controls crawler access at the HTTP level.
In the context of ChatGPT expanding its index, llms.txt is a complement to crawler access, not a substitute for it. If OAI-SearchBot can crawl your site, an llms.txt file can help the model understand your content hierarchy and prioritise what matters. If the bot is blocked at the firewall, the llms.txt file is irrelevant. Access comes first. Structured guidance comes second.
That said, as AI search engines develop more sophisticated ingestion pipelines - and the Cloudflare pilot suggests they're actively investing in this - the value of machine-readable content signals is likely to increase. Getting your technical foundations in order now means you're positioned for each incremental expansion of the index, not scrambling to catch up after it.
More Index Doesn't Mean More Citations Automatically
Being indexed is a necessary condition for AI search visibility, not a sufficient one. ChatGPT, like other AI answer engines, is selective about what it cites. Content that answers a question clearly, comes from a source with coherent entity signals, and is structured for extraction tends to get cited more than content that's merely present in the index.
The expanded index raises the floor of what's possible. It doesn't flatten the quality signal. If anything, a larger pool of indexed content means the competition for citations becomes more intense. Being technically accessible and substantively useful are both requirements now. Neither alone is sufficient.
This is where the measurement question becomes important. Teams running AEO programmes need to know whether their AI-referred traffic is actually coming from pages that are indexed and returning citations, or whether they're being excluded and losing ground to competitors whose content is now in scope. Log file analysis, referrer tagging in GA4 for AI sources, and regular prompt testing across ChatGPT, Perplexity, and Google AI Overviews are the tools for answering that question with evidence rather than assumption.
The Infrastructure Decision Is Already Made
The Cloudflare pilot is a reminder that AI search infrastructure decisions made by platforms have direct consequences for brand visibility. OpenAI didn't ask brands whether they wanted to be included in a larger index. The index expanded. Brands either had their content accessible or they didn't.
That pattern will continue. As AI search engines build out their infrastructure - through CDN partnerships, crawler improvements, or licensing arrangements - the brands that benefit will be those that maintained open, well-structured, crawlable content estates. The brands that lose ground will be those that treated crawler access as a legacy SEO concern with no bearing on what happens in AI search surfaces.
Run the access audit. Check the logs. If you don't know whether OAI-SearchBot is successfully crawling your priority pages, that's the first thing to find out - not the last.