OpenAI appears to be testing when ChatGPT should use its own search index instead of relying on results supplied by third-party search providers.
New research published by Peec AI maps a collection of internal ChatGPT search indexes and, more interestingly, points to experiments comparing OpenAI’s own index with external search results.
The existence of an OpenAI search index itself isn’t entirely new. OpenAI acknowledges that ChatGPT can retrieve information from its “indexed and cached web content,” while previous investigations have identified an internal system known as Labrador.
What’s new in Peec’s findings is the apparent scale of that infrastructure and evidence that OpenAI is actively testing how prominently its own index should be used.
Peec identified experiment names including prefer-index-over-serp-v3, which appears to test OpenAI’s index against third-party SERP results. The researchers also found references to separate indexes for news, shopping, PDFs, YouTube, academic papers and other specialized sources.
The analysis suggests ChatGPT’s search infrastructure is increasingly hybrid: OpenAI can combine its own crawled and indexed data with results obtained from external providers, choosing between them depending on the query and experiment.
Peec also reports observing a large-scale crawling experiment involving up to one billion pages, although OpenAI has not publicly confirmed the experiment or Peec’s interpretation of the internal systems.
That’s potentially the more important development for search marketers.
OpenAI building an index puts it closer to operating the infrastructure of a traditional search engine. Testing that index directly against external SERPs suggests the company is also working out how much of ChatGPT Search it can eventually power itself.
For publishers and SEOs, that could make OpenAI’s own crawling and indexing behavior considerably more important than it is today.
Search Engine Watch has reached out to OpenAI for clarification on the experiments and will update this story if we hear back.
Stargate AI infrastructure. Image Credit: OpenAI.
Start the conversation by posting the first comment