Google's Matt Cutts Confirms AdSense Bot Helping Googlebot With Indexing

Matt Cutts, who is
speaking
at this week’s PubCon,

confirmed that the AdSense mediapartners bot is doing double duty by not
only targeting ads for AdSense but also
indexing for
the regular Google search database, in a bandwidth saving move.

Matt also noted that there is no advantage to being indexed by one bot or the
other, however, those cloaking content and serving different pages to each bot
could run into problems in the search index. More details on
JenSense.

Postscript From Danny: Matt

adds more about this on his blog, and it’s a super-important clarification:

Pages with AdSense will not be indexed more frequently. It’s literally just a
crawl cache, so if e.g. our news crawl fetched a page and then Googlebot wanted
the same page, we?d retrieve the page from the crawl cache. But there’s no boost
at all in rankings if you’re in AdSense or Google News. You don?t get any more
pages crawled either.

In other words, there are two big issues with the AdSense crawler helping
Googlebot:

1. Since the AdSense crawler swoops in fast, it could be a way for people to
   effectively get fast inclusion of their pages. Just add AdSense, wait for the
   AdSense bot to fly in, and you’re set.
2. Is having the AdSense crawler likely to get you a RANKING boost, in
   addition to getting INDEXED faster. I’ve capitalized both words to stress
   them, as a reminder that being in the index isn’t the same as ranking well for
   a query.

Matt’s saying that no to both cases. There is no ranking boost. As for fast
indexing, no to that as well. The AdSense bot is simply refreshing the cached
copy of your page — but the copy in the index, what people are searching on,
won’t be updated.

That brings up an entirely new point. It means that Google is now potentially
presenting its results as fresher than they are.

What you search on is in the index, not the cache, as I’ve explained. So if a
page changes and the index isn’t updated — only the cache — then Google won’t
know about those changes to help with searching.

Matt also noted that there is no advantage to being indexed by one bot or the other, however, those cloaking content and serving different pages to each bot could run into problems in the search index. More details on JenSense.

For searchers, the date on the cache is a useful way to know if Google’s
index is updated. Now, you can’t tell. For site owners, the cache has been a
useful way to know that Google has indeed updated your pages. Now it no longer
serves that function.