Search Engines Come Together on Sitemaps Auto-discovery
Google, Yahoo, Microsoft and Ask.com revealed a new auto-discovery protocol and the addition of Ask.com to the Sitemaps.org group.
Google, Yahoo, Microsoft and Ask.com revealed a new auto-discovery protocol and the addition of Ask.com to the Sitemaps.org group.
While Google, Yahoo, Microsoft and Ask.com compete fiercely for users and advertisers, they also find it in their own best interests to cooperate on some things. They have recently come together to present a unified solution to help webmasters, and therefore searchers. Yesterday, the four search engines revealed the latest enhancements to sitemaps at the Search Engine Strategies conference in New York, including a new auto-discovery protocol and the addition of Ask.com to the Sitemaps.org group.
“Publishers know their sites better than anyone, and can give unique insight to help search engines index and rank a site,” Tim Mayer, VP of global Web search, told Search Engine Watch. “I think every search engine wants to make it easy for webmasters to do that in a common format.”
A sitemap is an “inclusion protocol,” as Dan Crow, product manager at Google, describes it. It allows a webmaster to create an XML file which suggest to search engines which pages of a site are important, as well as include optional metadata like the time the page last changed. In contrast, robots.txt is an “exclusion protocol,” because it tells a search engine what pages not to spider.
It’s most useful in helping to index a site with dynamic pages, which may confuse a search engine’s crawler and lead to issues with indexing multiple versions of the same page, or not indexing some pages at all, Mayer said.
From a search engine’s perspective, it’s a more efficient way to know when a page has been updated than recrawling an index, Mayer said. None of the engines plan to stop crawling the Web to index sites, but will use the sitemaps protocol to do so more efficiently.
Back in November, Google, Yahoo and Microsoft announced a common protocol for sitemaps. They formed a working group and promised to continue development of the sitemaps protocol, which was originally developed by Google in 2005.
This week, the three are joined by Ask.com in the effort, and the four search engines announced that all will begin supporting auto-discovery of a sitemap through a line of code in a site’s robots.txt file, such as Sitemap: http://www.mysite.com/sitemap.xml. Google, Yahoo and Ask will begin using the auto-discovery of sitemaps immediately, and Microsoft promises to do so some time in 2007. All four will continue to accept sitemap submissions by webmasters as well.
The news was jointly announced this week by Google, Yahoo, Microsoft, and Ask.
Yahoo engineers came up with the idea while brainstorming ways to make the sitemaps protocol more useful, and to improve the experience for webmasters, according to Amit Kumar, senior engineering manager for Yahoo Web Search. Yahoo took the idea to the other engines, and they agreed that it made sense, he said.
The Sitemaps.org effort, and today’s extension of that, are parts of a larger trend toward giving webmasters more control. Google’s Webmaster Central and Yahoo’s Site Explorer, for example, are two programs that provide tools to help a webmaster influence the way a site is indexed by a search engine. Microsoft is planning its own version of a webmaster tool, though it is not clear when that might launch.
We report the top search marketing news daily at the Search Engine Watch Blog. You’ll find more news from around the Web below.