Google Slowed Crawl Rate to Aid Websites Protesting SOPA

As announced in advance by Google UK’s Webmaster Trends Analyst Pierre Far, the search giant intentionally slowed crawling on SOPA blackout day to prevent harming the rankings of sites participating in the protest. Bing actually crawled 2.35 percent faster, possibly giving them an edge in producing fresh results, according to data gathered by The Next Web.

Covario just released an infographic demonstrating the number of users affected by blackouts on popular sites Wikipedia, Reddit, WordPress, and Craigslist. Over at online competitive intelligence firm Hitwise, Research Director Heather Dougherty took a look at 10 key stats from the web-wide SOPA & PIPA protests.

GoogleBot Slowed… Well, to a Crawl

The Next Web cites web performance and security company CloudFlare in showing a 60.7 percent decrease in Google’s crawl rate on January 17. This is an inevitable and natural result of its spying an increase in 503 headers, the recommended cue to GoogleBot that the temporary blackout content should not be indexed. Baidu also slowed to a lesser extent, by 11.23 percent.

Cloudflare CEO Matthew Prince told The Next Web, “The drop in Google’s crawl traffic varies substantially across small/medium/large sites with the relative drops being 80.84, 48.67, and 24.92 percent, respectively.”

Bing, on the other hand, crawled 2.35 percent faster. TNW’s Brad McCarty noted, “The unintended consequence here is that today Google perhaps has results that are not as ‘fresh’ as Bing. However, it’s also possible that Bing’s increased crawl rate has left it with a few more results that are indicative of the blackout, instead of what a site’s content actually is.”

Far noted in his post that the reduced crawl rate shouldn’t have long-term effects and crawling should return to normal within a few days.

Wikipedia Lost About 11,000 Edits; 18.1 Million Minutes Not Spent on Reddit

Covario’s Who Did the SOPA Blackout Affect? infographic (below) shows the impact of the blackout on tens of millions of users, including:

  • 20,275,000 college students who couldn’t access a majority of the site
  • Predominantly male users who would have spent 302,222 hours on Reddit
  • 35 million WordPress bloggers

Craigslist could have lost up to $825,000 in revenue, based on the average number of new job postings per day. It’s worth noting, though, that users may have simply waited a day to post their listings.

According to Hitwise, almost 50 million visits could have been impacted by the blackout on Craigslist, the third largest U.S. site in terms of traffic. Another fun fact from her list of 10 key blackout stats:

“Searches for variations of SOPA and PIPA were (day of 1/16/12) driving about half as much search traffic as searches for variations of ‘Kim Kardashian’. To put that into perspective, searches for Ms. Kardashian drive 6 times more traffic than searches for President Obama.”

See who else was affected across the four major sites studied by Covario:


Related reading

SEO survey 2019 - Stats on competitor research
How to lead SEO teams and track its performance effectively: Experts tips
SEO is a team sport: How brands and agencies organize work
How to pitch to top online publishers: 10 exclusive survey insights