Google On How To Let Googlebot In, Keep Bad Bots Out
One of the things that came out of our
Course at SES San Jose last month was a wish that search engines somehow
made it possible for site owners to know they were sending “trusted” or
“certified” spiders. Now Google’s suggested one way this can be done.
Those blocking rogue spiders through IP filtering run the risk that they
might accidentally keep some of the “good” bots out. If you don’t know all the
Google IP addresses, there’s a chance you might reject a Google spider
accidentally. That might cause your pages to be dropped from Google.
How to verify Googlebot from Matt Cutts at the Official Google Webmaster
Central Blog covers a suggested technique to avoid this. Basically, all Google
spiders will report they are from the googlebot.com domain. So do a DNS lookup
on the IP address. If it comes back as googlebot.com, then you’re halfway there.
Halfway? Yes, that’s because people can lie about domain names. To avoid
spoofers, you then have to look up the domain name you found to see if it
matches the original IP range.
The blog post explains more, and it’s going to make the most sense to
tech-savvy webmasters that are implementing some type of IP filtering or
blocking already. Not doing that? Then don’t worry about this — it’s not really
Down the line, perhaps we’ll see less tech-savvy solutions come up, for those
sites getting slammed by bad bots but without IP filtering. But this is a great
start for now.
Matt’s also mentioned this on his personal blog, where people are
commenting on the technique.
Keyword difficulty is one of the essential metrics when it comes to carrying out keyword research. Explore the methodology behind SEMRush's new keyword difficulty metrics, how it is calculated and what it could do for your business. Download Now
Less than 4% of websites passed all Core Web Vitals tests. Is yours one of them? In this study we crawled over 2 million URLs, crunched the numbers, and performed correlation analysis across the top 20 organic Google search positions to get a sense of how websites are performing before Google rolls out the Core Web Vitals Update in June this year. This enables us to provide real performance benchmarks, insights, and data-backed guidance ahead of the algorithm update. Download Now
Having high-quality backlinks directing to your website should be a major factor in your overall SEO strategy. Crowdo.net is a unique link-building service. They provide Crowd Marketing Links (Crowdo Links), which work great for SEO and Web Traffic. Download Now
Every business owner is raving about Digital Marketing. Download Now
Google just updated its search algorithm, making it the perfect opportunity to boost your site’s page rankings with these Core Web Vitals tips
As search evolves to make sense of complex queries, influencers can positively affect the outcome of a site’s rankings when properly utilized
A stitch in time saves nine, and a lot more when it’s about the cookie death, search marketing and SEO
Highlights of the gaming industry’s challenges, the lack of SEO and internet marketing utilization, and scope for innovation