IndustrySearch Engine EKGs

Search Engine EKGs

The Search Engine EKG charts how well search engines are crawling the web. Spider activity is graphed back for months for different web sites.

A good search engine should gather more than a few pages from each web site it visits. The more pages it gathers, the more likely it will have a comprehensive index of the web. It should also make regular visits to the web sites in its index. This keeps the index fresh.

Unfortunately, it can be difficult for general search engine users to know how comprehensive and fresh a search engine is. There’s no way for them to know how many pages have been gathered or how often search engines revisit sites.

The Search Engine EKGs listed below are meant to help provide a rough guide as to how comprehensive and fresh the major search engines are. They illustrate how each search engine has visited three different web sites. They show how many pages were requested from each site and how often each search engine came back.

For search engine users, the EKGs tell you which search engines are more likely to have a complete and up-to-date index of the web. For webmasters, the EKGs give you an idea of when, or if, to expect a visit from the different search engines.

How To Read The EKGs

Most search engines visit web sites every few weeks, some more often, other less often.

When they come, they request pages from the web site, just as a human does. They usually spread their requests out over a series of days, in order to be “polite” and not overburden the web server by trying to read everything all at once. Then they go away, to return again at a later date. You can read more about this on the How Search Engines Work page.

This rhythm can be seen as a series of spikes on the EKGs. There’s a period of no requests, represented by a flat line. When the search engine visits, its requests cause the line to spike upward. When the search engine leaves, the line drops back down until the next visit.

What’s a “healthy” EKG? Ideally, a search engine would visit every day and gather everything at the web site, creating a steady line well above zero.

Realistically, there can be some periods of flat line, but they shouldn’t last long. When weeks or months pass without a return visit, there’s a good chance that the search engine is missing new or updated information. Just as with a human being, a long, flat line is indicative of a search engine in trouble.

The EKG Survey Sites

There are three survey sites used with the EKGs. Take a moment to learn about them, before viewing the graphs.

Site 1

This is the calafia.com web site. It was established in June 1996 and hosted the predecessor to Search Engine Watch, A Webmaster’s Guide To Search Engines, until June 1997.

There were many links to the guide from all over the web, and the content was well respected by many in the web development community. These factors meant that the site was considered important enough for some of the search engines to visit it differently than a “normal” site.

It currently has about 50 pages within it.

Site 2

This is an anonymous web site. It was established at the end of 1996. However, it was not until late April 1997 that it was submitted to the search engines. Submission speeds up the discovery process, though a good search engine should discover new sites through the course of regularly crawling the web.

In July 1997, the site expanded from roughly 20 pages to 230 pages. The extra pages are part of a sub-level within the site. Some search engines have crawled far more of these pages than others. This is a good indication of how comprehensive each search engine may be.

Site 3

This is another anonymous web site. It was established in Nov. 1997 and submitted to each of the major search engines. It has about 25 pages within it.

The Search Engine EKGs
Please choose the search engine you are interested in. You may also use the drop-down box at the bottom of each page to move between search engines.
All links below are “web page archives,” viewable with IE4+ or Netscape 4+

AltaVista Infoseek
Lycos Northern Light
Excite
also powers
AOL NetFind, Netscape Search
Inktomi
powers:
HotBot, MSN Search

Notes

The EKGs have been generated by analyzing web site activity logs with Marketwave Hit List. They do not include visits from instant indexing spiders, requests for the robots.txt file or duplicate page requests on the same day.

If none of that makes sense, you probably don’t need to worry about it. Otherwise, learn more about search engine visits by reading the SpiderSpotting page in this web site.

Resources

The 2023 B2B Superpowers Index
whitepaper | Analytics

The 2023 B2B Superpowers Index

8m
Data Analytics in Marketing
whitepaper | Analytics

Data Analytics in Marketing

10m
The Third-Party Data Deprecation Playbook
whitepaper | Digital Marketing

The Third-Party Data Deprecation Playbook

1y
Utilizing Email To Stop Fraud-eCommerce Client Fraud Case Study
whitepaper | Digital Marketing

Utilizing Email To Stop Fraud-eCommerce Client Fraud Case Study

1y