More On Query Refinement, The Human Scale Problem & Creating The Search Dialog

Yesterday, I wrote of how search engines could do a better job of query
refinement and indeed did so in the past, especially because there was more
human involvement in the search process. That drew out Jim Lanzone, senior vice
president of search properties at Ask Jeeves, who sent me an email raising a
good point that humans haven’t gone away just because of the expense. Humans
also have a “scale” problem. More comments from Jim on that are below, along
with a further look at the scale issue and the need for the Google Generation to
rediscover query refinement.

I agree with both Jim and Christopher Coulter, who

on Robert Scoble’s

to my blog post:

Yeah well the Yahoo Human-Edited model couldn’t scale, so you get Google
Pagerank automational noised chaos. It’s back to 1996 all over again. And the
best ‘search’ is with a database

Indeed, humans haven’t scaled well in terms of helping us gather content from
across the web. Crawlers do a great job of that. I used a library metaphor on my
ODP Founder
Comments & Moving Past Directories
post earlier this year to explain why
directories, after some promise, went away in the face of crawlers.

In short, imagine you go into a library and can use one of two card catalogs
to find books on a topic you are interested in:

  • The human-compiled card catalog looks only at book titles and short
    human-written descriptions of the books, maybe 25-75 words in all.
  • The crawler-compiled card catalog will let you scan every word on every
    page of every book in the library.

The crawler-built catalog is far more comprehensive. It’s also far more
up-to-date. Remember, in the library of the web, the books often rewrite
themselves or add pages in the way books in a physical library do not. Humans
simply can’t keep up with that activity.

The key, of course, is that the crawler service isn’t just comprehensive but
relevant. It will find not just all the matching pages but often rank them so
you are getting the very best ones.

While humans don’t scale well in the info gathering and retrieval side, they
can play a role. More on that in a moment, but first, here’s what Jim said in
response to my post:

To say that the problem with human editors was due to it being “expensive” is
true, but I don?t think it goes far enough to explain the problem.

Sure, Ask had great relevancy, but only for a single-digit percentage of the
overall query stream. That is not how people search, and neither you or I or any
number of Web Search Universities is going to change that for the vast majority
of searchers.

Algorithmic search was the only solution to that problem because only an
index of billions of pages could meet the user need that exists across the long
tail of rare queries.

At its peak, the Ask Jeeves “knowledgebase,” as it was known internally,
matched on about 85% of searches. That was a lot. However, it was only picked
20-25 percent of the time, despite having premium placement at the top of the

Sure, some searches resulted in far higher pick rates than this. But the vast
majority did not. Therein lay the problem. And due to the exquisite overpromise
made by the premise of question-answering and the butler, this had consequences
for the Ask Jeeves brand.

The brand was lucky, on the other hand, to gain a foothold in the market
early on, and to hold on to millions of users because of it. But at the end of
the day, people use a search engine to find what they need – quickly. That
foothold would have become more tenuous had we not bolstered our ability to
answer searches (questions or keywords) more accurately, more quickly, and more
easily, than the original Ask product could deliver.

Today, with a combination of algo search, structured data search (Smart
Answers), and unique tools that help people find things faster (Zoom,
Binoculars), we are delivering against user needs exponentially more than we did

At the end of the day, this is about playing defense, not offense, against
that wide and vast stream of searches. Most people are going to do what they
want with that little white box, and will not have the patience of learning how
to “search better” or set up a bunch of parameters ahead of time. The minute we
understood this and started intuitively responding to their searches with better
results and options, the more loyal those users became to our search engine.

Again, I agree with much of what Jim says. Historically, we know that searchers
don’t read help documentation, don’t make use of options, don’t do anything much
beyond put a word or two in that black hole of a search box and get sucked in to
click on whatever seems like the first “normal” result that comes up. Put a
bunch of refinement links/suggested searches in front of them, and they’re
likely to just ignore those as being “weird” and move down toward the real

Solutions? First and foremost, do break the habit! I’ve taught search classes
for many years, and people are amazed that when they look a bit past the
standard 10 listings, there are options and suggestions that are useful.

In fact, we’ll be doing a “Pimp Your Search Engine” or “Bling Your Search
Engine” series shortly via
to try and help many readers understand some of the many features
that are offered to you on your own favorite search engines that habit just may
have blinded you to. Get 100 results at a time! Pop open new windows from search
results! Discover easy ways to refine your queries!

Beyond that, let’s see the search engines do more to make use of both humans and
automation. I do want a human-created knowledgebase at Ask Jeeves and elsewhere
to return.

Maybe it has to be much smaller and serve only the very most popular queries.
But why not make it that if I type in
hotels on Ask Jeeves,
the automatically-generated “Narrow Your Search” options off to the right-hand
side might be determined to make more sense to show up in the main part of the
results, to better help people narrow in.

Moreover, look at what I get now at Ask Jeeves in that Narrow Your Search

Why not step beyond the automation and for this type of broad, common search,
come up with some human-generated suggestions, such as:

  • Find a hotel by price, location or other options
    Find the official site of a hotel chain
  • Find reviews of particular hotels

Those aren’t perfect solutions/options, but I think they make the point. There
ought to be more the search engine can do to have an actual dialog with people
in the right cases.

I know, the risk is the less Google-like the results are, the more likely people
are to feel uneasy or unsure about using a search engine. Well, the Google
Generation needs to be smacked upside the face.

Honestly. Google has been an absolute, horrible failure in helping people refine
their queries. It’s no wonder people today may not even realize there are query
refinement options out there beyond Google and before it existed, given that the
major leader in the space hasn’t made use of these. Let’s see all the search
engines dare to experiment more with these features and applaud them with they

By the way, speaking of applause, notice this over at Yahoo for


Yahoo! Shortcut –


That’s listed right above the first search result. I checked it out AFTER making
my suggestions above, and you can see Yahoo’s hit two of the three things I
thought made sense. Great work, Yahoo! OK, cynics will say these links just help
push people into Yahoo’s own travel search areas. Yes, but they are good areas.
People should be checking them out!

Why might you miss these links? Because they still don’t look like the “regular”
results that are numbered. So maybe they need to change. Maybe instead of being
like this:

  1. Find your hotel at Get huge discounts on rooms
    at over 16,000 hotels and 800 cities. Rates for special events and
    sold out dates. Book online or call.

It needs to be more like this:

  1. The Yahoo Shortcut links above will take
    you to our special travel search areas designed to let you scan the web
    for hotel prices and information. You can sort by price, location,
    amenities and more.
  2. Find your hotel at Get huge discounts on rooms
    at over 16,000 hotels and 800 cities. Rates for special events and
    sold out dates. Book online or call.

Again, top of my head stuff. But I think it makes the point of the “dialog”
being presented in a more listings-style format. Maybe, maybe, that might help
users see stuff they’re missing.

Related reading

Case Study: How BDCenter transformed a reputation from 48% negative on Google to neutral
How to set up SEO on WordPress [Video]
Report: Future of search in 2020 according to SEO specialists
Four common Google Analytics myths busted