Searching for Images by Similarity

Searching for pictures needn’t take a thousand words. An experimental search engine helps you find images based on colors, textures or characteristics similar to a reference image.

Generally speaking, images pose more of a challenge for search engine developers than words on web pages. Why? Words on a web page are called text strings, made up of individual characters, and characters are stored in computer memory as unique numbers. It’s comparatively easy for a search engine to recognize and manipulate text strings, because they form recognizable patterns that can be compared to one another.

By contrast, images on the web are made up of hundreds or thousands of discrete points (pixels), each with its own color and brightness, which together make up a gestalt whole that our brain perceives as a picture. Like alphabetic characters, each point in an image is also represented by a number, but these numbers don’t form the same kind of recognizable patterns that text strings do, because each picture is unique.

The image search offered by Google, Yahoo and other search engines therefore relies heavily on other clues to identify the subject of an image—things like filename, image ALT tags that describe the image, text immediately above or below an image (which is often a caption of sorts), and links pointing to images. With enough of these clues, a search engine can make a decent guess about the content of the image.

Photo sharing sites like Flickr take this a step farther, allowing users to tag images with descriptive labels. This helps improve search results, but it’s far from a perfect solution. If someone uses the tag “orange,” does it describe a sunrise or a fruit? Or a pumpkin or an orange house or…

Another approach to image search involves analyzing certain characteristics of an image, attempting to “see” the image in the same way humans do. And taking this approach, you can also ask a search engine to, in essence, show you images that have similar characteristics to the one you’re currently viewing.

Tiltomo is an experimental search engine that works in just this fashion, allowing you to find similar images in two ways. You can search for images that have similar color and texture characteristics, or you can look for images with similar “themes,” which adds an analysis of subject matter to the color/texture mix.

Tiltomo searches a sample collection of images from Flickr, both from the general images group and those included in the “catchy colors” group. You start your search by entering a tag into a search box; results display thirty largish thumbnails from Flickr.

Here are some examples using color and texture:

And others that try to group images by “themes:”

Links beneath each image in these search results allow you to explore other sets of images, either by characteristics or theme.

As you see, search results aren’t perfect, but that’s not the point. Tiltomo’s developers want to provide a way to easily explore similar images, and perhaps stumble upon images you might not otherwise see. If you like images, it’s a great way to have a fun romp through Flickr.

Another experimental image search service is found at Russia’s State Hermitage Museum. You can search for images with certain color characteristics or layout characteristics using photoshop-like controls.

Results are 12 thumbnail images based on the criteria you selected. Clicking on a result displays a larger image of the artwork with text describing it—for example, van Gogh’s Boats at Saintes-Maries. Want more images like this? Use the menu at the bottom of the page to view similar artwork in the collection by the same artist, of the same style, from the same country/region, using the same technique, of the same genre or with a similar visual layout.

Why aren’t the major search engines using these types of image search techniques? They are, but mostly behind the scenes. As image search becomes more popular, however, I expect we’ll gradually see more and more of these types of query by image content search tools emerge into the mainstream.

Want to know more about content-based image retrieval? This Wikipedia entry is a good place to start.

Search Headlines

NOTE: Article links often change. In case of a bad link, use the publication’s search facility, which most have, and search for the headline.

Google Search Captures Lion’s Share Of Visitors
Media Post May 22 2006 10:02PM GMT
For the New Search Workforce, Offer A Career
Media Post May 22 2006 10:01PM GMT
Stats Say Google Losing Something
Internet News May 22 2006 9:46PM GMT
New service scans search results for viruses, spyware
tom’s hardware guide May 22 2006 9:09PM GMT
Photos: Inside Google Australia
CNET May 22 2006 6:12PM GMT
Bill Gates Outlines His Answer To Google
Information Week May 22 2006 4:03PM GMT
Internet searches: Librarians do it better
Reuters May 22 2006 4:03PM GMT
Google retains lead in U.S. search market
ZDNet May 22 2006 4:02PM GMT
Snap mixes ads in with search results
Australian IT May 22 2006 4:01PM GMT
Google Responds to Orkut Child Porn Allegations May 22 2006 2:30PM GMT
Google Class Action Emails Going Out
Search Engine Guide May 22 2006 2:24PM GMT
Firms will embrace the web or die, claims Google
PC Magazine UK May 22 2006 1:07PM GMT
Beyond Google and Yahoo: Advanced Search May 22 2006 11:02AM GMT
Was Google’s Summer of Code a Boon or Bust?
Internet News May 22 2006 10:27AM GMT
Analyzing Search Behavior for SEO
ClickZ Today May 22 2006 3:28AM GMT

Related reading

Search engine results: The ten year evolution
Five ways PPC customer support can help SMBs
#GoogleDoBetter The latest on internal issues at Google and Alphabet
Google Sandbox Is it still affecting new sites in 2019