Wikia, the commercial site led by Wikipedia founder Jimmy Wales, is taking the next step toward launching its open source, human-assisted Web search tool, Search Wikia, by buying distributed search crawler Grub from LookSmart.
Wikia, the commercial site led by Wikipedia founder Jimmy Wales, is taking the next step toward launching its open source, human-assisted Web search tool, Search Wikia.
Speaking at the O’Reilly Open Source Convention (OSCON) last week, Wales announced that Wikia has acquired Grub, the distributed search spidering technology previously owned by LookSmart, and will begin using it to build an index for the Search Wikia project.
Grub, which LookSmart bought in January 2003 for $1.4 million, is a distributed crawling service that LookSmart had implemented as a screensaver that would use idle CPU time on a user’s PC to crawl the Web. The data was used to supplement its own centralized crawler’s indexing efforts.
LookSmart formerly crawled the Web to create its own index with its Wisenut crawler, acquired in 2002. The company’s fortunes took a turn for the worse when Microsoft dropped them as search provider for MSN at the beginning of 2004.
Since 2005, LookSmart now solely operates as a directory, either via its paid LookListings, its non-commercial Zeal directory, or the publisher-focused FindArticles.com. LookSmart also offers a white-label search-targeted and contextual pay-per-click advertising platform and bookmarking services to publishers, most notably Ask.com, Local.com, and Reed Business Information.
Wikia will immediately release Grub to the open source community, and make both the crawler and source code available at Grub.org. Users who download the application can run it either as a screensaver or a background process while other applications are running.
Specifics of the deal were not revealed, though it is part of a larger advertising deal between Wikia and LookSmart which was announced last week.
Under the deal, LookSmart will provide text and display ads in Wikia’s freely hosted wiki communities, and eventually on the Search Wikia site, Wales said. Ads will be sold by Wikia on either a cost-per-click (CPC) or cost-per-thousand impressions (CPM) model. Inventory not sold by Wikia will be back-filled by ads from LookSmart’s distributed ad network.
Wales’ Search Wikia project began garnering interest in December, when Wales let some details slip in an interview with London’s Times. Since then, Wales has been in discussions with both the open source community, and with commercial partners, Wales said. In May, the project brought on Jabber creator Jeremie Miller as technical lead on the project, and today the project is moving to the next phase, Wales said.
Grub’s modular design allows developers to extend and add functionality to the crawler, improving the quality and performance of the entire system. By combining Grub’s distributed processing network with the power of a wiki to form social consensus, the project has taken the next major step towards a future where search is open and transparent, Wales said.
The Search Wikia project is a mix of open source and commercial interests. The site itself will be commercial, monetized by LookSmart’s ads. The technology that is created by the effort, as well as the index of Web sites, will be available under a free license. “We’re basically creating its own competitors,” Wales said.
The project also incorporates several aspects of community, Wales said. For developers, the project presents a programming challenge that will help both Wikia and the open source communities around search applications like Nutch and Lucene, which Wikia will incorporate into its search engine.
The general public can participate first by downloading the Grub application to begin creating an index with the unused computing power. An early version of a Search Wikia search engine should be available in beta by the end of the year, Wales said, but users should not expect the completed product by then.
“When I opened Wikipedia, it had three articles, yet it was called an encyclopedia. That’s how we plan to do this: publicly and openly,” Wales said.
Once the Search Wikia site is up, users will have several social elements to participate in. They will be able to rank sites and pages, provide feedback, get into editorial discussions – many of the same activities that they would at Wikipedia. As with Wikipedia, Wales envisions a core group of editors will lead the activities at Search Wikia.
Wales thinks adding a human element can help solve some of the issues with spam and deceptive content showing up in search results. “It’s easy for people to tell what a good result is, and what’s spam or deceptive. If you have a trusted community, you have a way to manage the process,” he said.
Search Wikia will be different than the human-edited directory approach of Mahalo, he said. Besides the fact that Search Wikia is an open source project, Wales notes that the biggest difference is that Mahalo is about “paying people to write a Web index and supplementing it with Google.”
“It’s not particularly game-changing,” Wales said.
Wales’ goal is to eventually supplant Google, Yahoo, Microsoft and Ask as an everyday, broad Web search engine. “I don’t know yet if we can do that. We’re going to try to get something up by the end of the year. I don’t know how good it’s going to be, or how long it’s going to take to get good, but we’re going to
We report the top search marketing news daily at the Search Engine Watch Blog. You’ll find more news from around the Web below.