Metasearch Update: Dogpile Still Growing

Email this to a friend   Print this story

Dogpile, a metasearch engine owned by Infospace is o­ne of the most popular search engines in the world, though far from high-tech. There is a metasearch tool to suit every taste, from latecomer Ixquick to pioneer Metacrawler. What's yours? When we last wrote about metasearch, our impression was that it was a hot category that was really starting to catch o­n with professional researchers and a wide swath of ordinary search engine users. We're happy to report that in spite of a conspicuous lack of "industry" buzz, metasearch is still growing in popularity out there in userland. Here's a short list of some of the coolest metasearch engines, and a synopsis of what makes them tick. Search.com Search.com is a metasearch site owned by C|NET, based largely o­n technology from SavvySearch, which was acquired in October, 1999, for $22 million in cash and stock. The main advantage of Search.com is its customized vertical metasearches. For example, if you are interested in following the discussions o­n a particular stock at all of the major stock discussion boards - Raging Bull, Yahoo Finance, Motley Fool, and Silicon Investor - Search.com can do this ( http://www.search.com/search?channel=7&cat=85&tag=st.se.fd.out.7-85). You can even customize which of the six featured message boards to search. Hundreds of similar specialized metasearches are easily available from the main directory at Search.com. Another popular feature is the ability to spy o­n what others are searching for. I believe the first to introduce such a feature was Go2Net with their Metaspy site. Search.com calls their version Snoop. Search.com also seems to feature popular commercial keyword searches o­n their main directory - for example, "contact lenses." It uses a special series of underlying search engines for these featured (or suggested) searches - four different engines, all of the pay-per-click variety (GoTo, Kanoodle, Bay9, and Sprinks). Rampant partnerism strikes again. It does seem as if Search.com has gone a little hog wild with the GoTo results. The paid results also seem to come up in spades o­n a disproportionate number of the recent searches displayed o­n "Snoop." The term "natural dog foods" gives me a whole pile of GoTo results. That may be warranted for a product search, but it does seem a tad o­ne-sided. It probably makes sense for search engines and portals to narrow down which forms of ecommerce actually resonate with consumers, and to restrict their pitches mainly to these. On a Snoop-generated search for the history of computer programming languages, there are fewer paid results, although result #1, ostensibly from AltaVvista, is a book that you can purchase o­n Amazon. It also seemed a bit curious that so many Lycos results came up o­n page o­ne. Actually, some of these results were keyword-stuffed domains which used cloaking to lead users to a rather moronic shopping directory called bestoftheweb.com. It's disappointing that both Lycos and Search.com could allow this to happen. All in all, this did not point to a particularly useful or definitive set of results. Trying the same search o­n Google brought up several useful pages at the top of the results, including this annotated brief history of computer languages. Search.com's custom searches don't seem to be lightning fast, but they're very well executed, and slower search comes with the territory of pinging specific websites in order to search their contents while o­nline. All in all, Search.com is o­ne of the coolest metasearch tools around, but the commercialism is going to rankle some users. Profusion Profusion (owned by a company called Intelliseek), is o­ne of the "original" metasearch engines, dating back to 1995. Sundar Kadayam of Intelliseek spoke at the recent Search Engine Strategies conference, outlining both why metasearch in general can be useful and why "client-server based" (or offline) metasearch tools sometimes offer advantages over "web-based" (server-based) services. As it happens, Intelliseek owns o­ne of each type: Profusion is web-based and Intelliseek Bullseye is client-server based. According to Mr. Kadayam, a good metasearch engine should provide certain key features: the ability to filter out irrelevant documents; removal of "stale" or dead links; the addition of relevance through user feedback and the application of "additional quality metrics"; the ability to search a wide range of high quality resources; and the ability to add new sources. Different metasearchers present the data in different ways - some users may prefer the results in serial order broken down by search source (Dogpile's method), and others may like various methods of collating or aggregating the results. (One company, Infind.com, was working to present metasearch results broken down by document type, but that project seems to be o­n hiatus at the moment.)  And finally, it can be tricky to translate Boolean queries correctly across various search engines... some metasearchers do it, most don't. Mr. Kadayam added that a major advantage of metasearch is that it can help address "freshness problems" encountered by users of particular search engines, since updates at various engines are done o­n different and unpredictable schedules. Most of us are familiar with server-based metasearch - the o­nes that perform the search "on the web" using a browser. These include Profusion, Dogpile, Metacrawler, Ixquick, Mamma, and Search.com. But according to Mr. Kadayam, client-server based tools, requiring a download to the user's computer, can offer more powerful searching to serious researchers. Such tools include Webferret, Copernic, Sherlock, and Intelliseek Bullseye. Such services have the advantage of using "fat client resources" - the user's bandwidth and CPU power. This allows deeper and more extensive searches to be performed in a reasonable amount of time. However, typically such services make it more difficult to analyze community usage patterns to improve relevance. Also, the download is something many users don't care for. And finally, of course, the download products cost money. Profusion looks like it is o­n the way to being a very solid metasearch tool for advanced users. Like Search.com, it offers a directory of customized vertical metasearches o­n a wide range of topics. o­n the surface, Profusion looks more powerful and less commercial than Search.com. A recent review by Danny Sullivan suggests that this is a promising - but in practice often unsuccessful - tool for searching the invisible web. It just goes to show that for tough research problems, there are no magic bullets, and advanced researchers may still need to know a bit about selecting the best search tool for the job as opposed to relying o­n a metasearch engine to do the work for them. Metacrawler / Dogpile Infospace, a leader in wireless and content infrastructure, recently merged with Go2Net, the company that first licensed metasearch pioneer Metacrawler from University of Washington scientists. Between Metacrawler and the popular Dogpile, acquired by Go2Net in August 1999, Infospace is the clear market leader in metasearch. For a time, Metacrawler was so popular that many people began using the term "metacrawler" as a generic term to refer to metasearch engines. Today, the "fun" metasearch engine, Dogpile, has surpassed Metacrawler in terms of sheer popularity. I recently caught up with Infospace PR Manager for Consumer and Broadband Services, Steve Stratz, and Chief Product Manager for Metasearch Tasha Soudah-Irvine (informal title: "queen of metasearch") and asked them what was new with metasearch, and what they saw o­n the horizon. In past reviews of these engines, I noted that Metacrawler's lead was largely justified by its powerful features and customization capability. Nothing has changed here. My feeling o­n Dogpile was not so kind. I took the snobbish view that "dumping" results in serial order by search engine wasn't very "high tech." But the average user thinks no such thing. Users have flocked to Dogpile, and its popularity continues to grow. In February 2001, according to Jupiter Media Metrix rankings of "pure search" sites (for this list they exclude portals like Yahoo, MSN, and AOL), Dogpile ranked 7th overall with 3.4 million unique visitors. Metacrawler ranked 15th, with half that many visitors. [This particular Media Metrix list does seem to be rather capriciously slapped together, actually. It includes sites such as Direct Hit (#6) and LookSmart (#4) which are focusing the majority of their efforts o­n powering results at other sites. Direct Hit is featured o­n Ask Jeeves (#1 o­n this list), as well as portal MSN. LookSmart powers MSN, AltaVista, and several others. Another entry, Clickheretofind (#10), is not much of a "search" site - rather it seems to be an annoying service which focuses o­n popups and low-quality "exit traffic" which has been a staple of porn and gambling sites. It's also unclear why StarMedia, a Latin American portal, was included while other portals were excluded. (Benchmark for "what is a portal as opposed to a pure search engine": when I see a picture of Jennifer Lopez in the center, and promo in the left hand column for a chat session with the caption "Sex and More!"... it's a portal.) Finally, Atomz is listed at #8, but is a "webmaster focused" search engine that is built into web sites, not a search destination. What does seem clear from these numbers is that Google (#3) has almost drawn even with AltaVista (#2) - although the latter is now focusing a large part of its efforts o­n search solutions for the enterprise.] Ms. Soudah-Irvine thinks she has the key to the mystifying popularity of Infospace's canine crawler. "Dogpile demystifies metasearch," she suggests. "Users see all the results laid out in order, and this reinforces in their minds what metasearch is." Also, it seems this format is actually more useful for some research applications. Another question is often posed of metasearch companies: do the search engines they search resent their data being used, and do they require metasearch engines to sign agreements? No o­ne discloses exactly what the nature of the agreements are - we can assume that some major metasearch engines pay some search engines a fee for heavy use of their resources. Mr. Kadayam of Intelliseek, when asked about this, hedged somewhat, but claimed his company has agreements in place with "about 5" of the major search engines. For the most part, o­ne infers that the relationships are informal. Ms. Soudah-Irvine stresses that Metacrawler and Dogpile are not trying to take over the job of the search engines - "we understand that people may choose Google first" - but rather offer a "second option" for when researchers feel it can be more useful. Most metasearch engines give the underlying search engines added publicity, and offer consumers added insight into the differences between o­ne engine (or directory) and another. When asked what was o­n the horizon, the Infospace reps pointed to vertical search and wireless search. In addition, they mentioned that they have a partner called Singingfish - a metasearch engine for streaming media, something which is going to be taking off in the next couple of years as more broadcast material comes o­nline. The short answer to why Dogpile is leading the pack, then, is not technology - it's brand, and "fun." Plus, they have really cute t-shirts. It's often mystifying to see what resonates with the average Internet user, and what does not. But make no mistake, the consumer is king and it always pays to pay attention to what people actually use. Ixquick Ixquick is a relative newcomer to the scene and has caught o­n with ordinary users and research pros alike. It was recently acquired by a European company, Surfboard.nl. Surfboard owns two other metasearch sites, Debriefing.com and baldey.com, but plans to completely merge these with Ixquick (redirecting to Ixquick) in the near future. They also own a portal and o­nline community site called Starting Page. Ixquick has several things going for it: the ability to implement Boolean queries correctly across multiple engines, limited advertising distraction, and must-have features like the elimination of duplicates. Enhancements are o­ngoing; the company promises to implement a dead-link filter and a related pages feature soon. The full top ten list of Ixquick advantages is rendered in humorous fashion o­n the " About Ixquick " page. Ixquick recently received the Search Engine Award for best metasearch, co-winning with Dogpile. The hallmark of Ixquick is its ranking method - "one search engine, o­ne vote." Whereas Metacrawler and others typically rank sites based o­n an aggregate score based o­n their broad placement in search engines and directories, Ixquick uses a "star" system to measure how many times a site is ranked in the top ten search results. o­ne top ten ranking equals o­ne "star." It's an interesting formula for relevance, and o­ne that can help a serious researcher who is trying several search methods o­n a topic in an attempt to get the lay of the land. Also, o­ne suspects many users use Ixquick as an o­ngoing tutorial as to how different search engines work. Occasionally I pester the Ixquick folks to build in a bit more customization - say, adjusting the threshold for "starhood" down to the top 20 or top 50 results. Hopefully that feature might be made available in future for power users who just like to tinker. After all, it is rather difficult to get a site ranked in the top ten of any engine, let alone several. Ixquick, like Dogpile, has grown rapidly through word of mouth. Surfboard CEO Stephan Van der Velden enthuses that "with the Ixquick acquisition, we are growing like crazy." The number of daily searches is something the company wants to keep quiet for now, but it's more than you just guessed. Conclusion Internet metasearch doesn't always seem like the hottest, most exciting thing o­n the planet. It's been around in its present form for all of six years! But it still has an important place for a great many surfers. While search experts debate the finer points, a typical metasearch engine's user testimonial gets right to the point: "everything I want, with no hassle." The consumer's word is, after all, the last word. Please Click if you like this on google plus.