What's the Best Search Engine?

It's a relatively straightforward question. Or is it? What does "best" mean? Do you want the one with the biggest index, the most features, or a focus on a particular topic area? The answer depends upon what you are trying to do:

• Find general information
• Find answers to very specific questions that require subject area expertise
• Locate competitive prices for products and services
• Optimize your website to improve your search engine ranking
• Purchase advertising to be seen by search engine users

The same answer is unlikely to apply to all of these uses or others that may interest you. In addition, the relative merits of each search engine may vary by topic and are likely to change over time as search technology and the vendor landscape changes.

What is a Search Engine Anyway?

Again, a straightforward question with not-so-obvious answers. A typical definition of a search engine would describe sites where you can type in a word or phrase and be presented with a list of pertinent sites. According to this definition, Google, MSN Search, and Yahoo are search engines. What about Amazon and Cars.com? They may not index much or any of the web external to themselves, but within their respective specialties (consumer goods and cars), users find very useful information. In many circumstances, such "vertical search" sites are better because they are more focused and often include additional resources that the general search sites might not.

Yahoo didn't start out as a search engine, rather it began its life as a directory. Whereas search engines scan, index, and retrieve information automatically, directories are compiled by real people who sort information into topics and subtopics to facilitate navigation. The organizational benefits come at a cost: people process information much more slowly, thus directories include only a very small portion of the Internet. Many search engines (including Google and MSN Search) now include directories, and all major directories include some sort of search (although often limited to within the material they've included).

From a sales perspective, vertical search sites can provide exposure to pre-qualified potential customers, perhaps even some people that circumvent standard search engines entirely. A word of caution: If you are considering advertising or paid inclusion for more specific sites, it is very important to see if there is sufficient traffic to warrant your expenditures. Analyze such campaigns much like you would a traditional media ad campaigns: look at usage statistics and user demographics to evaluate target ROI and pricing.

What a Tangled Web We Weave...

There is an ever-changing network of interrelationships between major search engines that makes it difficult to compare them to each other. For instance, as of early January 2004, Google provides search results for AOL, Netscape, Yahoo, and numerous other search sites. Yahoo purchased the Inktomi search engine (which powers HotBot, among others) in the fourth quarter of 2003, and the "word on the street" is that Yahoo will thus drop Google in early 2004. Overture powers some search engines (including recently acquired AllTheWeb) and provides a large share of pay-per-click (PPC) search advertising. Google is Overture's chief rival for providing PPC advertising. Ask Jeeves owns and shows results from Teoma, a Google rival, but both show paid results from Google.

Confused? If you really want to understand this tangled web (or get more confused), take a look at this chart (PDF format) from Bruce Clay, LLC. For the typical user, these relationships are not very important, but they may be if you are looking optimize your site for search. These relationships are very important if you are considering the purchase of advertising.

Some "Top" Search Engines

As of early 2004, the following search engines are among the most popular for general use:

Each has its own particular strengths and weaknesses, although due to the aforementioned interrelationships, some of them may provide very similar results.

The Metasearch Option

From a searcher's perspective, the best solution to finding information online may be to look at the results from a number of search engines to get a broad set of results and an idea of which engine is best for your particular purpose. Fortunately, this process has been automated by "metasearch" sites like Dogpile, MetaCrawler, and Vivisimo. These sites allow you to submit a single query that gets sent out to multiple search engines, then the search results are recombined and presented to you all at once. They also provide methods for refining your results, such as identifying sub-groups based on additional keywords (Dogpile) or clustering (Vivisimo).

Search Engine Statistics

Companies like comScore Media Metrix provide detailed search engine usage statistics, although typically for a fee. Sites like Search Engine Watch provide some free aggregate statistics.

Summary

Most of the major search engines provide rich enough functionality for most searchers. For cases where there is significant differentiation in results, the most appropriate search engine depends upon a variety of factors. In short, there is not single "best" search engine, rather there are many tools to choose from, some of which are better suited to particular tasks.

Additional Information

Additional information about some of the major search engines can be found in this Berkeley article. Note that the features table is probably somewhat out of date as search engine capabilities change rapidly. More information is available Search Engine Watch which provides abundant, detailed analyses of trends and technology.