The Major Search Engines
[Home]

Why are the services below considered to be the major search engines? They are all either well known or well used.

For webmasters, these services are the most important places to be listed, because they can potentially generate so much traffic.

For searchers, these well-known, commercially-backed search engines generally mean more dependable results. These search engines are more likely to be well maintained and upgraded when necessary, to keep pace with the growing web.

Not all of the services below are "true" search engines that crawl the web. For instance, Yahoo and the Open Directory both are "directories" that depend on humans to compile their listings. In fact, most of the services below offer both search engine and directory information, though they will predominately feature one type of results over the other. See the How Search Engines Work page to understand more about the difference between search engines and directories. This information was gathered at www.SearchEngineWatch.com. Visit this site for more information and lots more search engines, metacrawlers, etc.

q Ask Jeeves www.ask.com Ask Jeeves is a human-powered search service that aims to direct you to the exact page that answers your question.

q Excite www.excite.com Excite offers a medium-sized crawler-based web page index, as well as access to human-powered directory results from LookSmart. Excite was launched in late 1995. It grew quickly in prominence and consumed two of its competitors, Magellan in July 1996, and WebCrawler in November 1996. Magellan was discontinued in April 2001. WebCrawler continues to operate as a separate service, but it provides the same results at the Excite.com site itself.

q FAST Search www.alltheweb.com FAST Search consistently has one of the largest indexes of the web. FAST also offers large multimedia and mobile/wireless web indexes, available from its site. The site, also known as AllTheWeb.com, is a showcase for FAST's search technologies. FAST's results are provided to numerous portals, including those run by Terra Lycos. FAST Search launched in May 1999.

q Google www.google.com Google is a top choice for web searchers. It offers the largest collection of web pages of any crawler-based search engine. Google makes heavy use of link analysis as a primary way to rank these pages. This can be especially helpful in finding good sites in response to general searches such as "cars" and "travel," because users across the web have in essence voted for good sites by linking to them. The system works so well that Google has gained widespread praise for its high relevancy. Google provides web page search results to a variety of partners, including Yahoo and Netscape Search (see below). Google also provides the ability to search for images, through Usenet discussions and its own version of the Open Directory (see below). 

q HotBot www.hotbot.com HotBot is a favorite among researchers due to its many power-searching features. In most cases, HotBot's first page of results comes from the Direct Hit service (see above), and then secondary results come from the Inktomi search engine, which is also used by other services. It gets its directory information from the Open Directory project (see below).

q Lycos www.lycos.com Lycos started out as a search engine, depending on listings that came from spidering the web. In April 1999, it shifted to a directory model similar to Yahoo. Its main listings come from the Open Directory project, and then secondary results come from the FAST Search engine. Some Direct Hit results are also used. In October 1998, Lycos acquired the competing HotBot search service, which continues to be run separately.

q MSN www.msn.com Microsoft's MSN Search service is a LookSmart-powered directory of web sites, with secondary results that come from Inktomi. RealNames and Direct Hit data is also made available.

q Iwon home.iwon.com Backed by US television network CBS, iWon has a directory of web sites generated automatically by Inktomi, which also provides its more traditional crawler-based results. iWon gives away daily, weekly and monthly prizes in a marketing model unique among the major services. It launched in Fall 1999.

q Yahoo www.yahoo.com Yahoo is the web's most popular search service and has a well-deserved reputation for helping people find information easily. The secret to Yahoo's success is human beings. It is the largest human-compiled guide to the web, employing about 150 editors in an effort to categorize the web. Yahoo has well over 1 million sites listed. Yahoo also supplements its results with those from Google (beginning in July 2000, when Google takes over from Inktomi). If a search fails to find a match within Yahoo's own listings, then matches from Google are displayed. Google matches also appear after all Yahoo matches have first been shown. Yahoo is the oldest major web site directory.

q Alta Vista www.altavista.com AltaVista is one of the oldest crawler-based search engines on the web. It has a large index of web pages and a wide range of power searching commands. It also offers news search, shopping search and multimedia search.

q LookSmart www.looksmart.com LookSmart is a human-compiled directory of web sites. In addition to being a stand-alone service, LookSmart provides directory results to MSN Search, Excite and many other partners. Inktomi provides LookSmart with search results when a search fails to find a match from among LookSmart's reviews.

q Netscape Search search.netscape.com Netscape Search's results come primarily from the Open Directory and Netscape's own "Smart Browsing" database, which does an excellent job of listing "official" web sites. Secondary results come from Google. At the Netscape Netcenter portal site, other search engines are also featured.

q Open Directory www.dmoz.org The Open Directory uses volunteer editors to catalog the web. Formerly known as NewHoo, it was launched in June 1998. It was acquired by Netscape in November 1998, and the company pledged that anyone would be able to use information from the directory through an open license arrangement. Netscape itself was the first licensee. Netscape-owner AOL also uses Open Directory information, as does Google and Lycos

q About.com www.about.com About.com, formerly the Mining Company, features hundreds of "guides" offering original content in various areas. While About.com isn't really a search service, the guides do have extensive links to other sites -- not to mention top-notch content of their own.

q Britannica.com www.britannica.com Links to top web sites and content from the Encyclopedia Britannica, in one place.

q WebCrawler www.webcrawler.com WebCrawler is operated by Excite@Home and offers the same results as can be found at Excite@Home's flagship site, Excite.com. WebCrawler was originally a completely independent service, opened to the public on April 20, 1994. It was started as a research project at the University of Washington.

Metacrawlers

Unlike search engines, metacrawlers don't crawl the web themselves to build listings. Instead, they allow searches to be sent to several search engines all at once. The results are then blended together onto one page. Below are some of the major metacrawlers.

q Ixquick www.ixquick.com Meta search engine that ranks results based on the number of "top 10" rankings a site receives from the various search engines.

q Infonetware RealTerm Search www.infonetware.com This site is primarily designed to demonstrate classification technology from Infogistics. It's a meta search engine, and it does topical classification of results, like Vivisimo. However, it is unique in that you can select several different topics, then "drill down" to see results from all of them, rather than being restricted to the results from only one topic. 

q Vivisimo www.vivisimo.com Enter a search term, and Vivismo will not only pull back matching responses from major search engines but also automatically organize the pages into categories. Slick and easy to use.

q QbSearch www.qbsearch.com Want to get multiple pages of results from a search engine combined into one single page? QB-Search will quickly join up to 200 pages of listings from major search engines. It's a simple, helpful tool that you'll no doubt need at some point. Another powerful offering is QB-Masterpage, which will combine URLs that you select into a single page for quick browsing. Both are available from QuickBrowse.

q SurfWax www.surfwax.com Searches against major engines or provides those who open free accounts the ability to chose from a list of hundreds. Using the "SiteSnaps" feature, you can preview any page in the results and see where your terms appear in the document. Allows results or documents to be saved for future use.

q ProFusion www.profusion.com Brings back listings from several major search engines as well as "Invisible Web" resources. Formerly based at the University of Kansas, the site was purchased by search company Intelliseek in April 2000.

q Query Server www.queryserver.com Search against major web-wide search engines, as well as major news, health, money and government search services.

q Kartoo www.kartoo.com If you like the idea of seeing your web results visually, this meta search site shows the results with sites being interconnected by keywords.

Popular Choices
Many of the meta search engines below are popular with users, but they aren't listed as top choices by Search Engine Watch. This is because they typically mix paid listings within editorial results, without any type of disclaimers. The article below covers this issue more. It also explains how you can control the ads you see and turn the metacrawlers listed into better resources, by making use of customization options.
Meta Search Or Meta Ads?
The Search Engine Report, June 4, 2001
A review of meta search services by Search Engine Watch shows that some are providing results where more than half of their listings are paid links. A guide to what's paid, what's not and how to get the most from your meta search service.

q Mamma www.mamma.com Sends search requests to major search services.

q Dogpile www.dogpile.com Popular metasearch site that sends a search to a customizable list of search engines, directories and specialty search sites, then displays results from each search engine individually.

q MetaCrawler www.metacrawler.com One of the oldest meta search services, MetaCrawler began in July 1995 at the University of Washington. MetaCrawler was purchased by Go2Net, an online content provider, in Feb. 97. The commercial backing has helped improve the responsiveness of the service. MetaCrawler now also powers searches at the Go2Net portal site.

q Search.com www.search.com Search.com is a meta search engine operated by Cnet. It offers both web-wide search and a wide variety of specialty search options. Search.com uses technology from SavvySearch, which was acquired by Cnet in October 1999. The SavvySearch site itself no longer operates.

How Search Engines Work

By Danny Sullivan, Editor, SearchEngineWatch.com
Updated: June 26, 2001

The term "search engine" is often used generically to describe both crawler-based search engines and human-powered directories. These two types of search engines gather their listings in radically different ways.

Crawler-Based Search Engines

Crawler-based search engines, such as HotBot, create their listings automatically. They "crawl" or "spider" the web, then people search through what they have found.

If you change your web pages, crawler-based search engines eventually find these changes, and that can affect how you are listed. Page titles, body copy and other elements all play a role.

Human-Powered Directories

A human-powered directory, such as Yahoo, depends on humans for its listings. You submit a short description to the directory for your entire site, or editors write one for sites they review. A search looks for matches only in the descriptions submitted.

Changing your web pages has no effect on your listing. Things that are useful for improving a listing with a search engine have nothing to do with improving a listing in a directory. The only exception is that a good site, with good content, might be more likely to get reviewed for free than a poor site.

"Hybrid Search Engines" Or Mixed Results

In the web's early days, it used to be that a search engine either presented crawler-based results or human-powered listings. Today, it extremely common for both types of results to be presented. Usually, a hybrid search engine will favor one type of listings over another. For example, Yahoo is more likely to present human-powered listings. However, it does also present crawler-based results (as provided by Google), especially for more obscure queries.

The Parts Of A Crawler-Based Search Engine

Crawler-based search engines have three major elements. First is the spider, also called the crawler. The spider visits a web page, reads it, and then follows links to other pages within the site. This is what it means when someone refers to a site being "spidered" or "crawled." The spider returns to the site on a regular basis, such as every month or two, to look for changes.

Everything the spider finds goes into the second part of the search engine, the index. The index, sometimes called the catalog, is like a giant book containing a copy of every web page that the spider finds. If a web page changes, then this book is updated new information.

Sometimes it can take a while for new pages or changes that the spider finds to be added to the index. Thus, a web page may have been "spidered" but not yet "indexed." Until it is indexed -- added to the index -- it is not available to those searching with the search engine.

Search engine software is the third part of a search engine. This is the program that sifts through the millions of pages recorded in the index to find matches to a search and rank them in order of what it believes is most relevant

Major Search Engines: The Same, But Different

All crawler-based search engines have the basic parts described above, but there are differences in how these parts are tuned. That is why the same search on different search engines often produces different results

[Back To Top]

 

[Home]   [Link To Office Schedule]

This Page Last Updated on: 01/21/2014                          E-Mail: hherrington@pccua.edu