Search Engines

A brief History

The first automated Internet search engine was MIT student Matthew Gray’s World Wide Web Wanderer. The Wanderer robot was intended to track the growth of the Web counting only web servers initially. Soon after its launch it captured URLs as well. This list formed the first database of websites, called Wandex. Robots at this time were quite controversial. For one, they occupied a lot of network bandwidth and they would index sites so rapidly it was not uncommon for the robots to crash servers.

The first real modern search engine as we know it today was Yahoo!  Yahoo! grew out of two Stanford University students, David Filo’s and Jerry Yang’s, web pages with their favorite links (such pages were quite popular back then). Introduced in  April 1994 as a way to keep track of their personal interests, Yahoo soon became too popular for the university server.

Yahoo’s user-friendly interface and easy to understand directories have made it the most used search directory. But because everything is reviewed and indexed by people, their database is relatively small, accounting for approximately 1% of web pages

But then came the big guns! Late in December of 1995 the second search engine appeared. Its roots came from before Yahoo!'s founders graduated from elementary school. But when a search failed on Yahoo! it defaulted to this search engine for results. AltaVista was from Digital Equipment Corporation (known as DEC) who was the second largest computer company in the world in the early 1980's. They were most noted for not only being an innovative company with a lot of firsts, but for always being one of the top three computer service companies in the computer industry. Sharing that position on a rotating basis with IBM and HP.

Digital had a massive customer support center in Colorado Springs which accumulated massive amounts of data on every service call including the symptoms, the problem and the fix. But doing a text search through its volumes of data was time consuming and if you did not put in the exact text as was in the problem description it would not find it. So a group of DEC employees at the "Spings" developed a search and indexing method that allowed for natural language queries as well as Boolean search techniques. And it was fast. This was the early 1980's In 1995 they adapted it to not only search their huge network based data base, but to go out to the internet and do the same thing. Thus AltaVista was born.

AltaVista was not only big, but also fast. And to aid in this, it was the first to offer "Tips" for good searching prominently on the site. These advances made for unparalleled accuracy and accessibility.

Next came HotBot, introduced May 20, 1996 by Paul Gauthier and Eric Brewer at Berkeley. Powered by the Inktomi search engine, it was initially licensed to Wired Magazine website. It has occasionally boasted it can index the entire Web. Indexing 10 million pages per day, it is the most powerful search engine.

Google was running as a research project at Stanford University since late 1997, taken live by the developers in 1998, and is currently the number one used search engine today (2006).