WebCrawler is one of the first crawler-based search engines. It was developed by Brian Pinkerton during his studies at the University of Washington. WebCrawler was acquired by AOL in 1995, passed to Excite in 1997 and sold to InfoSpace in 2001. InfoSpace used the name but not the crawler, and switched to a metasearch instead. Today Bing is used for the search.

In 2001 the search engine WebCrawler was bought by InfoSpace. Since then, the search engine has been offered as a search portal with a web, news and photo search. In 2004 the photo search was renamed to picture search and an audio and video search was added. The audio search was stopped in January 2009. Until January 2016 WebCrawler worked as a meta search engine. Bing's search tools have been used since 2016.

Language English

Launched April 20, 1994
Closed No

Developer Pinkerton, Brian

Country of Origin US America

1994 - 1995 Pinkerton, Brian
1995 - 1997 AOL
1997 - 2001 Excite (later Excite@Home)
2001 - [...] InfoSpace (In 2012 Infospace was renamed Bluecora, in 2016 the Infospace business was sold to OpenMail, which renamed itself System1 in 2017.)

Topic Universal

Region No Limitation

Technical functionalities
Robot/Crawler based, algorithmic search
Meta search engine
Search Portal

Used SeEn WebCrawler
Google Search Engine
MSN Search
MSN Live Search
Ask Jeeves / Ask SeEn

Robot: ArchitextSpider (Source)

Older Version Internet Archive / WebCite

WebCrawler Timeline »January 27, 1994 Brian Pinkerton, a CSE student at the University of Washington, starts WebCrawler in his spare time. At first, WebCrawler was a desktop application, not a Web service as it is today. WebCrawler spat out its first Top 25 list on March 15, 1994.

April 20, 1994 WebCrawler goes live on the Web with a database containing pages from just over 4000 different Web sites. [...] About a month and a half later, I announced WebCrawler on comp.infosystems.announce, the Usenet group where new Web sites were announced.

November 14th, 1994 WebCrawler serves its 1 millionth query ...

December 1, 1994 WebCrawler acquires two sponsors, DealerNet and Starwave. Both companies provided money to help keep WebCrawler operating. WebCrawler was fully supported by advertising on October 3, 1995 but maintained a strict separation between the advertising and the search results.

June 1, 1995 America Online acquires WebCrawler. At the time of the acquisition, AOL had fewer than 1 million users, and no capability to access the Web. It was believed that AOL's resources could help make the most of WebCrawler's future.

September 4, 1995 Spidey is born. In the first of WebCrawler's many design changes, we moved to a new look, with Spidey as the mascot. Spidey took on many personalities over the years and exemplified the fun, lighthearted spirit that WebCrawler strove for.

April, 1996 WebCrawler extends its functionality from pure search to include the best human-edited guide for the Web: GNN Select. Formerly known as the Whole Internet Catalog, GNN Select was the editorial product of a small team of Internet-savvy researchers headed by Abbot Chambers.

April 1, 1997 Excite acquires WebCrawler. AOL sold WebCrawler to the Mountain View, CA company Excite. WebCrawler was initially supported by its own dedicated team within Excite, but that was eventually abandoned in favor of running both WebCrawler and Excite on the same back end.

2001 InfoSpace acquires WebCrawler. Excite, now Excite@Home, went belly up. In the bankruptcy, Infospace acquired WebCrawler. Today Infospace runs WebCrawler as a meta-search engine. And they've given Spidey a new name and turned him purple!« Source

Danny Sullivan (2004): »4/20/94: WebCrawler launches with information from 6,000 different web servers. It is a project by Brian Pinkerton, at the University of Washington.«
»Oct. 1994: WebCrawler is serving 15,000 queries per day.«
»3/29/95: AOL buys WebCrawler in March, and it moves from the University of Washington on this date.«
»Jan. 1996: WebCrawler replaces Magellan on Netscape Net Search page.«
»Nov. 1996: Excite acquires WebCrawler« »6/16/97: WebCrawler relaunches with new look, the first major changes since Excite acquired the service.«
»9/15/97: WebCrawler debuts new channels, following the lead of sister search engine Excite. It also announces a 2-year partnership with bookseller Barnes & Noble. This follows a recent partnership formed in July between Excite and Amazon, Barnes & Noble's chief competitor.«
»Dec. 97: WebCrawler stopped using retrieval technology from PLS and shifts over to using Excite's own technology.«
»3/9/98: WebCrawler redesign lauched, debuting "friendlier" look« Source

Sullivan, Danny (Mar 3, 2003): »Ch-Ch-Ch-Changes: ... WebCrawler (1994; reborn 2001): WebCrawler still exists as a meta search engine that gets results from other search engines, rather than through its own efforts. Now owned by Infospace, WebCrawler was arguably the web's first crawler-based search engine in the way we know them today. It launched in early 1994 as a University of Washington research project, was purchased by AOL in 1995, then sold to Excite at the end of 1996. The WebCrawler spider was deactivated in December 2001.« Source

Sherman, Chris (Mar 22, 2005): »Formerly a crawled-based search engine owned by Excite, Webcrawler was acquired by InfoSpace in 2002 and uses the same underlying technology as the other InfoSpace meta search engines, but offers a fast and clean, ad-free interface.« Source

wiseGEEK: »In 1994, though, the world of the search engine had a major breakthrough. A company called WebCrawler went live with a search engine that not only captured the title and header of pages on the Internet, but grabbed all of the content as well. WebCrawler was enormously successful — so successful that a great deal of the time it couldn’t even be utilized because its system resources were all being used.« Source

Sherman, Chris (Sep 9, 2003): »Created by University of Washington student Brian Pinkerton, Webcrawler subsequently was purchased by Excite, AOL, Excite again, before finally ending up in the hands of InfoSpace, which uses its name (but not technology) today.« Source

Pinkerton, Brian (1994): »I finally got a forms query interface hacked together for the WebCrawler index. Give it a try, at [...] This index was assembled in slightly less than 24hrs of WebCrawling, so it's not particular deep. On the other hand, it seems to have reasonable breadth, so general search terms should work well. It's not fast yet, either. MMOC (mere matter of code).« Source

Example results page for "digital camera technology" (2005): Source

Example results page for "digital camera technology" (2007): Source

»WebCraweler was the first search engine to provide full text search. It was bought by AOL in June 1995. It still receives over 6 million monthly visitors.« Source


Critical points

Features & Functionality

»Phrasensuche möglich« Source


Example results page for "sand": Source

References & further Publications

Wikipedia (EN):
Wikipedia (Others):

Other Sources

WebCrawler. Online available at SEO Logic. URL:
WebCrawler Timeline. Online available at URL:
Danny Sullivan (2004): Search Engine Timeline URL:
Sullivan, Danny (Mar 3, 2003): Where Are They Now? Search Engines We've Known & Loved URL:
About WebCrawler URL:
Frequently Asked Questions URL:
WAYNE E. HARDING (1997): How To Find It On The Net URL:
Metamend: WebCrawler Search Engine Optimization Strategy URL:
Brian Pinkerton (1994): Finding What People Want: Experiences with the WebCrawler URL:
Brian Pinkerton (2000): WebCrawler: Finding What People Want URL:
History of Domain Names : WebCrawler URL:

Created: 2013-01-22