The purpose of this paper is to compare five major web search engines (Google, Yahoo, MSN, Ask.com, and Seekport) for their retrieval effectiveness, taking into account not only the results, but also the results descriptions.
The study uses real‐life queries. Results are made anonymous and are randomized. Results are judged by the persons posing the original queries.
The two major search engines, Google and Yahoo, perform best, and there are no significant differences between them. Google delivers significantly more relevant result descriptions than any other search engine. This could be one reason for users perceiving this engine as superior.
The study is based on a user model where the user takes into account a certain amount of results rather systematically. This may not be the case in real life.
The paper implies that search engines should focus on relevant descriptions. Searchers are advised to use other search engines in addition to Google.
This is the first major study comparing results and descriptions systematically and proposes new retrieval measures to take into account results descriptions.
