The purpose of this paper is to examine the significance of the differences between the actual technical principles determining relevance ranking, and how relevance ranking is understood, described and evaluated by the developers of relevance ranking algorithms and librarians.
The discussion uses descriptions by PLWeb Turbo and C2 of their relevance ranking products and a librarian's description on her blog with the responses which it drew, contrasting these with relevancy as it is indicated in studies of the ISI citation record reported by White.
The study finds that product descriptions and librarians consistently use the term “relevance ranking” to mean both the artificial relevance ranking by statistical methods using various surrogates assumed to reliably indicate relevance and the real relevance as determined by the searcher. The paper indicates the misunderstandings arising from this terminological confusion and its consequences in the context of the invalid user models and artificial searches which accompany discussions of “relevance ranking”.
Evaluations of relevance ranking must be based on real users and real searches. Theorising relevance as a judgement about information rather than a property of information clarifies many issues.
The design of search engines and OPACs will benefit from incorporating metadata that contain indications of user‐determined relevance.
The activity of subject analysis and indexing by human beings is presented as an activity identical in kind to the real searcher's determination of relevance, a definite statement of relevancy arising from a real communication situation rather than a statistically indicated probability.
