The Evaluation of Research by Scientometric Indicators

Dr Vinkler is respected for his work in both chemistry and bibliometrics:

Vinkler defines scientometrics as:

From the many citations through the text, a reader may get the impression that, like a latter‐day Disraeli, if Dr Vinkler needs a good reference then he writes it himself. Spot‐checking several such self‐references for broad statements (e.g. to his own 1998 paper for the assertion that “References made by the authors can be assumed to be proof of the use of the information in the publication references”), reveals that Vinkler's papers adequate recap that particular statement – even if sometimes somewhat dated or superficially. Nonetheless, when finer details later become necessary (for instance, the motives for making references and the relative frequency of those motives), Vinkler succinctly summarises, often in clear tables, the results of appropriate research including material published after his own broad reference. Conversely, sometimes Vinkler is indeed the origin or major source on the concept discussed (e.g. his π and π_V‐indexes).

Some of Dr Vinkler's work is well‐cited (a superficial search giving him an h‐index of 15 and a g‐index of 23), however his idiosyncratic suggestion that [two‐year] impact factor be instead called the Garfield Factor (GF) has been solidly ignored by others in the field, and will naturally remain ignored as long as Journal Citation Reports use the more tradition term – even though Garfield Factor obviously makes it easy to divorce concepts of “impact” from that otherwise ubiquitous factor.

After reading chapter 2 (pp. 7‐9: “Basic categories of scientometrics”) and at least skimming chapter 3 (pp. 11‐21: “Classification of the indicators …”), the reader is in a position to look up specific indicators or topics of interest in the subject index. Nonetheless, the book is best used in its printed order – especially if the reader wishes a good overview of scientometrics. Publication growth naturally leads to both journal eminence and the ageing of scientific information, which give the tools to consider indicators assessing publications; and so on. It may seem that a myriad of terms, some rather contrived, are foisted upon the reader – but Vinkler's standard technique is to introduce technical terms and indicators at the point where they enhance the reader's understanding or are a necessary step to the next, core concept. Admittedly, the reviewer often failed to appreciate the relevance or importance of a term when it was first introduced: should other readers feel the same, may one recommend that they lightly skim that section and carry on in the assurance that the significance will become apparent later?

The concluding chapters are topics that are core to Vinkler's interests (standardising scientometric indicators and assessments; applying them to science policy; and modelling how scientific information is institutionalised) and which reflect his expertise. In discussing the grant process, he emphasises that time and effort on participants, successful or otherwise, is considerable:

The boggy field of journal eminence or prestige is a danger to all charge into it on a high horse; but Vinkler leads the reader by smaller, measured steps and the gains made are illustrated with pertinent examples. He begins his chapter on the subject with “Relevance, validity and applicability are the main factors used to characterize the scientific eminence of journals”. After quickly explaining those points, he continues, “The citedness of publications may be used to represent the extent of use”, and shortly arrives at the “impact factor” (Garfield Factor). There is the perfunctory warning about the multitude of measure of impact, and the conflicts between them [which is picked up several chapters later] and the Garfield Factor (GF) is clearly defined and an example given. Which, as a defined mean, leads on to how diversely papers are cited and the GF is then proven to be the “mean chance for citedness” – which is not strictly true, since the GF is actually a biased overestimate, due to its inflated numerator (Seglen, 1997, p. 500). At this point, Vinkler brings us back to concrete examples: the mean number of references per paper has increased over time, and the GF [impact factor] also increased, although this does not mean that the proportion of recent citations has increased … as is illustrated by yet another example. And the chapter continues similarly.

Vinkler states “it may be concluded that the GF of journals is an appropriate scientometric measure for characterising relative international eminence within a set of journals with similar bibliometric features” (p. 51); however the subsequent list of bibliometric features is long and detailed (pp. 163ff), appeasing even the harshest critic of impact factors.

Vinkler deliberately ignores almost all of the indicators of journal eminence, other than a decidedly incomplete table (5.14) of “some indicators and methods”. No definition, explanation, or interpretations are given. That chapter, “Scientific eminence of journals”, ends:

It should be noted, actually, that Vinkler's writing style and logical argument normally flow more elegantly than in this example. Why mention those indicators, if no explanation is given? Why not add the corresponding values to the various tables of journal indicators and add brief descriptions to the text earlier in the chapter. Similarly Vinkler ignores the SCImago Journal Rank (Falagas et al., 2008), or SJR, which may make his book seem dated now that SJR and Source‐Normalized Impact per Paper (SNIP) are available in Scopus (produced by Elsevier) from January 2010 and later. The [extended] h‐index appears as a measure of journal prestige only to show how inferior it is to Vinkler's own π_V‐index. Whereas the g‐index, superior in that aspect to the h‐index, does not even make it into Table 5.14 – although common sense demands, and the references of his original papers on the π and π_V‐index prove, that Vinkler is quite aware of the g‐index.

However, in Vinkler's defence, there are instructive advantages in concentrating at this point on only two diverse indicators of journal eminence or prestige that generate surprisingly similar rankings: a long‐established, apparently “average” measure of impact [the Garfield Factor] and a novel indicator derived solely from the small, high‐performing elite set of papers [the π_V‐index]. Moreover, other GF‐derived indicators are discussed in later chapters.

The editorial proofing is to a high standard. Typographical errors are rare and insignificant (e.g. “power low” for “power law”). Grammatical irregularities are minor. The author's meanings are clear; in the rare case when the reviewer was confused by a phrase or sentence, the successive sentence or two would clarify the sense of the puzzling passage.

Although the mathematics is occasionally rough or used to illustrate a trivial statement (e.g. pp. 28‐30), the definitions of terms are clear and the corresponding formula definitions use consistent and unique codes, and the same codes are also clearly explained when used in the many tables. Despite a close examination, no error was detected in the equations. In the rare case where a previously used symbol was slightly ambiguous, the precise meaning was often explicit in the text and/or obvious in context –, e.g. whether “t” was being measured forward or backwards; whether “y” indicated a specific epoch/date or a relative count of the year with respect to a reference period.

This exemplary practise, together with a thorough subject index, make it easy for the reader to leap ahead to a topic of interest or, more likely, to refer back to technical terms that have been briefly forgotten. There is also a comprehensive author index. The reviewer's only quibble is that – unlike, for example, Moed (2005) – the dates of the various authors' work are not indicated in that index, making it difficult to follow Vinkler's use of items of interest, especially for heavily cited authors such as Braun, Glänzel, Moed, etc.

Dr Vinkler has produced a short, well‐written book that covers a lot of ground. Combining theory with apposite examples, “The evaluation of research by scientometric indicators” does not require of its readers any detailed prior knowledge or advanced mathematical skills, while being relevant to scientific researchers and their support staff, including librarians and science administrators.