Recommendations concerning the use of bibliometric indicators in the assessment of publications of individual researchers

Research Strategy Committee

R&D PARTNERS

Doc. no. 2-1242/2013
5/3/2013
Version 1.0

In recent years, there has been a clear increase, both nationally and internationally, in the use of bibliometrics. It has now become an established method for, through statistical analyses of publication data, the quantitative and qualitative analysis of research output. On larger quantities of publications, high bibliometric indicators have been shown to correlate with high appraisals from expert panels. The methods, however, are less suitable for the assessment of individuals or smaller groups. It is unusual for these to achieve a publication quantity sufficient for the results to be reliable and stable.
At Karolinska Institutet/SLL there is the possibility for all employees to order analyses from the organisation’s own bibliometric system. The purpose of the bibliometric system is

• to provide surveys of Karolinska Institutet’s/SLL’s research output as well as tools for comparing it with the rest of the world
• to enable one as individual researcher to enhance one’s impact
• to supply background data to assist resource allocation for departments/clinical departments within Karolinska Institutet/SLL

With good knowledge of the limitations existing at the level of the individual, certain bibliometric measures can also be used to supplement visual inspection of an individual’s publication lists. Since it may be difficult for researchers themselves to find these measures for their published papers in databases such as Thomson Reuters Web of Science, there is an advantage for them in being able to obtain data instead from the bibliometric system, even if the majority of measures can be produced also from a more publicly accessible system.

Indicators can be used at the individual level

Where bibliometric measures are to be applied to the assessment of individuals or groups, these shall either give their approval or be encouraged to send in the material themselves.

Where it is intended to call in bibliographic measures from individuals within Karolinska Institutet/SLL, these measures are recommended

• Number of publications¹
• Total Journal Impact Factor²
• Number of publications amongst the world’s 5% most cited, field adjusted³
• Average citation count for the 5, 10 or 15 most highly cited publications⁴
• Average Journal Impact Factor (JIF) for the individual’s 5, 10 or 15 articles with highest JIF⁵
• Field-adjusted Journal Citation frequency (JCf), on average, for the individual’s 5, 10 or 15 articles with JCf⁶

The composition of the above indicators provides a balance between those measures made at the journal level and article level, those measures dependent on size and those independent of size as well as field-adjusted measures and ‘raw’ citation figures, thereby illuminating several aspects of the individual’s pattern of publications.
It is not appropriate to calculate the average values for the entire sum of publications of individual researchers, since this would give an undesirable incentive for publication and verification. On the other hand, it may be of interest to look at the average for the individual’s 5, 10 or 15 highest indicator values for each individual indicator to identify whether the researcher has the potential to produce e.g. very well cited publications. When the link between peer review and bibliometric measure at the individual/group level is studied it is shown, moreover, that the extra information that can be deduced from the average values for the entire number is small.
Whether one should use 5, 10 or 15 publications in the sample for average value indicators depends on the expected number of publications in the person’s publication list. Unless very productive researchers are concerned, 5 is usually an appropriate number.

For a more complete bibliometric picture it is also possible to supplement this with one or more measures

• Total number of citations for the individual’s articles ⁷
• Average field-adjusted citation percentile (Perc Cf) for the individual’s 5, 10 or 15 articles with highest Perc Cf ⁸
• Total field-adjusted citation percentile (Perc Cf) for all the individual’s analysed articles⁹

In certain cases one might wish for indicators for individual publications in a person’s list. A suitable measure, in that case, is the number of citations and known Journal Impact Factors for the journal that the article is published in.
Reviewers should only take the bibliometric measures into account after a traditional review of the CV and publication list has been made, since the intention with a bibliometric element in an assessment of individuals should be to come up with information complementary to that which is already known through the application.

H-Index

The h-index is a measure that has attracted attention and whose primary purpose is to illustrate an individual researcher’s impact. It has the advantage that it is relatively simple for the individual researcher to find but, on the other hand, it is hard to interpret correctly. There is an in-built bias in favour of researchers with long careers and within highly cited fields of research. Moreover, the value is stable, or indeed increases, even for those persons who are no longer being published or who, in recent years, have had a low number of citations for their production. While certain attempts have been made to enable the indicator to take account of these factors, the advantages of a combination of the more advanced bibliometric measures outweigh these at present.

Period subject to analysis

The bibliometric system contains data from 1995 and later and cannot, therefore, cover the entire active period for more experienced researchers. However, what is desired most often is an overview of the most recent years. A 10-year period is normally suitable at the individual level for the number of publications analysed to be sufficient in number. For more productive researchers, however, a 5-year period may be relevant. The citation indicators do not offer a true and fair picture until a year at the earliest after publication and several indicators are consequently left out for the most recent 1-2 years.
The time interval from when articles from different journals are published until they appear in the Web of Science databases varies very greatly between different journals. It is, therefore, in most cases, inappropriate to include the current year even where other indicators are calculated.

Document types for analysis

There is an international consensus that Articles and Reviews, above all, shall be included in bibliometric citation analyses and that in field-adjusted indicators these shall each be treated separately.¹⁰ This also characterises the standard for analyses supplied from the bibliometric system.

Factors that influence the evaluation of bibliometric analysis results

In the evaluation of a bibliometric analysis at the individual level, one must always take into account

• the person’s active research period and the number of publications – a large number of publications offers greater scope for obtaining high values, also on measures of mean value
• which subject area the person is active in – the citation pattern varies sharply between different disciplines. Journal categories and/or MeSH terms for the publications can give some guidance as to the area for the researcher in question.
• whether the person has been published mainly in general or narrowly focused, subject-specific journals (since this affects the balance between JCf and JIF)
• that good bibliometric indicators can frequently identify good quality, but not necessarily the reverse
• whether the person has been mainly responsible for the essential work
• whether a large share of the highly cited articles may be considered to be reviews or some other type of published paper that, as a matter of course, can be expected to receive more citations than regular, original articles
• possible occurrence of negative citations¹¹ or a high rate of self-citation

These factors are only partly taken into account, or not at all, where existing bibliometric methods are used.