"The Leiden Manifesto" – ten principles for a better research evaluation

As a researcher, your research is regularly and in different contexts evaluated using bibliometrics. But how reliable are the bibliometric methods? Are they being applied in a correct manner? And do they perhaps have a tendency to be over-interpreted?

There is a scepticism towards bibliometrics among many researchers – but now there are sound recommendations published, packaged in a manifesto comprising ten guidelines!

The Leiden Manifesto for Research Metrics from Diana Hicks on Vimeo.

April this year saw the launch of the Leiden Manifesto for research metrics, published in Nature, which has been written by five renowned researchers who are, in one way or another, involved in research evaluation. The name comes from the manifesto having been drafted and adopted at the 2014 STI conference in Leiden. Leiden University is also home to CWTS, which is a world leading research centre within bibliometrics.

But why write a manifesto on a subject like quantitative research evaluation?

According to the authors, the problematic elements are as follows: Quantitative methods for evaluating research, such as bibliometrics, have become an integral and natural part of the research sphere today, and are used by decision makers at different levels to assess research quality. At the same time, bibliometrics are often applied uncritically, and without an awareness of the methodological limitations.

They concisely sum it up by stating the following:

"Metrics have proliferated: usually well intentioned, not always well informed, often ill applied."

They have therefore now formulated a checklist which consists of ten points that excellently summarise how quantitative research evaluation should be used and the pitfalls to look out for.

"We offer this distillation of best practice in metrics-based research assessment so that researchers can hold evaluators to account, and evaluators can hold their indicators to account."

So, what is it like at Karolinska Institutet and Stockholm County Council, bodies that utilise a common model for applying methods such as bibliometrics when evaluating and allocating resources? Do we adhere to the "best practice" recommended in the manifesto?

I put three questions to Catharina Rehn, coordinator of bibliometrics activities at the University Library:

1. What do you think about the manifesto in general?

We attended the conference where the manifesto was initiated, and there was a large consensus on the guidelines. For a bibliometrics specialist, the points are neither especially new nor particularly controversial , but I think that it is a good initiative to compile the principles in a relatively accessible way. Increasing numbers of researchers are endeavouring to produce or use bibliometric methods and results, and it is good for beginners to have a reference to go by.

2. Do we follow the ten principles in the manifesto when evaluating the research at KI and SCC?

Yes, I believe so. In terms of the analyses carried out with the help of the bibliometrics system at the University Library, we have for many years had our own set of principles that are fully in line with those listed in the manifesto, and I feel that we are in tune in this regard when it comes to both clients and steering groups.

3. Lastly, what do you see as the major challenge for bibliometrics in the future?

Bibliometric methods are, and must be, constantly changing, and there is much that needs to be done to improve methods and indicators. But I think the biggest challenge is to create more widespread awareness of how bibliometrics can and should be used in a nuanced and appropriate manner and, in extension, to enable the scientific community to understand and feel confident about how their output is measured and analysed. The Leiden Manifesto is definitely of great value in this context.

Hicks D, Wouters P, Waltman L, de Rijcke S, Rafols I. Bibliometrics: The Leiden Manifesto for research metrics. Nature. 2015 Apr 23;520(7548):429-31.

Carl Gornitzki

Librarian engaged in research support in various forms. Coordinating the search consultation service. Teaching medical students.