Evaluating sources

The Evidence Pyramid

To facilitate the evaluation of scientific articles, a so-called evidence pyramid is often used. The strength of the scientific foundation increases the higher up in the pyramid a scientific article is found.

Systematic reviews are usually placed at the top of the evidence hierarchy and hold a special position in evidence-based healthcare. In a systematic literature review, researchers have systematically worked to find and evaluate all articles relevant to a clinical question. Many review articles include a meta-analysis, meaning that data from several articles have been combined and a new statistical analysis has been performed.

Systematic reviews generally form the scientific basis for national and regional guidelines.

Scientific articles

Evaluating search results

Perhaps your search has resulted in more articles than you can read, and you need to make an initial evaluation to choose which articles to proceed with.

• Relevance. Does the article align with your research question?
• Timeliness. Is the information in the article still current? Are there newer articles on the topic that you might choose instead? The importance of the timeliness requirement can vary across different subject areas and also depends on how much has been written about a topic. A maximum of 5 or 10 years is a common delineation.
• Study population. Is the same group being studied that you are interested in so that the results are relevant to your research question? Are they of the same gender, age, and do they have similar characteristics otherwise?
• Study type. What type of study is it? What method has been used? Different types of studies are suitable for different purposes. Your supervisor or teacher can advise you on which study types you should primarily look for concerning your research question. You can read more about study design in literature on research methodology.
• Study size. How large is the study? A large study carries more weight than a small one and can yield more reliable results.
• Study quality. Is the study well-executed and clearly described in the article? To fully assess this, you need to read the whole article, but you can make an initial assessment based on the article's abstract.
• Central articles. Does it seem that the article is central within the subject area? One sign of this could be that it appears as a reference in other articles on the same topic. Keep in mind that completely new articles may not have been cited at all yet, and that older articles usually have accumulated more citations.

Critical Quality Review

The next step is to critically review the articles you read, taking a closer look at study design, study quality, and results. The purpose is to assess whether the study's results are reliable or if there are any shortcomings in the method that may have affected the results in any way.

In the process of creating a review, the evaluation is usually done using a review template or checklist. There are many different templates available for reviewing scientific articles. Many programs/courses provide guidelines on which templates should be used.

In the course book How to do a systematic review in nursing, there is a review template that can be used for different types of studies (both qualitative and quantitative). The template is reproduced in the book (but the original source is Caldwell, Henshaw & Taylor, 2011).

Peer review and impact factor

When evaluating a scientific article, it can also be a good idea to assess the journal in which the article was published.  

• In which journal has the article been published? Who publishes the journal?
• Does the journal apply peer review before publication?
• Is it a reputable journal in the field? What is its impact factor? 

Impact factor 

The impact factor is a measure of how often articles in a particular journal are, on average, cited. The impact factor for a particular journal should always be compared with other journals within the same subject area.

The basis for the impact factors is the citations in the Web of Science database. Thus, only journals included in Web of Science have impact factors. A journal can, therefore, very well lack an impact factor but still be important in its field.

You can find the impact factors of journals in the Journal Citation Reports database or, through individual articles, in Web of Science.

If you are about to publish your own article, you can read more about Strategic publishing.

Peer Review

Peer review is a process where scientific publications are read and evaluated by subject experts before they are accepted for publication. In summary, it is a form of quality assurance that ensures that the published research maintains a high standard.

There are different ways to see if an article is peer-reviewed:

1. On the journal's website. The information can be found, for example, under headings such as “Journal Information” or “About the journal”. However, if no information is found there, it does not necessarily mean that the article is not peer-reviewed, there are more ways to verify this. Keep in mind that journals generally contain a number of different document types. Editorials, letters, news, and comments are part of the document types that can be included in a journal without undergoing a structured peer-review process.
2. The Ulrichsweb database contains detailed information about over 300,000 journals of various types. Ulrichsweb is accessible via our database list. When you search for a journal, there is a tab called “Additional title details” where there may be information about whether a journal is peer-reviewed. This is indicated either by the term “refereed” or “peer-reviewed”. However, the same applies here as above, meaning that even if there is no mention of peer-review, it does not necessarily mean that the journal is not peer-reviewed.

3. In some databases, there is an option to limit to peer-reviewed material. Unfortunately, PubMed does not have a feature that provides information on peer review. Most journals in PubMed are peer-reviewed, but if you want to be certain whether a journal is peer-reviewed, you have to find out through another source. A database that offers information on peer review is CINAHL. 

4. One last possible option is to contact the journal or publisher and ask them to confirm whether the journal and the article in question are peer-reviewed. 

Many articles have something like 'Accepted' or 'Submitted' followed by a date. In many cases, these articles have undergone peer review, but it is not an absolute guarantee that this is the case. It may mean that the article has been accepted for publication without going through a peer review process.

Grey literature and other ways of publishing science

Research results are published in other ways than in scientific articles and books. Grey literature is the type of material that cannot easily be categorized into the conventional publication types released by regular publishers. It can include reports, dissertations, manuscripts, clinical guidelines, produced by authorities, universities, and other organizations and companies.

Scientific theses are written works presented at a disputation and lead to a licentiate or doctoral degree. In the biomedical field, compilation theses are the most common. Compilation theses consist of a number of published articles and a cover, that is, an introductory text that integrates the various articles. In compilation theses, the included articles have undergone the usual review process at their respective journals prior to publication. Theses can also consist of a single coherent text. 

In connection with a conference, the contributions presented at the conference are often published in a conference proceeding or conference publication. The contributions can take the form of abstracts or full articles. The results may be more preliminary than in a scientific article, and the contributions have not always undergone peer review. Often, this is where one can find the very latest research.

Reports or research reports present research results from, for example, authorities, organizations, research institutions, and companies.