Evaluating sources
Information exists in many different types of sources and different sources are suitable for different purposes. Assess whether the source contains information that is relevant to your purpose, but also evaluate the reliability of the source.
Quick checklist for evaluating information
- Who is behind the information? Does the person/organization have expertise in the field?
- What is the purpose of the information? To inform, persuade, or promote something?
- When was the information published or updated? Is the information still current?
- Has the information been reviewed before publication?
- Are there sources and references? Are these sources current and relevant?
- To assess whether a report or conference publication is scientific, you can also investigate whether there is a transparent account of the research process.
The Evidence Pyramid
To facilitate the evaluation of scientific articles, a so-called evidence pyramid is often used. The strength of the scientific foundation increases the higher up in the pyramid a scientific article is found.
Systematic reviews are usually placed at the top of the evidence hierarchy and hold a special position in evidence-based healthcare. In a systematic literature review, researchers have systematically worked to find and evaluate all articles relevant to a clinical question. Many review articles include a meta-analysis, meaning that data from several articles have been combined and a new statistical analysis has been performed.
Systematic reviews generally form the scientific basis for national and regional guidelines.
Scientific articles
Evaluating search results
Perhaps your search has resulted in more articles than you can read, and you need to make an initial evaluation to choose which articles to proceed with.
- Relevance. Does the article align with your research question?
- Timeliness. Is the information in the article still current? Are there newer articles on the topic that you might choose instead? The importance of the timeliness requirement can vary across different subject areas and also depends on how much has been written about a topic. A maximum of 5 or 10 years is a common delineation.
- Study population. Is the same group being studied that you are interested in so that the results are relevant to your research question? Are they of the same gender, age, and do they have similar characteristics otherwise?
- Study type. What type of study is it? What method has been used? Different types of studies are suitable for different purposes. Your supervisor or teacher can advise you on which study types you should primarily look for concerning your research question. You can read more about study design in literature on research methodology.
- Study size. How large is the study? A large study carries more weight than a small one and can yield more reliable results.
- Study quality. Is the study well-executed and clearly described in the article? To fully assess this, you need to read the whole article, but you can make an initial assessment based on the article's abstract.
- Central articles. Does it seem that the article is central within the subject area? One sign of this could be that it appears as a reference in other articles on the same topic. Keep in mind that completely new articles may not have been cited at all yet, and that older articles usually have accumulated more citations.
Critical Quality Review
The next step is to critically review the articles you read, taking a closer look at study design, study quality, and results. The purpose is to assess whether the study's results are reliable or if there are any shortcomings in the method that may have affected the results in any way.
In the process of creating a review, the evaluation is usually done using a review template or checklist. There are many different templates available for reviewing scientific articles. Many programs/courses provide guidelines on which templates should be used.
In the course book 'How to do a systematic review in nursing,' there is a review template that can be used for different types of studies (both qualitative and quantitative). The template is reproduced in the book (but the original source is Caldwell, Henshaw & Taylor, 2011).
A scientific article is one where research is presented (original articles), compiled, or reviewed (review articles).
The article is published in a scientific journal and thus becomes accessible to other researchers and interested parties. Scientific journals have an editorial board that decides which of the submitted articles will be published.
Most scientific journals apply peer review, where submitted articles are reviewed and approved by experts in the field.
Scientific articles are characterized by a formal, objective style. The research process should be described clearly enough that other researchers can follow how the researcher proceeded. This structure is commonly referred to as IMRoD, an acronym formed from the article's four main parts:
- Abstract. An abstract is a brief summary of the article's content. Often, the abstract is structured in the same sections as the entire article: introduction, method, results, and discussion. Alongside the article, the author's keywords that describe the article's subject are often found in a further condensed form.
- Introduction. The introduction describes the purpose of the study and the author's problem statements/research questions. Any delimitations are specified. Sometimes, the most important results are also presented. Background facts that help the reader to understand the problem area are usually included in the introduction.
- Method (sometimes Materials and method). The method section describes which method(s) have been used – quantitative or qualitative. The description should be detailed and precise enough for the research process to be followed and repeated. The methods should be reasonable in relation to what is to be investigated.
- Results. Here, the results of the research are presented in an overview description without repeating what has previously been stated in the article. Important data are presented in running text or in tables and figures. Unexpected or negative results are also reported.
- Discussion. In this section, principles, relationships, and generalizations that are supported in the results section are discussed. Exceptions, lack of correlations, and weaknesses in the study are highlighted. Here, scientific theoretical, ethical, and societal issues may arise. The conclusions from the research must be easy to follow in relation to the observed data.
- Acknowledgements. Here, individuals who have provided various types of support during the course of the work are mentioned: personal, professional, technical, or financial. Everyone mentioned should be informed about this.
- References. The reference list collects all references mentioned in the article so that the reader can go back to the source.
Learn more
Read more about IMRaD and the structure of academic texts.
The table below presents a comparison between traditional and systematic reviews (taken and translated from Jesson, Matheson & Lacey, 2011, p. 105).
Traditional review | Systematic review | |
---|---|---|
Aim | To gain a broad understanding, and description of the field | Tightly specified aim and objectives with a specific review question |
Scope | Big picture | Narrow focus |
Planning the review | No defined path, allows for creativity and exploration | Transparent process and documented audit trail |
Identifying studies | Searching is probing, moving from one study to another, following up leads | Rigorous and comprehensive search for ALL studies |
Selection of studies | Purposive selection made by the reviewer | Predetermined criteria for including and excluding studies |
Quality assessment | Based on the reviewer's opinion | Checklists to assess the methodological quality of studies |
Analysis and synthesis | Discursive | In tabular format and short summary answers |
Methodological report | Not necessarily given | Must be presented for transparency |
Peer review and impact factor
When evaluating a scientific article, it can also be a good idea to assess the journal in which the article was published.
- In which journal has the article been published? Who publishes the journal?
- Does the journal apply peer review before publication?
- Is it a reputable journal in the field? What is its impact factor?
Impact factor
The impact factor is a measure of how often articles in a particular journal are, on average, cited. The impact factor for a particular journal should always be compared with other journals within the same subject area.The basis for the impact factors is the citations in the Web of Science database. Thus, only journals included in Web of Science have impact factors. A journal can, therefore, very well lack an impact factor but still be important in its field.You can find the impact factors of journals in the Journal Citation Reports database or, through individual articles, in Web of Science.If you are going to publish your own article, you can read more about Strategic publishing.
Journal Citation Reports (JCR)
Impact factors and publishing trends for scientific journals.
Access
KI only.
Web of Science
Multidisciplinary database with citation indexes covering sciences, social sciences, medicine and technology.
Access
KI only.
Peer Review
Peer review is a process where scientific publications are read and evaluated by subject experts before they are accepted for publication. In summary, it is a form of quality assurance that ensures that the published research maintains a high standard.
There are different ways to see if an article is peer-reviewed:
- On the journal's website. The information can be found, for example, under headings such as “Journal Information” or “About the journal”. However, if no information is found there, it does not necessarily mean that the article is not peer-reviewed, there are more ways to verify this. Keep in mind that journals generally contain a number of different document types. Editorials, letters, news, and comments are part of the document types that can be included in a journal without undergoing a structured peer-review process.
- The Ulrichsweb database contains detailed information about over 300,000 journals of various types. Ulrichsweb is accessible via our database list. When you search for a journal, there is a tab called “Additional title details” where there may be information about whether a journal is peer-reviewed. This is indicated either by the term “refereed” or “peer-reviewed”. However, the same applies here as above, meaning that even if there is no mention of peer-review, it does not necessarily mean that the journal is not peer-reviewed.
Ulrichsweb
Information on 300 000 periodicals of all types: academic and scholarly journals and e-journals with table of contents, popular magazines, newspapers, newsletters and more.
Access
KI only.
3. In some databases, there is an option to limit to peer-reviewed material. Unfortunately, PubMed does not have a feature that provides information on peer review. Most journals in PubMed are peer-reviewed, but if you want to be certain whether a journal is peer-reviewed, you have to find out through another source. A database that offers information on peer review is CINAHL.
4. One last possible option is to contact the journal or publisher and ask them to confirm whether the journal and the article in question are peer-reviewed.
Many articles have something like 'Accepted' or 'Submitted' followed by a date. In many cases, these articles have undergone peer review, but it is not an absolute guarantee that this is the case. It may mean that the article has been accepted for publication without going through a peer review process.
Gray literature
Research results are published in other ways than in scientific articles and books. Gray literature is the type of material that cannot easily be categorized into the conventional publication types released by regular publishers. It can include reports, dissertations, manuscripts, clinical guidelines, produced by authorities, universities, and other organizations and companies.
Scientific theses are written works presented at a disputation and lead to a licentiate or doctoral degree. In the biomedical field, compilation theses are the most common. Compilation theses consist of a number of published articles and a cover, that is, an introductory text that integrates the various articles. In compilation theses, the included articles have undergone the usual review process at their respective journals prior to publication. Theses can also consist of a single coherent text.
In connection with a conference, the contributions presented at the conference are often published in a conference proceeding or conference publication. The contributions can take the form of abstracts or full articles. The results may be more preliminary than in a scientific article, and the contributions have not always undergone peer review. Often, this is where one can find the very latest research.
Reports or research reports present research results from, for example, authorities, organizations, research institutions, and companies.
If you would like us to get back to you, please submit your contact information in the form below along with your feeback.