"I am preparing a systematic review" – this is something we frequently hear at KIB. But what exactly is a systematic review? How does it differ from other types of reviews? And how do you systematically search the literature?
If you are a Bachelor's or Master's student, you will find our guide Structured literature reviews – A guide for students useful. As a student you should rather do a structured literature review, including only certain aspects of the methodology described on this page.
According to the Cochrane Handbook, a "systematic review attempts to collate all empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question". A systematic review also adheres to a specific methodology and often includes meta-analyses, wherein the collected data is combined using statistical methods.
Since they summarise the results from all original studies within a given field, systematic reviews are commonly regarded as high quality evidence. A pyramid is often used to visualise this hierarchy of evidence, with systematic reviews placed at the apex of the pyramid.
In this pyramid, only study design variations have been included. Other, more complex pyramids add other types of layers to the systematic reviews, for example synopses of syntheses (see Alper & Haynes, 2016).
Since the total output of medical scientific literature produced every year is increasing at an exponential rate, systematic reviews that collate the available evidence have become increasingly important. Consequently, more and more systematic reviews are published every year, a trend evident in scientific databases.
This is one reason why articles summarising the literature are of such importance today. Accordingly, the number of published systematic reviews has also increased (see Bastian, Glasziou & Chalmers, 2010). Today, there are even reviews of systematic reviews (see for example Aromataris et al, 2015).
Nowadays, there are also journals – for example, The Lancet and Alzheimer’s & Dementia – that require primary studies provide a structured, comprehensive and documented literature search ("Research in context").
For more about systematic reviews in general, see the further reading section below.
What is the difference between a systematic review and a traditional review?
- It should be exhaustive: all relevant literature in a research field should be included.
- A rigorous methodology must be followed throughout – from defining the research question, writing a protocol and searching the literature, to gathering, screening and analysing. The entire process should also be thoroughly documented.
- At least two people should be involved, particularly for screening articles and extracting data.
- Plenty of time resources are needed, but also in terms of availing yourself of others' expertise – for instance in database searching – and tools and software.
There are a variety of review types, each using a more or less different methodology; the terminology can be confusing. In an article from 2009, Grant & Booth described 14 review types, for example scoping reviews, and their associated methodologies.
For a condensed overview, see the comparison below (from Jesson, Matheson & Lacey, 2011, p. 105).
|Traditional (scoping) review||Systematic review|
|Aim||To gain a broad understanding, and description of the field||Tightly specified aim and objectives with a specific review question|
|Scope||Big picture||Narrow focus|
|Planning the review||No defined path, allows for creativity and exploration||Transparent process and documented audit trail|
|Identifying studies||Searching is probing, moving from one study to another, following up leads||Rigorous and comprehensive search for ALL studies|
|Selection of studies||Purposive selection made by the reviewer||Predetermined criteria for including and excluding studies|
|Quality assessment||Based on the reviewer's opinion||Checklists to assess the methodological quality of studies|
|Analysis and synthesis||Discursive||In tabular format and short summary answers|
|Methodological report||Not necessarily given||Must be presented for transparency|
Systematic search techniques
Do you need search help? Read more about our support to students and researchers under contact us below.
It is also important to conduct the search in several databases. Cochrane recommends using at least three. In biomedicine and health, standard databases include PubMed/Medline, Embase and Cochrane Library. Depending on the topic, a multi-disciplinary database like Web of Science or subject specific databases, such as Cinahl, Psycinfo or Eric, should also be considered.
In addition, grey literature, for example dissertations and clinical trials, should be considered in your search. These publication types demand other databases. The literature search is usually also supplemented with a screening of reference lists, article suggestions from colleagues and sometimes with a citation analysis – i.e. an analysis of which articles have cited an older, still relevant study.
Sensitivity, specificity and precision
To construct an exhaustive search strategy, and at the same time avoid too many references to screen, is a challenging task. In library and information science, the concepts of sensitivity, precision and specificity are used. According to the Cochrane Handbook, sensitivity "is defined as the number of relevant reports identified divided by the total number of relevant reports in existence." A high sensitivity search strategy should thus retrieve all relevant studies on a topic. Precision and specificity state the part of non-relevant literature in the search.
Sensitivity and precision/specificity are almost always irreconcilable: a highly sensitive search is also often less precise. This is illustrated in the figure below, where the less theoretical concepts wide (sensitive) versus narrow (specific) search are used.
For a systematic review, the search strategy should be highly sensitive. A large portion of the search result will thus not be relevant. In systematic literature searching, a precision of two-three percent is common, i.e. two-three references out of a hundred will be relevant.
Do not reinvent the wheel
Before you create a search strategy, it is a good idea to look at what has already been done. Start by collecting published literature reviews on the topic. Even if they are not totally updated or exactly match your topic, it is often worthwhile to have a look at the method section of the reviews; sometimes the search strategies are available as supplemental material.
In addition, there are validated search filters that might be useful. Search filters, or hedges, are sets of search terms chosen to restrict a search to a selection of references, for example based on study type (in Cochrane reviews usually randomised controlled trials) or method (for example qualitative methods). A search filter is often developed in different variants based on different levels of sensitivity, specificity and precision.
Some search filters are integrated into databases like Pubmed/Medline, Psycinfo and Cinahl. In Pubmed, Clinical Queries allows you to use search filters to restrict the search to clinical studies, genetic studies or systematic reviews.
Web sites for validated search filters:
Create search blocks
A well-defined and clear research question is an essential starting point for a systematic search. To create a logical search strategy, always start by identifying the key elements of the research question – i.e., establish what the main concepts of the topic are. With these concepts, you can then create the search blocks that form the basis for the search strategies used in the different databases.
We will use this research question as an example:
- Does routine use of inhaled oxygen in acute myocardial infarction improve patient-centered outcomes, in particular pain and death?
Some of the potential concepts that could form the search blocks are marked in bold. The PICO structure is a common way of formulating clinical research questions: Population, Intervention, Control and Outcome. In general, you should not include all parts of the PICO question in the search. Focus is generally on the population and the intervention, in our example P = patients with myocardial infarction and I = inhaled oxygen.
The search blocks are combined with the boolean operator AND in the search strategy. A general principle is that a search for a systematic review should consist of few blocks. More search blocks means a less sensitive search and a higher risk of missing relevant articles. Two up to maximum four search blocks is a rule of thumb.
The boolean operator NOT should be avoided in systematic search strategies. NOT decreases the sensitivity of the search and the risk of missing relevant articles increases. Please note that this applies to the final search strategy. NOT can be a useful tool when constructing and comparing searches – see the section Analyse your search below.
Find search terms
In every search block, you should include all relevant search terms and variants, combined with the boolean operator OR. In contrast to the number of search blocks, you should also try to include as many relevant terms as possible, since this makes your search more sensitive.
Use both subject headings and free-text terms. In this way you will use the potential of the controlled vocabulary and retrieve articles where the authors used various terminologies. You will also retrieve articles that lack subject headings, or are indexed with other subject headings than those you have used in your search.
In many databases, the default search setting is all fields or a combination of fields, which means searching in both free-text and subject headings. However, in a systematic literature search we emphasize the importance of specifying the search fields manually. This gives you more control and facilitates a more logical and cohesive search strategy.
The process of finding relevant search terms is an important part of the systematic search. Usually you start off with a scoping search, with the subject headings and synonyms you already know of. A scoping search will give you a sense of how much has been written on your topic; by screening titles, abstracts and subject headings more relevant search terms can be found.
You should also identify a set of key articles, i.e. significant studies in your field. These key articles should correspond exactly to your research question and be retrieved by your search strategy. In sum, the key articles could be used for both constructing and for validating your search strategy. If you don't retrieve these studies in your search, modify your search query.
An additional suggestion is to check the bibliographic information available for the key articles: title, abstract, author keywords, subject headings etc. These might provide additional search terms.
There are tools specific tools for finding search terms which analyses a search query or a set of references. The tools below are adapted for Pubmed, but the free-text terms are generic and can thus be used in all databases:
In many databases, articles are for better retrieval indexed with subject headings or controlled terms from a thesaurus. In PubMed, the controlled vocabulary is MeSH: Medical Subject Headings.
You need to find subject headings to build a good search strategy. Search for your key concepts in the controlled vocabulary and check what subject headings your key articles are indexed with to find relevant terms. Explore the hierarchies and related terms. Please note that there might be several subject headings for closely related concepts, and that these headings might be part of different hierarchies.
Check if some of the terms are new as subject headings. If so, you might also consider including the previous subject heading used for indexing the concept.
Avoid limits for subject headings, such as subheadings or main concept, for example MeSH Major Topic in Pubmed.
The default setting in most databases is the exploding of subject headings, i.e. narrower terms of the subject headings are included. In general, this functionality is helpful, but you might in some cases consider if a subject heading should be searched without being exploded, i.e. not including narrower terms.
Include all the relevant synonyms and spelling variations in title and abstract from the key articles and other relevant studies retrieved by the test search. The controlled vocabularies are useful tools to find free-text terms. You should, of course, search for the subject heading terms as free-text too. In addition, you can find more free-text terms among the synonyms of a subject heading. These synonyms are named differently in different databases. In Pubmed, they are called Entry terms.
Narrower terms should also be considered as free-text. These terms are included automatically in the controlled vocabulary search, thanks to the explode functionality, but for the free-text search they have to be explicitly stated to be included. One example of this is different forms of myocardial infarction, STEMI and non-STEMI,. Those terms are automatically included in the exploded MeSH search for Mycardial Infarction, but have to be added in the free-text part.
Use truncation to find different variants of a word; therap* retrieves for example:
Quotation marks are useful for keeping words in phrases together. On the other hand, you need to be careful with this in systematic searching: quotation marks make the search more precise and hence can relevant literature be missed. One alternative is to use proximity operators, which lets you specify the number of words that can appear between two search terms. Proximity operators thus retrieves different variations of phrases, for instance in the word order, as a contrast to phrases with quotation marks.
Let us use oxygen treatment as an example. If searching for this concept as a phrase, we'll miss to catch potential relevant references including:
- "oxygen (HBO) treatment"
- "treatment with oxygen"
- "oxygen in the treatment"
However, these variations are retrieved if we use a proximity operator (oxygen NEAR/3 treatment in Web of science).
Spelling differences in British and American English
Some medical terms, a few of them quite common, are spelled differently in British and American English. Consider if your search strategy includes a word with a different spelling. If so, include both of them. Some databases can compensate for this automatically, so called lemmatization, but you should not rely completely on this functionality.
Six examples (British vis-à-vis American English):
- Tumour / Tumor
- Gynaecology / Gynecology
- Coeliac / Celiac
- Ageing / Aging
- Behaviour / Behavior
- Labour / Labor
Analyse your search
When a tentative search strategy has been constructed, you should analyse if your key articles have been retrieved by the search.
- Conduct your search (A).
- Do a new search (B) for your key articles only, for instance in Pubmed by using PMID.
- Which key articles are not retrieved by your search Search B NOT A. If all key articles are retrieved, the search result will be zero.
If key articles are not retrieved, you should analyse why. Try to identify which search block that causes the exclusion of the article. What search terms can be added to include this key article? Maybe you should consider removing a complete search block? However, sometimes it's not possible to retrieve all key articles and, at the same time, have a reasonable precision.
You might also use NOT to find additional subject headings or free-text terms. Conduct a search (A) including subject headings only and one search (B) with free-text only. Then search A NOT B. In this case, you'll retrieve all studies indexed by the relevant subject headings but without the free-text terms you have included. Check for additional free-text terms in the title and abstract. After that, do the opposite (B NOT A) to find more relevant subject headings.
The next step is to screen the search results. Do the references correspond to your topic? Because the search strategy should be highly sensitive, the precision will surely be quite low (hence many irrelevant hits). However, you should of course also find some eligible references.
Maybe some of the search terms will generate a lot of false hits. If so, you might consider to remove them. You can use the NOT operator again in order to analyse the search.
- Conduct a search including the term (A).
- Conduct a search without the term (B).
- Search A NOT B.
Analyse the result. If you do not find any relevant hits, consider excluding the term from the search strategy.
Databases – examples and tips
In biomedicine, you'll usually begin the literature search in Pubmed and thus include the subset Medline – the larger part of indexed articles in Pubmed. However, when conducting more advanced search strategies, Medline via the Ovid interface is commonly used (see below).
You are probably quite familiar with Pubmed, and it is an easy database to start with. There are, however, a few things to consider:
- You can use truncation, but only the last word i phrases.
- Not all phrases are searchable, only the phrases included in the phrase index. Look out for error messages or check if your phrase is searchable by browsing the phrase index: select Title/Abstract in the Advanced Search Builder, add your phrase and choose Show index list. If your phrase is not included in the phrase index it will be separated and searched as independant terms combined with the boolean operator AND.
- Do not trust the automatic term mapping, specify the fields to be searched instead to gain full control. Moreover, if you are using truncation it will turn off the automatic term mapping. If you want to see how the automatic term mapping works, search for a term without specifying which field to search in and then click on Search Details.
- If you are using field tags (as in the example below) it automatically translates into a phrase search, similar to using quotation marks.
- You cannot use proximity operators, in contrast to almost all other advanced databases.
In Medline Ovid, the indexed part of Pubmed is included, but also newly added references ("Epub Ahead of Print" and "In-Process") and other non-indexed material. Ovid is a database platform used for many different databases. Through the KI library subscriptions, you also have access to the databases Psycinfo and Global Health via Ovid.
There are several advantages of searching Medline via Ovid instead of Pubmed. In Ovid it is possible to use proximity operators, truncate (without any limitations) and search for all phrases (as opposed to Pubmed, which is limited by the phrase index). It is also easier to structure comprehensive and advanced search strategies. Information specialists tend to prefer Medline Ovid and the platform is, in most cases, the standard interface for Cochrane Reviews.
As in the example below, we deconstruct our search strategy into lines and then combine them, first each line within a search block with OR, then the search blocks with AND.
There are several different options for searching in Embase (via embase.com). In the example below, the search strategy is divided in lines as in Ovid: one for each Emtree term, one line for all free-text. These are then combined into one search block. Emtree is the controlled vocabulary in Embase, similiar but not identical to MeSH in Pubmed. You can access the Emtree terms in the top menu under "Browse".
Web of Science
Web of Science (Core Collection) is a multi-disciplinary database that includes research areas other than biomedicine and health. There is no controlled vocabulary in Web of Science and only free-text searches are possible.
A few things to consider in Web of Science:
- Use quotation marks around phrases to keep the words together for greater precision.
- Truncation and proximity operators can be used and are more important because it's a free-text database.
- Consider dividing the search strategy as in the example below, using one search field per search block.
- The default Basic Search is a good start, but also familiarize yourself with Search History for a better overview and editing possibilities.
Structure and documentation
To be systematic implies a focus on structure, organization and documentation. As in all research, the review process should be transparently documented in all parts, reported clearly in the final publication, and reproducible.
As a support in the review process there are the PRISMA Guidelines: "an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses." PRISMA – an acronym for Preferred Reporting Items for Systematic Reviews and Meta-Analyses – consists, among other things, of a check list and a flow diagram. Many journals, for example Lancet, BMJ and Plos One, endorse the guidelines stated by PRISMA and require a PRISMA flow diagram when publishing systematic reviews.
According to PRISMA, the search strategy should be reported in full for at least one database; however, we recommend attaching all search strategies as an appendix to the published article. The majority of journals allow you to upload supplemental material. If the complete search strategies are attached, you only need to briefly describe the search process in the methods section and thereafter refer to the supplemental material section.
In PRISMA, two items concern the literature search:
- Item 7: Information sources – Describe all information sources (e.g., databases with dates of coverage, contact with study authors to identify additional studies) in the search and date last searched.
- Item 8: Search – Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated.
As with a clinical trial, a protocol should be established for a systematic review; preferably, it should also be registered. This is actually one of the items for reporting in the PRISMA Guidelines. Some journals – for instance BMJ, The Lancet and British Journal of Dermatology – requires a registered protocol for systematic reviews. There are several reasons for this. For instance, it increases the overall transparency of the review, making it more scientifically sound. Additionally, the registration of the protocol also adds visibility to your review. This can help to avoid duplication of the work.
There is also a guideline for developing protocols för systematic reviews: PRISMA-P and a specific database aimed for the registration of protocols for systematic reviews: PROSPERO – International Prospective Register of Systematic Reviews. It's also possible to publish protocols in the journal Systematic Reviews.
Systematic literature searches always produce a large amount of references. Also, when searching in several different databases many duplicate references will appear – i.e., identical references that are included in more than one database.
Reference management software, such as Endnote, is recommended for collecting, storing and organizing your references. Using a reference manager, you can organize your references in different groups, for example, included and excluded studies. You can also remove duplicates by sorting the references by title or by using the duplicate functionality. A deduplication method has been developed for Endnote.
However, Endnote and other types of reference managers have limitations when it comes to supporting the systematic review process. Other software has therefore been developed and is dedicated to supporting the systematic review process, for instance in the screening of abstracts and references according to inclusion and exclusion criteria. For the time being, KI offers no institutional licens on this kind of software.
An thorough summary of available tools can be found at Systematic Review Toolbox.
PRESS – checklist for search strategies
As the literature search is a fundamental part of a systematic review, it's important that the search strategy is of high quality. To ensure this, and to avoid errors, there are several options. One option is to involve an experienced librarian/information specialist in the review project. In either case, it's good if a second person has a thorough look at the search strategy. There is also a tool for validating search strategies called Peer Review of Electronic Search Strategies (PRESS).
PRESS was originally published in 2008–2010, but was revised in 2015. The PRESS checklist was originally made for expert searchers, such as librarians, but can also be used by students and researchers when creating more extensive search strategies. The evidence and conclusions in PRESS are based on a comprehensive research project.
- SBU method book (in Swedish).
- 11 instructional films which describe the steps in the process of a systematic literature review. By Cushing/Whitney Medical Library at Yale University.
- PRESS: Peer Review of Electronic Search Strategies from Canadian Agency for Drugs and Technologies in Health. Interesting and thorough study that identifies and lists the errors and omissions that occur in systematic literature searches.
- Another informative guide about systematic literature searches by Bernard Becker Medical Library, Washington University School of Medicine, St. Louis, Missouri.
- Bettany-Saltikov, J, McSherry, R. How to do a systematic literature review in nursing: A step-by-step guide. London: McGraw-Hill Education/Open University Press; 2016.
- Gough D, Oliver S, Thomas J, editors. An introduction to systematic reviews. Los Angeles, Ca.: SAGE; 2012.
- Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.1 (updated September 2020). Cochrane, 2020.
- Borenstein M. Introduction to meta-analysis. Oxford: Wiley-Blackwell; 2009.
- Egger M, Smith GD, Altman DG, editors. Systematic reviews in health care: meta-analysis in context. 2. ed. London: BMJ; 2001.
If you are a bachelor's or master's student, or a doctoral student writing a literature review for the half-time summary, please contact the librarians in the library. You can also can make a booking for a certain time.
If you're a researcher or a doctoral student preparing a systematic review, please contact the search group.