Skip to Main Content
Douglas College Library About Us Articles & Databases Research Guides Services Faculty News Events Learning Centre

Guide to Systematic Reviews & Meta-Analyses

Planning Stages of a Systematic Review

First steps:
 

  1. Recruit your team - a systematic review (SR) requires at minimum two members, but ideally there will be enough members to break ties during your blinded screening process and to cover any experience/expertise gaps you may have. 
     
    1. Consider adding a member with expertise in database searching, e.g., "information specialists...librarians or trial search co-ordinators" (Cooper et al. 2018). 
       
      1. You may even want to recruit a second database search expert to peer-review the search strategies being proposed.  See the CADTH/PRESS Peer-review site to learn more about the rationale for this.

         
    2. Also consider inviting a statistician, one or more experts in knowledge synthesis - e.g., have prior experience in conducting a systematic review, and one or more subject specialists - if needed to complement your own expertise.  Teams can typically number anywhere between 3 and 20+ members.

       
    3. When recruiting it's important to ensure that team members will be available for the duration of the project - quality SRs can easily take a year or more.

       
  2. Define your research question.  This is best accomplished by using a framework such as PICO, PEO, SPIDER, SPICE, ECLIPSE. etc.

     
  3. Choose the SR methodology you intend to follow, e.g., JBI (for Health Sciences), Cochrane (for SRs of Interventions); Campbell Collaboration and/or MOSAIC (for Social Sciences)

     
  4. Write a protocol.  A protocol is a declaration of your research intent. It outlines the rationale for your project, the theory you're attempting to prove/disprove, the search strategies you intend to employ, and the inclusion and exclusion criteria for the studies that you will review. You may find it useful to consult a protocol planning resource or template such as:
     
  5. Plan your search.  Conducting a quality systematic review is time-intensive, so it's important to do some initial planning and research. 
     
    1. Read a selection of systematic reviews from your field.  These can alert you to the most relevant databases for your own search, as well as suggest key-words and subject headings that will be useful to add to your search strategies.

       
    2. Check as many review databases as possible to see if your proposed review has been done before, as your project should fill a gap in the literature, not retread old ground.

       
    3. Also check as many systematic review protocol registries as possible to see if similar/duplicate work is already underway.  Note, it's free to browse & search the following, but requirements for registering your own protocol vary:
       
    4. Researchers in the social sciences may find it useful to check the Campbell Evidence and Gaps Maps (EGMs) for "the availability (or lack thereof) of rigorous evidence for a particular policy domain."

       
  6. Register your protocol once you've verified that your project isn't going to duplicate others' work. 
     
    1. Note, some disciplines may not have a registry for their research areas, but it's still highly recommended that you write a protocol with your team to ensure you have a shared understanding of the project scope, the precise question to be answered and the exclusion/inclusion criteria you've agreed on.

       
  7. Try some initial scoping searches to get a sense of the volume of research you'll need to review.

Searching & Screening your Reviews

Searching & Screening

 

A rigorous search is required "to avoid missing key studies and to minimize bias since a systematic review based only on published (or easily accessible) studies may have an exaggerated effect size" (Cooper et al. 2018). 

 

  1. Conduct your searches according to your pre-set inclusion & exclusion criteria.
     
    1. A comprehensive literature search should include searching the relevant databases for your discipline, plus review-specific databases such as Cochrane Database of Systematic Reviews, Evidence Based Medicine Reviews etc.
       
      1. Also search the open web, e.g., Google Scholar, Theses Canada, clinical trials registries, professional organizations' websites, government sites, social media, conference websites, institutional repositories - especially to surface pre-prints and theses & dissertations that may not be uploaded elsewhere etc.
         
      2. Additional search techniques to try: e.g., search all the titles in multiple issues of leading journals, comb through your articles' bibliographies and/or follow Google Scholar's "cited by" links to surface content not found elsewhere. You could also reach out out to leading researchers in your field to see if they're working on/aware of relevant works that are soon-to-be-published.
         
    2. Be prepared to re-run your search later in the project to capture the most recent publications - especially If your initial search took place a year or more previously.  This reduces the likelihood of missing important, new publications / inadvertently biasing your results. 
       
      1. You can mitigate this somewhat by signing up for "new publication" search alerts with all the database hosts that offer this feature, using your established search strings.  To learn more see the Database Exporting Tools box in this guide.
         
    3. Quality Assessment:  Be prepared to eliminate low-quality studies from consideration.  There's no advantage to including work that doesn't meet your scholarly standards, solely for the purpose of "being thorough." 
       
    4. Tools such as AMSTAR   and  ROBIS will help with your quality and risk of bias assessments.

       
  2. Record your search strategies carefully, using reporting guidelines such as PRISMA, EQUATOR etc. 
     
    1. You'll need to detail your search strategies in the methods section of your review to ensure that your work is reproducible.
       
    2. See the Guidance & Resources page for links to additional reporting guidelines. 

       
  3. Save your studies in a reliable citation management tool, such as Zotero, Mendeley, etc.  Citation management tools make it easy to collaborate with others and to manage large volumes of citations.

     
  4. Screen your results to ensure they comply with your inclusion criteria on an anonymous basis - until everyone's decisions have been made. 

    You will need a minimum of two screeners, plus a tie-breaker.  You may want to consider using a screening tool to streamline this process.
     
    1. Start by screening titles and abstracts to identify potentially relevant papers.
       
    2. Then screen the full papers identified as possibly relevant in the initial screening.
       
    3. Your review protocol should "specify the process by which decisions on the selection of studies will be made. This should include the number of researchers who will screen titles and abstracts and then full papers, and the method for resolving disagreements about study eligibility" (CRD.  Systematic Reviews: 1.2.2.5 Study selection).

Extraction, Analysis & Publication

Final Steps
 

  1. Extract the data: You and your team-members must locate and compile the relevant information that will form the basis of your review. e.g., the literature you reviewed, the study characteristics that met your inclusion criteria; population(s) studied; research methodolog(ies) employed; risk of bias assessment(s); outcomes/results and more.
     
    1. To learn more see Cochrane Reviews, Section 5.5: Extracting data from reports
       
    2. Useful data extraction templates are available from:
      1. Cochrane (Randomized Controlled Trials [RCTs] only).  Word
      2. Cochrane (RCTs & non-RCTs). Word
         
    3. SRDR+ is "a free platform for extracting, archiving, and sharing data during systematic reviews and accessing shared data related to systematic reviews....(and also) helps build, customize, and share data extraction forms."  From the US Agency for Healthcare Research and Quality. 
       
      1. Citations can be added manually, "fetched" from pubmed, and/or batch uploaded for RIS, .csv, and EndNote file formats.
         
      2. Extensive help guides & tutorials are available.

         
  2. Synthesize your results:
     
    1. "A qualitative synthesis is a narrative, textual approach to summarizing, analyzing, and assessing the body of evidence included in your review. It is a necessary part of all systematic reviews, even those with a focus on quantitative data.
       
    2.  A quantitative synthesis, or meta-analysis, uses statistical techniques to combine and analyze the results of multiple studies. The feasibility and sensibility of including a meta-analysis as part of your systematic review will depend on the data available"  (Ohio State Health Sciences Library.  Systematic Reviews).
       
    3.  To learn more see the Disciplinary Guidance & Resources page of this guide.  Most guides include a section on data / narrative synthesis.

       
  3. Write up your findings & publish: Present your findings and any recommendations you may have for fellow researchers in the field.

     
  4. Deposit your research data -  As with other forms of research, many journals and funders are now requiring that all datasets that you used to perform your systematic review be made accessible to your readers for their own scrutiny/potential re-use, e.g., your data extraction forms, any statistical code/data files/programs you used for any part of your data analysis, your protocol(s), precise searches that were conducted etc.

    1. Ideally you would deposit your data into a trusted data repository to coincide with the publication of your review.
       
    2. To learn more about managing and depositing research data see our Guide to Research Data Management
       
    3. The University of Calgary Library has a great Guide to the DMP Assistant Template for Systematic Review Projects if you're planning to use "Canada's DMP Assistant template for Systematic Reviews (English version) ."
       
    4. The University of Calgary has also developed its own rdm template for Systematic Reviews that you may find very helpful.