Skip to Main Content

Evidence Synthesis

What is a Systematic Review?

Systematic Review seeks to systematically search for, appraise and synthesize research evidence, often adhering to guidelines on the conduct of a review.

Systematic reviews are research projects which are more than just an organized collection of articles; they are primary studies which provide synthesized information, built on other studies but providing new insight into a phenomenon, topic, or situation. The most straightforward and useful definition of systematic reviews, applied across all questions and disciplines is that they are reviews of the research literature using systematic and transparent methods (EPPI Centre 2015; Gough, Oliver, and Thomas 2012). These processes are the following: plan, identify, evaluate, collect and combine, explain, and summarize (PIECES) what can be ascertained from existing literature in order to answer a well-defined research question. 

Excerpt from:
Foster, M. J., & Jewell, S. T. (Eds.). (2017). Assembling the pieces of a systematic review : A guide for librarians. ProQuest Ebook Central https://ebookcentral.proquest.com

Steps to the Systematic Review Process (PIECES)

Planning: Every type of review needs to start with planning. As you plan, you may have to revise the type of review you conduct. You do not have to follow these steps in a strict order. For example, you might form the team first and then together decide the research question.
Planning Step
Explanation
Determine the purpose of the review

To help you decide what type of review you want to conduct.
Pose a research question

Which should use an established framework, if possible.
Conduct exploratory searches

To identify:

  • existing reviews
  • assess the potential volume of relevant studies
  • locate 2-4 sample studies
Determine whether the project is feasible

Based on time, resources, originality, relevance, etc.
Pick a team

Of at least 3-6 members, including:

  • subject experts
  • method experts
  • information experts
Select preliminary synthesis and analysis methods

Will depend on types of studies (can be qualitative, quantitative, or integrative--a review that integrates both quantitative and qualitative studies)
Design a protocol

Includes eligibility criteria, a timeline, data management, and project management.

May be tweaked as the research progresses but gives direction for the project.
Helpful Tools

Identifying: Identifying all the possible studies that address your question is key to a rigorous systematic review. Your initial searches should yield hundreds if not thousands of citations. Some steps can be done concurrently. For example, one team member can work on searching databases while another works on searching the grey literature.
Identifying Step
Explanation
Identify databases you will search

Including both general and subject specific databases.
Create a search based on your research question

Using advanced search strategies such as combining synonyms with OR and using wildcards and proximity operators to find variants.
Translate the search For each database using appropriate controlled vocabulary and correct syntax.
Store the results of each search

Separately, using a citation manager.
Document each search

Including:

  • number of citations
  • exact search string
  • database
  • date search was run
  • any limits applied
Search grey literature

Such as:

  • theses
  • conference proceedings
  • repositories
  • government documents
  • clinical trial registries

to address publication bias
Search other sources Such as reference lists, relevant journals that are not entirely indexed in databases, and researcher bibliographies.
Helpful Tools

Evaluating: In evaluating your search results, you winnow down the thousands of citations you initially identified to dozens which you will examine more closely. Often some of these steps are done concurrently. For example, after 500 citations have been evaluated for relevancy, the studies chosen for inclusion can be evaluated for quality and bias while the next 500 citations are evaluated for relevancy.
Evaluating Step
Explanation
The citations from the searches are de-duplicated

These are combined into a master list with the resulting number of citations documented.
Each article is evaluated for relevancy

At least two reviewers determine whether a citation meets the eligibility criteria for inclusion as set up in the protocol.

Often inclusion/exclusion can be based on title and abstract but sometimes full-text is required for the determination.

Tools are available that can help track the status of each citation. Some have AI which can speed up the process.
Full text is obtained

For those citations which inclusion cannot be determined by title and abstract alone.

For all citations to be included in the synthesis.
Master list of studies is complied

Some studies may be written up in more than one article.

Some articles may include more than one study.
Each study is evaluated for quality and bias (Critical Appraisal)

At least two reviewers determine the quality of each study.

A valid quality assessment tool or checklist appropriate for the type of study should be used.
A list of studies to be included in the synthesis is compiled If any studies are deemed to be of lower quality or biased, documentation of their inclusion or exclusion is necessary.
Helpful Tools
  • Checklists for Critical Review
  • This liCovidence
    A workflow platform that enables researchers to to bring together all the research from around the globe and turn it into summaries of scientific knowledge. Import citations, use machine learning to screen for randomized controlled trials, screen titles and abstracts, upload references, screen full text, extract data, conduct risk of bias, and export data.
  • Rayyan
    A free online tool from Qatar Computing Research Institute and Rayyan Systems that allows you to import lists of articles for tagging and selection.
  • EPPI Reviewer
    A web-based software program for managing and analysing data in literature reviews. Developed for all types of systematic reviews, it manages references, stores PDF files and facilitates qualitative and quantitative analyses such as meta-analysis and thematic synthesis. It also contains some new ‘text mining’ technology which is promising to make systematic reviewing more efficient.
  • SUMARI
     
 

Collecting & Combining Data: You will need to collect or extract data from all the studies you identified to form your evidence base. You then can analyze the evidence base and synthesize any trends you find.
Collecting & Combining Data Step
Explanation
Confirm synthesis and analysis methods

Will depend on the types of studies.

Can be qualitative, quantitative, or integrative.
Determine the data elements

That are needed to answer the research question.

Drafting potential evidence tables and figures that can help confirm what data should be eventually shown.
Develop forms

Determine how each data element will be extracted/coded.

Categorical coding allows for faster and more consistent extraction.

Open coding allows for more information and nuance but takes longer to extract and analyze.
Helpful Tools
  • Data Elements for Synthesis
  • Methods and Approaches for Synthesis and Analysis
  • Covidence
    A workflow platform that enables researchers to to bring together all the research from around the globe and turn it into summaries of scientific knowledge. Import citations, use machine learning to screen for randomized controlled trials, screen titles and abstracts, upload references, screen full text, extract data, conduct risk of bias, and export data.
  • EPPI Reviewer
    a web-based software program for managing and analysing data in literature reviews. Developed for all types of systematic reviews, it manages references, stores PDF files and facilitates qualitative and quantitative analyses such as meta-analysis and thematic synthesis. It also contains some new ‘text mining’ technology which is promising to make systematic reviewing more efficient.
  • Google Forms
    can be used to collect data elements from each study
  • Microsoft forms
    can be used to collect data elements from each study
  • Qualtrics
    can be used to collect data elements from each study
  • SUMARI
    The System for the Unified Management, Assessment and Review of Information (SUMARI) is the Joanna Briggs Institute's software for the systematic review of literature. Supporting 10 review types, it facilitates the entire review process, from protocol development, team management, study selection, critical appraisal, data extraction, data synthesis and writing your systematic review report.

Explaining the Synthesis & Analysis: Systematic reviews include synthesis which summarizes and organizes the information found in the studies that are identified. This synthesis informs the conclusions that are drawn in the systematic review and focuses on both the methodology and results of the studies. Depending on the project and the types of studies investigated, the synthesis can be simply descriptive or can include more in depth analysis. Generally, synthesis and analysis involve looking for trends and patterns to use in comparisons, to discover explanatory or confounding variables, to develop themes or frameworks, to inform best practices, etc. All systematic reviews include a narrative explanation but other kinds of explanations can also be used.
Explaining the Synthesis & Analysis
Explanation
Narrative explanations

Can describe trends, themes, frameworks, perspectives, characteristics, quality, etc.

Especially useful for empirical research.

Can use structured narratives.

Often accompanied by tabular explanations.
Tabular explanations

Uses tables to explain the synthesis.

Can be used to describe study characteristics, study measures, study quality, study results, etc.

Accompanies narrative explanations.
Graphical explanations

Uses graphical methods to explore and present data.

Can include concept maps, forest plots, harvest plots, idea webs, logic models, mind maps, and network analysis.
Helpful Tools
  • Methods and Approaches for Synthesis and Analysis
  • Data and Digital Scholarship Information
    Baylor Libraries' Data Scholar Program & Workshops, Data Research Fellows
  • Center for Statistical Consulting
    Baylor Statistical Collaboration Center seeks to enhance the quality of research involving statistical methods conducted at Baylor University and with external partners. Provides short-term statistical analysis, long-term collaborative research, and assistance in the preparation of research grant proposals
  • EPPI Reviewer
    a web-based software program for managing and analysing data in literature reviews. Developed for all types of systematic reviews, it manages references, stores PDF files and facilitates qualitative and quantitative analyses such as meta-analysis and thematic synthesis. It also contains some new ‘text mining’ technology which is promising to make systematic reviewing more efficient.
  • NVivo
    NVivo is a qualitative data analysis (QDA) computer software package produced by QSR International. It has been designed for qualitative researchers working with very rich text-based and/or multimedia information, where deep levels of analysis on small or large volumes of data are required. (Descriptions from https://en.wikipedia.org/wiki/NVivo)
  • SUMARI
    The System for the Unified Management, Assessment and Review of Information (SUMARI) is the Joanna Briggs Institute's software for the systematic review of literature. Supporting 10 review types, it facilitates the entire review process, from protocol development, team management, study selection, critical appraisal, data extraction, data synthesis and writing your systematic review report.
  • OpenMeta[Analyst]
    Open Meta-Analyst is open-source software for performing meta-analyses of binary, continuous, or diagnostic data, using a variety fixed and random-effects methods, including Bayesian and maximum likelihood analysis. Open Meta-Analyst also enables you to do cumulative, leave-one-out, subgroup, and meta-regression analyses.

Summarizing: Summarizing your systematic review allows it to be shared with others.
Identifying Step
Explanation
Identify your audience(s)
  • research funders
  • policy makers
  • practitioners
  • research community
  • media
  • public
Choose your summary type
  • protocol
  • report
  • presentation
  • peer review publication
Review Standards

Most standards will include these sections:

  • background/introduction
  • methodology including a PRISMA diagram
  • results
  • discussion/conclusions
Write Your Summary

Make sure your work is clear, auditable, replicable, and transparent.
Submit Your Summary
Search grey literature

Such as:

  • theses
  • conference proceedings
  • repositories
  • government documents
  • clinical trial registries

to address publication bias
Search other sources Such as reference lists, relevant journals that are not entirely indexed in databases, and researcher bibliographies.
Helpful Tools

Frameworks for Systematic Review Research Questions

Framework - An established structure that helps define the research question.  Can address different disciplines and question types.

For example: PICO is the most common and well-known standard framework for medical and health systematic reviews.

Framework

Discipline/Question Type
PICO (Richardson et al. 1995)
Patient, Intervention, Comparison, Outcome		 Clinical medicine
BeHEMoTh (Booth and Carroll 2015)
Behavior of interest, Health context (service/policy/ intervention), Exclusions, Models or Theories		 Questions about theories
CHIP (Shaw 2010)
Context How Issues Population		 Psychology, qualitative
CIMO (Denyer and Tranfield 2009)
Context Intervention Mechanisms Outcomes		 Management, business, administration
CLIP (Wildridge and Bell 2002)
Client group, Location of provided service, Improvement/Information/ Innovation Professionals		 Librarianship, management, policy
COPES (Gibbs 2003)
Client-Oriented, Practical, Evidence, Search		 Social work, health care, nursing
ECLIPSE (Wildridge and Bell 2002)
Expectation, Client, Location, Impact, Professionals, SErvice,		 Management, services, policy, social care
PEO (Kahn et al. 2003)
Population, Exposure, Outcome		 Qualitative
PECODR (Dawes et al. 2007)
Patient/population/problem, Exposure, Comparison, Outcome, Duration, Results		 Medicine
PESICO (Schlosser and O’Neil-Pirozzi 2007)
Person Environments Stakeholders Intervention Comparison Outcome		 Augmentative and alternative communication
PICO specific to diagnostic tests (Kim et al. 2015)
Patients/participants/population, Index tests, Comparator/reference tests, Outcome		 Clinical medicine
PICO+ (Bennett and Bennett 2000)
Patient, Intervention, Comparison, Outcome, +context, patient values, and preferences		 Occupational therapy
PICOC (Petticrew & Roberts, 2006)
Patient, Intervention, Comparison, Outcome, Context		 Social sciences
PICOS (Moher et al. 2009)
Patient, Intervention, Comparison, Outcome, Study type		 Medicine
PICOT (Richardson et al. 1995)
Patient, Intervention, Comparison, Outcome, Time		 Education, health care
PIPOH (ADAPTE Collaboration 2009)
Population, Intervention, Professionals, Outcomes, Health care setting/context		 Diagnostic questions
ProPheT (Booth et al. 2016)
Problem, Phenomenon of interest, Time		 Screening
SPICE (Booth 2004)
Setting, Perspective, Interest, Comparison, Evaluation		 Social sciences, qualitative, library science
SPIDER (Cooke et al. 2012)
Sample, Phenomenon of Interest, Design, Evaluation, Research type		 Library and information sciences
WWH
Who What How		 Health, qualitative research

Adapted from Foster, M. J., & Jewell, S. T. (Eds.). (2017). Assembling the pieces of a systematic review : A guide for librarians.

Critical Appraisal and Evaluation

Critical Appraisal - the assessment of data published in research in an explicit and transparent manner with regards to factors such as bias, validity, methods, and conclusions. Also known as quality assessment.
 

These websites provide checklists for critical appraisal of different types of studies.

Tools for Critical Appraisal

Tool

Study Type
GRADE Approach http://www.ncbi.nlm.nih .gov/pubmed/26772609 General
Assessment of Multiple Systematic Reviews (AMSTAR) http://amstar.ca/Amstar _Checklist.php Randomized Studies
CASP Randomized Controlled Trial Checklist http://tinyurl.com/ zwqqugy Randomized Studies
Analysis Manual: Steps in the Academy Evidence Analysis Process http://tinyurl.com/ z6sy9ym Observational Studies
CASP Cohort Study Checklist http://tinyurl.com/ gub25e7 Observational Studies
CASP Case Control Checklist http://tinyurl.com/ pmujmu2 Observational Studies
Guide to scoring methods using the Maryland Scientific Methods Scale
http://www .whatworksgrowth.org/ public/files/Scoring -Guide.pdf		 Observational Studies
Interrupted time series (ITS) analyses http://tinyurl.com/ zu6wnog Observational Studies
STROBE Checklist (Strengthening the reporting of observational studies in epidemiology)
http://www.ncbi.nlm.nih .gov/pubmed/18064739		 Observational Studies
Transparent Reporting of Evaluations with Nonrandomized Designs (TREND) statement
http://www.ncbi .nlm.nih.gov/ pubmed/?term=14998794		 Observational Studies
10 Questions to Help You Make Sense of Qualitative Research http://tinyurl.com/ m4kbv7f Qualitative Studies
EPPI-Centre Methods for Conducting Systematic Reviews http://tinyurl.com/ zwp976k Qualitative Studies
Evaluation Tool for Quantitative Research Studies http://usir.salford .ac.uk/12969 Qualitative Studies
Guidelines for Critical Review Form: Qualitative Studies (Version 2.0) http://tinyurl.com/ hzqh5q8 Qualitative Studies
Quality in Qualitative Evaluation: A Framework for Assessing Research Evidence http://tinyurl.com/hgkxzlp Qualitative Studies
A Scoring System for Appraising Mixed Methods Research http://www.ncbi.nlm.nih .gov/pubmed/19233357 Mixed Methods Research
Evaluation Tool for Mixed Methods Studies http://usir.salford .ac.uk/13070 Mixed Methods Research

Adapted from Foster, M. J., & Jewell, S. T. (Eds.). (2017). Assembling the pieces of a systematic review : A guide for librarians. 

Data Elements for Synthesis and Analysis

Data extraction - the process of collecting and coding information from relevant studies to form the evidence base which will be used to compare studies and find patterns, themes, and trends.
 

You will need to extract data from the selected studies to synthesize and analyze.  In addition to collecting general information about each study and those extracting the data, you may want to consider if they are relevant to your research question.

Category

Data Elements
Participants Total number of participants, Setting, Diagnostic criteria, Age, Sex, Race/Ethnicity, Location (country, state, county, etc.), Co-morbidities, Socio-demographics, Spectrum of presenting symptoms and current treatments, Date of study, Date of recruitment and follow-up, Participant sampling
Intervention Total number of intervention groups, Specific intervention, Intervention details, Integrity of intervention
Outcomes Outcomes and time points (i) collected & (ii) reported, Outcome definition (with diagnostic criteria if relevant), Unit of measurement
Comparisons Comparison
Results Sample size, Missing participants, Summary data for each intervention group, Estimate of effect with confidence interval ( P value), Subgroup analyses, Adverse events and side effects for each study group
Interpretation Overall evidence, Generalizability: external validity of trial findings
Objectives Research questions and hypotheses
Method Reference standard and its rationale, Technical specifications, Study design, Total study duration, Sequence generation, Allocation sequence concealment, Blinding, Methods used to generate random allocation sequence, implementation, Other concerns about bias, Methods used to compare groups for primary outcomes and for additional analyses, Methods for calculating test reproducibility, Definition and rationale for the units, cutoffs and/or categories of the results of the index tests and reference standard, Number, training, and expertise of the persons executing and reading the index tests and the reference standard, Participant flow
Qualitative Noting patterns and themes, Seeing plausibility (ensuring conclusions make good sense), Clustering, Making metaphors, Counting, Making contrasts/comparisons, Partitioning variables, Subsuming particulars into the general, Noting relations between variables, Building a logical chain of evidence, Making conceptual/theoretical coherence
Miscellaneous Funding source, Key conclusions of the study authors, Clinical applicability, Miscellaneous comments from the study authors, References to other relevant studies, Correspondence required, Miscellaneous comments by the review authors

Adapted from Jonnalagadda, S.R., Goyal, P. & Huffman, M.D. Automating data extraction in systematic reviews: a systematic review. Syst Rev 4, 78 (2015). https://doi.org/10.1186/s13643-015-0066-7 and
Booth, A. (2016). Systematic approaches to a successful literature review (Second edition.). Sage.

University Libraries

One Bear Place #97148
Waco, TX 76798-7148

(254) 710-6702

Osofast A-Z Databases OneSearch Profiles Library Account Login Hours
Baylor University

Copyright © Baylor® University. All rights reserved.

Report It | Title IX | Mental Health Resources | Anonymous Reporting | Legal Disclosures