Skip to Main Content

Evidence Synthesis OLD VERSION

Steps to the Meta-analysis Process

Planning: Every type of review needs to start with planning. As you plan, you may have to revise the type of review you conduct. You do not have to follow these steps in a strict order. For example, you might form the team first and then together decide the research question.
Planning Step
Explanation
Determine the purpose of the review

To help you decide what type of review you want to conduct.
Pose a research question

Which should use an established framework, if possible.
Conduct exploratory searches

To identify:

  • existing reviews
  • assess the potential volume of relevant studies
  • locate 2-4 sample studies
Determine whether the project is feasible

Based on time, resources, originality, relevance, etc.
Pick a team

Of at least 3-6 members, including:

  • subject experts
  • method experts
  • information experts
Select preliminary synthesis and analysis methods

Will depend on types of studies (can be qualitative, quantitative, or integrative--a review that integrates both quantitative and qualitative studies)
Design a protocol

Includes eligibility criteria, a timeline, data management, and project management.

May be tweaked as the research progresses but gives direction for the project.

Identifying: Identifying all the possible studies that address your question is key to a rigorous meta-analysis. Your initial searches should yield hundreds if not thousands of citations. Some steps can be done concurrently. For example, one team member can work on searching databases while another works on searching the grey literature.
Identifying Step
Explanation
Identify databases you will search

Including both general and subject specific databases.
Create a search based on your research question

Using advanced search strategies such as combining synonyms with OR and using wildcards and proximity operators to find variants.
Translate the search For each database using appropriate controlled vocabulary and correct syntax.
Store the results of each search

Separately, using a citation manager.
Document each search

Including:

  • number of citations
  • exact search string
  • database
  • date search was run
  • any limits applied
Search grey literature

Such as:

  • theses
  • conference proceedings
  • repositories
  • government documents
  • clinical trial registries

to address publication bias
Search other sources Such as reference lists, relevant journals that are not entirely indexed in databases, and researcher bibliographies.
Helpful Tools

Evaluating: In evaluating your search results, you winnow down the thousands of citations you initially identified to dozens which you will examine more closely. Often some of these steps are done concurrently. For example, after 500 citations have been evaluated for relevancy, the studies chosen for inclusion can be evaluated for quality and bias while the next 500 citations are evaluated for relevancy.
Evaluating Step
Explanation
The citations from the searches are de-duplicated

These are combined into a master list with the resulting number of citations documented.
Each article is evaluated for relevancy

At least two reviewers determine whether a citation meets the eligibility criteria for inclusion as set up in the protocol.

Often inclusion/exclusion can be based on title and abstract but sometimes full-text is required for the determination.

Tools are available that can help track the status of each citation. Some have AI which can speed up the process.
Full text is obtained

For those citations which inclusion cannot be determined by title and abstract alone.

For all citations to be included in the synthesis.
Master list of studies is complied

Some studies may be written up in more than one article.

Some articles may include more than one study.
Each study is evaluated for quality and bias (Critical Appraisal)

At least two reviewers determine the quality of each study.

A valid quality assessment tool or checklist appropriate for the type of study should be used.
A list of studies to be included in the synthesis is compiled If any studies are deemed to be of lower quality or biased, documentation of their inclusion or exclusion is necessary.
Helpful Tools
  • Checklists for Critical Review
  • This liCovidence
    A workflow platform that enables researchers to to bring together all the research from around the globe and turn it into summaries of scientific knowledge. Import citations, use machine learning to screen for randomized controlled trials, screen titles and abstracts, upload references, screen full text, extract data, conduct risk of bias, and export data.
  • Rayyan
    A free online tool from Qatar Computing Research Institute and Rayyan Systems that allows you to import lists of articles for tagging and selection.
  • EPPI Reviewer
    A web-based software program for managing and analysing data in literature reviews. Developed for all types of systematic reviews, it manages references, stores PDF files and facilitates qualitative and quantitative analyses such as meta-analysis and thematic synthesis. It also contains some new ‘text mining’ technology which is promising to make systematic reviewing more efficient.
  • SUMARI
     
 

Collecting & Combining Data: You will need to collect or extract data from all the studies you identified to form your evidence base. You then can analyze the evidence base and synthesize any trends you find.
Collecting & Combining Data Step
Explanation
Confirm synthesis and analysis methods

Will depend on the types of studies.

Can be qualitative, quantitative, or integrative.

Meta-analyses may include a sensitivity analysis.
Determine the data elements

That are needed to answer the research question.

Drafting potential evidence tables and figures that can help confirm what data should be eventually shown.
Develop forms

Determine how each data element will be extracted/coded.

Categorical coding allows for faster and more consistent extraction.

Open coding allows for more information and nuance but takes longer to extract and analyze.
Helpful Tools
  • Data Elements for Synthesis
  • Methods and Approaches for Synthesis and Analysis
  • Covidence
    A workflow platform that enables researchers to to bring together all the research from around the globe and turn it into summaries of scientific knowledge. Import citations, use machine learning to screen for randomized controlled trials, screen titles and abstracts, upload references, screen full text, extract data, conduct risk of bias, and export data.
  • EPPI Reviewer
    a web-based software program for managing and analysing data in literature reviews. Developed for all types of systematic reviews, it manages references, stores PDF files and facilitates qualitative and quantitative analyses such as meta-analysis and thematic synthesis. It also contains some new ‘text mining’ technology which is promising to make systematic reviewing more efficient.
  • Google Forms
    can be used to collect data elements from each study
  • Microsoft forms
    can be used to collect data elements from each study
  • Qualtrics
    can be used to collect data elements from each study
  • SUMARI
    The System for the Unified Management, Assessment and Review of Information (SUMARI) is the Joanna Briggs Institute's software for the systematic review of literature. Supporting 10 review types, it facilitates the entire review process, from protocol development, team management, study selection, critical appraisal, data extraction, data synthesis and writing your systematic review report.

Explaining the Synthesis & Analysis: Meta-analysis include synthesis which summarizes and organizes the information found in the studies that are identified. This synthesis informs the conclusions that are drawn in the analysis and focuses on both the methodology and results of the studies. Generally, synthesis and analysis involve looking for trends and patterns to use in comparisons, to discover explanatory or confounding variables, to develop themes or frameworks, to inform best practices, etc. Meta-analysis should be graphical and tabular with narrative commentary. Meta-analysis incorporates a numerical analysis of measures of effect assuming absence of heterogeneity.
Explaining the Synthesis & Analysis
Explanation
Narrative Commentary

Can describe trends, themes, frameworks, perspectives, characteristics, quality, etc.

Especially useful for empirical research.

Can use structured narratives.
Tabular explanations

Uses tables to explain the synthesis.

Can be used to describe study characteristics, study measures, study quality, study results, etc.

Accompanies narrative explanations.
Graphical explanations

Uses graphical methods to explore and present data.

Can include concept maps, forest plots, harvest plots, idea webs, logic models, mind maps, and network analysis.
Helpful Tools
  • Methods and Approaches for Synthesis and Analysis
  • Data and Digital Scholarship Information
    Baylor Libraries' Data Scholar Program & Workshops, Data Research Fellows
  • Center for Statistical Consulting
    Baylor Statistical Collaboration Center seeks to enhance the quality of research involving statistical methods conducted at Baylor University and with external partners. Provides short-term statistical analysis, long-term collaborative research, and assistance in the preparation of research grant proposals
  • EPPI Reviewer
    a web-based software program for managing and analysing data in literature reviews. Developed for all types of systematic reviews, it manages references, stores PDF files and facilitates qualitative and quantitative analyses such as meta-analysis and thematic synthesis. It also contains some new ‘text mining’ technology which is promising to make systematic reviewing more efficient.
  • NVivo
    NVivo is a qualitative data analysis (QDA) computer software package produced by QSR International. It has been designed for qualitative researchers working with very rich text-based and/or multimedia information, where deep levels of analysis on small or large volumes of data are required. (Descriptions from https://en.wikipedia.org/wiki/NVivo)
  • SUMARI
    The System for the Unified Management, Assessment and Review of Information (SUMARI) is the Joanna Briggs Institute's software for the systematic review of literature. Supporting 10 review types, it facilitates the entire review process, from protocol development, team management, study selection, critical appraisal, data extraction, data synthesis and writing your systematic review report.
  • OpenMeta[Analyst]
    Open Meta-Analyst is open-source software for performing meta-analyses of binary, continuous, or diagnostic data, using a variety fixed and random-effects methods, including Bayesian and maximum likelihood analysis. Open Meta-Analyst also enables you to do cumulative, leave-one-out, subgroup, and meta-regression analyses.

Summarizing: Summarizing your meta-analysis allows it to be shared with others.
Identifying Step
Explanation
Identify your audience(s)
  • Audience tends to be the research community and practitioners.
Choose your summary type
  • protocol (often registered)
  • report
  • presentation
  • peer review publication
Review Standards

Most standards will include these sections:

  • background/introduction
  • methodology including a PRISMA diagram
  • results
  • discussion/conclusions
Write Your Summary

Make sure your work is clear, auditable, replicable, and transparent.
Submit Your Summary
  • Summaries are usually submitted to peer-reviewed journals.
Search grey literature

Such as:

  • theses
  • conference proceedings
  • repositories
  • government documents
  • clinical trial registries

to address publication bias
Search other sources Such as reference lists, relevant journals that are not entirely indexed in databases, and researcher bibliographies.
Helpful Tools

Methods and Approaches for Synthesis and Analysis

While synthesis summarizes what the literature says, analysis assesses the strength of the evidence for drawing conclusions based on the synthesis.

Analytical Methods for Quantitative Studies

  • Meta analysis
  • Use of validity assessment
  • Checking with authors of primary studies
  • Subgroup analysis
  • Sensitivity analysis
  • Fixed effects analysis
  • Random effects analysis
  • Identifying gaps
  • Risk of bias of individual studies
  • Reporting bias because of dissemination issues

Synthesis Approaches for Qualitative Studies

  • Grounded theory - constant comparison
  • Meta-ethnography
  • Meta-synthesis
  • Meta-study
  • Logical analysis
  • Data analysis techniques
  • Metaphorical analysis
  • Domain analysis
  • Hermeneutical analysis
  • Analytic induction

Adapted from Booth, A. (2016). Systematic approaches to a successful literature review (Second edition.). Sage.

Meta Analysis Steps and Considerations

Meta analysis is a statistical technique that can be used to aggregate the results of individual quantitative studies.

  1. Deciding whether to base the analysis on a fixed- or random-effects model
    • Fixed effect if the factor is assumed constant across studies and variation among studies is random
  2. Computing a summary estimate of effect (if appropriate)
    • Weighted based on precision of estimates
    • Inverse variance weighting (sample size/measurement error)
  3. Presenting the data (usually graphically)
    • Usually a “forest plot” – point estimate and confidence interval of effect in each study
  4. Evaluating possible reasons for heterogeneity of study results (i.e. why different studies produce different estimates)
    • Cochran’s Q, Higgins I2
    • Subgroup analysis, stratified analysis
    • graphical (Galbraith plot, scatterplot of effect size vs factor)
    • meta regression (I2, adjusted R2, Tau2)
  5. Searching for evidence of publication bias and evaluating the influence that individual studies have on the outcome
    • Funnel plot of each study’s SE against estimated effect
    • Tests based on funnel plot (Begg, Egger)

Glossary of Terms

Analysis - assesses the strength of the evidence for drawing conclusions based on the synthesis. The trends and patterns can be used in comparisons, to discover explanatory or confounding variables, to develop themes or frameworks, to inform best practices, etc.

Bias - systematic error in research studies that can lead to erroneous conclusions. Can occur in clinical trials, systematic reviews, and all types of research.

Forrest Plot - also known as a blobogram. A graphical display designed to illustrate the relative strength of effects in multiple studies

Funnel plot - a graphical device for for exploring possible publication bias by plotting study size vs. effect size

Synthesis - summarizes what the literature says and informs the conclusions that are drawn in the systematic review and focuses on both the methodology and results of the studies

University Libraries

One Bear Place #97148
Waco, TX 76798-7148

(254) 710-6702

Osofast A-Z Databases OneSearch Profiles Library Account Login Hours
Baylor University

Copyright © Baylor® University. All rights reserved.

Report It | Title IX | Mental Health Resources | Anonymous Reporting | Legal Disclosures