If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Systematic reviews of the medical literature are key tools in comprehensively summarizing data and helping clinicians and policy makers to make informed, evidence-based decisions regarding patient care and health policy.
Systematic reviews often contain a meta-analysis, a statistical method that synthesizes the available data from independent studies in order to answer a specific research question. According to the Cochrane Collaboration—an international organization dedicated to promoting and managing systematic reviews and evidence-based medicine—a meta-analysis (i) provides more precise estimates of the effects of an intervention than those from individual studies alone and (ii) allows for investigation of consistencies and differences across studies (
We review the basic methodology behind high-quality systematic reviews and meta-analyses, explain the statistical methods and analyses involved in meta-analyses, and emphasize how this methodological tool can be used in the field of dermatology.
STUDY DESIGN METHODOLOGY: FORMULATING A CLINICAL QUESTION, ELIGIBILITY CRITERIA, AND SEARCH STRATEGY
Guidelines on how to report systematic reviews and meta-analyses have helped to establish criteria for developing, carrying out, and evaluating these studies. The first set of guidelines was published in 1999 under the title QUOROM (QUality Of Reporting Of Meta-analysis) Statement (
). In 2009 these guidelines further evolved into PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses), which established a 27-item checklist to aid researchers in planning and reporting studies (http://www.prisma-statement.org) (
). The PRISMA statement is also a useful tool for readers to reference when evaluating a systematic review or meta-analysis.
Formulating an appropriate and specific research question is key to any study and underpins systematic reviews and meta-analyses. A research question should be feasible, interesting, novel, ethical, and relevant, known as the FINER criteria (
in the Journal of Investigative Dermatology on the use of imiquimod for actinic keratosis, the population was patients with actinic keratosis (AK), the intervention was use of topical imiquimod, the comparator was placebo vehicle cream, the outcome measures were multiple and included complete clearance of lesions in treatment area, partial clearance, and complete clearance of a particular lesion, and the time-frame was number of weeks of treatment needed. Structuring a research question by addressing each of these elements is independently associated with better reporting quality in randomized controlled trials (RCTs) (
A hallmark of the systematic review is prespecified inclusion and exclusion criteria for studies, known as eligibility criteria. These criteria stem from the PICOT question above, defining the types of participants that will be included and the types of interventions to be studied and potentially also limiting the type of study design (
). To use the same study example as above, inclusion criteria for the systematic review by Hadley et al. were randomized, double-blind trials investigating imiquimod for AK with efficacy or safety data. Exclusion criteria were reviews with clinical information published elsewhere, biochemical or immunological studies, abstracts, and studies of conditions other than AK.
Once the research question and eligibility criteria have been established, a search strategy should be laid out. This strategy will specify the electronic databases to be searched, as well as the search terms to be used. The Cochrane Collaboration recommends searching the electronic databases CENTRAL and MEDLINE as a minimum, together with EMBASE, if available. Searches should also include national and regional databases, gray literature (such as technical reports or working papers from research groups or committees), relevant journals, conference abstracts, other reviews and guidelines, and ongoing studies in trial registries, over a specified period of time (
). As per PRISMA, authors are encouraged to contact study authors for clarification and identification of further studies, and they should explicitly announce such contact, as well as specify the date last searched (
). The search results should be presented in a flow diagram that clearly illustrates the number of studies included and the reasons for exclusion of studies that met initial search criteria (Figure 1).
After key studies have been identified for inclusion in the systematic review, two independent reviewers extract data from the studies. These results should be extracted on piloted standardized forms.
ASSESSING RISK OF BIAS IN INDIVIDUAL STUDIES
Having identified appropriate studies and extracted data in duplicate, the author should assess individual studies for bias. Both PRISMA and Cochrane caution against the use of numerical scales to assess the quality of studies, recommending a more descriptive approach of assessed methodological components. Studies should be assessed for selection bias (appropriate generation of random allocation sequence, concealment of this sequence, and intention-to-treat analyses), performance bias (blinding of participants and providers), detection bias (blinding of outcome assessment), attrition bias (loss to follow-up), and reporting bias (differences between reported and unreported findings) (
, “Prospective Registration and Outcome-Reporting Bias in Randomized Controlled Trials of Eczema Treatments: A Systematic Review.”
Meta-analysis allows data from multiple studies to be statistically combined. First, a summary measure is decided upon and a summary statistic is estimated for each study. Next, an overall intervention effect is calculated. Finally, to ensure that the results are valid and robust, researchers typically test for heterogeneity and publication bias and perform sensitivity analyses. Multiple software programs are available for meta-analysis; these may vary in terms of usability, although most give consistent numerical results (
The first step in data analysis is to decide on a summary measure and calculate a corresponding summary statistic for each study. The choice of a summary measure will depend on the clinical question and type of data. For example, for dichotomous or “yes-or-no” data (e.g., did the patient develop melanoma: yes or no?), the risk ratio, odds ratio, absolute risk reduction, and number needed to treat all commonly use summary measures. For survival or time-to-event data (e.g., the number of months until melanoma development), the hazard ratio is the best summary measure. Researchers should choose the measure that is most appropriate for their data and that will give a consistent estimate of the treatment effect for the clinical situation (
) and then calculate the same measure for each study. Each of the common summary statistics listed above can be reexpressed in terms of each other; however, when the unit of analysis differs among studies, care must be taken to avoid bias. For example, summary statistics may vary significantly based on the level of randomization (individual or group), number of treatment attempts, number of observations per patient, and whether multiple observations or body areas were included per patient.
Once summary statistics have been derived for each study, the overall intervention effect can be calculated. This effect is the pooled or weighted average of the effects estimated from the individual studies. Two commonly used approaches for combining data include fixed-effects and random-effects models. Fixed-effects models (such as Mantel–Haenszel and inverse variance approaches) account for only within-study variability. Random-effects models (such as DerSimonian, Laird, and Bayesian approaches) account for both within- and between-study variability and are generally considered more appropriate in settings of greater between-study heterogeneity, as described below. A more detailed summary of each of these statistical approaches can be found in the Cochrane Handbook for Systematic Reviews of Interventions (
The final steps in a meta-analysis are to assess for sources of bias and test the consistency of the results. Heterogeneity, or whether there is greater variation in the results between the studies than would be attributable to chance, may be due to differences between the participants, interventions, outcomes, study design and funding, or the statistical methods used. To assess heterogeneity, statistical methods such as the χ2 test and I2 test are commonly used. I2 values <40% are generally considered low; however, the statistic may be imprecise when few studies are being compared. In addition to heterogeneity, publication bias may also affect the results. Smaller negative studies are less likely to be published, and the treatment effect may be overestimated if such studies are not included. Funnel plots are often used to assess for publication bias (Figure 3). Finally, sensitivity analyses may be used to gauge the robustness of results, which may involve removing certain studies that are considered heterogeneous or of lower quality and then repeating the analysis.
LIMITATIONS OF META-ANALYSIS
Meta-analyses may be performed for various reasons. Combining data from multiple studies may increase the power or chance of detecting a real effect, improve precision, and help to settle controversy where individual studies show conflicting results. Nonetheless, there is the risk that the results may be misleading if the individual studies are subject to bias, if there is significant heterogeneity among studies, or if there is publication or reporting bias that is not addressed appropriately. It is important for authors to reference any identified sources of heterogeneity, as well as whether there are subgroups for which evidence is stronger than for others, and then decide whether there is sufficient evidence to draw clear conclusions.
SUMMARY AND USE OF SYSTEMATIC REVIEWS AND META-ANALYSES
In a busy health-care setting, systematic reviews and meta-analyses allow a health-care provider or researcher to understand available evidence in a synthesized, coherent manner (
). Whether you are performing a systematic review and meta-analysis or reading the literature in order to make a health-care or policy decision, it is important to understand the methodology behind these techniques. Understanding the methods will allow you to assess the quality of the systematic review or meta-analysis and the strength of the evidence behind it.
Finally, in using results from a systematic review and meta-analysis, either in a clinical practice or in a guideline development setting, conclusions are drawn, either implicitly or explicitly, regarding the strength of recommendations. There has been a recent movement to employ a systematic and explicit approach to making these judgments (
). Table 1 describes two commonly used systems: GRADE and SORT. We refer to these scoring systems here because their use is increasing and they are of growing importance to informed dermatologists; Cochrane reviews now employ GRADE methodology in their summary of findings.
Table 1Rating the strength of clinical recommendations
•Used by the World Health Organization in their guideline development process
•Classifies strength of recommendations into strong or weak. A strong recommendation means that based on the available evidence, clinicians are very certain that the benefits either do or do not outweigh the risks of an intervention
•Grades strength of recommendation into A, B, and C, with A-level evidence based on consistent, good-quality, patient-oriented evidence, B based on inconsistent or limited-quality, patient-oriented evidence, and C based on consensus, usual practice, or opinion