Advertisement

Cochrane Reviews and Dermatological Trials Outcome Concordance: Why Core Outcome Sets Could Make Trial Results More Usable

Open ArchivePublished:December 05, 2018DOI:https://doi.org/10.1016/j.jid.2018.11.019
      Evidence-based health care requires that relevant outcomes for patients are included in clinical trials investigating treatment effects, allowing subsequent systematic reviews to summarize all relevant evidence to guide clinical practice. Currently, no gold standard of outcome choice for dermatology trials and reviews exists. We systematically assessed concordance between efficacy outcomes in a random sample of 10 Cochrane Skin systematic reviews and the 220 dermatology trials included. Reviews did not include 742 (68%) of the 1,086 trial outcomes. Of the 60 outcomes the reviews sought, 17 (28%) were not reported in any trial, while 12 were assessed in <50% of trials. For 11 of 23 (48%) primary review outcomes, meta-analysis was impossible, because trial outcomes were absent or unclear. This small overlap of review/trial outcomes could suggest that trials are not measuring the outcomes perceived to be the most important by patients, clinicians, systematic reviewers, and trialists. The lack of standardized outcome measures, poor reporting of outcomes in trials, and low concordance of outcomes between reviews and primary studies could be improved by the development and implementation of Core Outcome Sets. These are an agreed-upon minimum set of key outcomes, for specified conditions, to be reported in all trials.

      Abbreviations:

      CI (confidence interval), COS (Core Outcomes Sets)

      Introduction

      In evidence-based health care, a key determining factor of the scientific value of clinical trials is the choice of outcomes. It is of the utmost importance for trials to choose outcomes that are considered relevant to patients, clinicians, and other health care professionals, as well as decision makers, such as commissioners and policy makers. Outcome measurement instruments must be reliable, valid, and feasible (
      • Boers M.
      • Kirwan J.R.
      • Tugwell P.
      The OMERACT Handbook. OMERACT.
      ,
      • Boers M.
      • Brooks P.
      • Strand C.V.
      • Tugwell P.
      The OMERACT filter for outcome measures in rheumatology.
      ). Trials using inappropriate outcomes may overestimate, underestimate, or overlook the effect of the intervention under investigation (
      • Sinha I.
      • Jones L.
      • Smyth R.L.
      • Williamson P.R.
      A systematic review of studies that aim to determine which outcomes to measure in clinical trials in children.
      ) and standardization is crucial in order to allow cross-trial comparisons in systematic reviews. Similarly, meta-analyses are only possible with comparable outcomes. Moreover, using different outcomes across clinical trials can hinder clinicians who consider trial data, systematic reviews, or meta-analyses in their clinical decisions or when formulating clinical practice guidelines. Low or unclear reliability, validity, and feasibility of many outcome measurement instruments and lack of standardization in outcome assessment in trials have been identified as significant barriers to evidence-based decision making (
      • Chalmers I.
      • Glasziou P.
      Avoidable waste in the production and reporting of research evidence.
      ).
      In the current absence of formal standardization of outcome reporting in clinical trials, for almost all dermatological diseases, systematic reviewers need to have an overview of the outcomes used in relevant trials before they finally choose review outcomes. Nonetheless, through the involvement of clinical and methodological experts and consumers, the choice of outcomes in Cochrane reviews generally tend to capture the key benefits and harms of treatments. A low degree of overlap between trial and review outcomes would potentially indicate that trials were not measuring the outcomes believed by the wider health care community to be particularly important. It would also raise the possibility of weaknesses in systematic review conclusions.
      A preliminary overview of the choice of outcomes in all 69 reviews of the Cochrane Skin Group published until January 2015 suggested a substantial mismatch between the outcomes of interest to the review and those reported in the included clinical trials (
      • Schmitt J.
      • Deckert S.
      • Alam M.
      • Apfelbacher C.
      • Barbaric J.
      • Bauer A.
      • et al.
      Report from the kick-off meeting of the Cochrane Skin Group Core Outcome Set Initiative (CSG-COUSIN).
      ). Only 67% (n = 271) of 402 predefined review outcomes were found in at least one component trial of the reviews. There are almost no mutually agreed-upon standards for outcome selection in dermatological research. Such agreements aim to make trials relevant and comparable and the evidence more usable for clinical decision making. Our preliminary overview was based on the database of the Cochrane Collaboration, which provides information that can be used to investigate the overlap of review outcomes with trial outcomes, but not vice versa.
      The number of outcomes that trials report but are not included in systematic reviews, and the subsequent waste of research effort, is unknown and it was not addressed in our preliminary overview. If a sufficient lack of concordance were to exist between trial and review outcomes, then the development of clinical guidelines could suffer. Consequently, primary researchers could lose motivation if significant amounts of their research were arguably wasted by failing to become incorporated into clinical guidelines. Funding bodies wishing to see a demonstrable impact for their money may likewise wish to ensure that trial outcomes are likely to be used by policy makers and guideline developers.
      The primary aim of this study was to systematically investigate the overlap between outcomes reported in trials and those sought by Cochrane reviews and vice versa. Our secondary aim was to conduct a meta-epidemiologic study into whether the pooled treatment effect differs between trials that are concordant with review primary outcomes versus those that are not.

      Results

      The article selection process is displayed in Figure 1. A summary of the Cochrane reviews, efficacy primary outcomes, and meta-analyses is given in Table 1 and the characteristics of the included trials are shown in Supplementary Table 1 online. A total of 20 of 242 (8%) trials were excluded from the study due to the absence of full texts (n = 7), not being published in a language native to the study’s researchers (n = 8) (i.e., English, German, Italian, or Spanish) or containing no relevant information (n = 5). In one review (
      • Bamford J.T.
      • Ray S.
      • Musekiwa A.
      • van Gool C.
      • Humphreys R.
      • Ernst E.
      Oral evening primrose oil and borage oil for eczema.
      ), 11 of the trials evaluated were industry reports. All other trials were published in scientific journals. In the Cochrane reviews, 23 outcomes were specified as primary (Table 1, Supplementary Table 2 online) and 37 as secondary (Supplementary Table 3 online). Of note is that some texts contained information on more than one trial.
      Table 1Summary of included Cochrane reviews, primary efficacy outcomes, and realization meta-analyses
      Cochrane Review (number of trials included in this study/number in review)Mention of primary outcome in abstract or plain language summaryPrimary review outcome(s)If sub-group or meta-analysis was conducted, number of trials included/not includedReason for non-inclusion of trials in meta-analysisIf no meta-analysis was conducted, reason
      • Bamford J.T.
      • Ray S.
      • Musekiwa A.
      • van Gool C.
      • Humphreys R.
      • Ernst E.
      Oral evening primrose oil and borage oil for eczema.
      (atopic eczema) (26/27)
      In plain language summaryPrimary efficacy outcome I: Global degree of improvement in symptoms and signs as rated by participant or medical doctorParticipant: 12 of 27 reported primary efficacy outcome, 7 of 12 included in meta-analysis. 2 subgroup analyses (1 included 2 trials, 1 included 5 trials).

      Clinician: 19 of 27 reported primary efficacy outcome 8 of 19 included in meta-analysis. 2 subgroup analyses (1 included 3 trials, 1 included 5 trials)
      Trials reported this outcome, but in different ways
      Primary efficacy outcome II: Improvement in quality of lifeNo meta-analysisOnly 2 trials reported the primary outcome but these could not be pooled as they compared different interventions
      • Bath-Hextall F.J.
      • Jenkinson C.
      • Humphreys R.
      • Williams H.C.
      Dietary supplements for established atopic eczema.
      (atopic eczema) (10/11)
      In plain language summaryPrimary efficacy outcome I: Degree of long-term (over 6 mo) control, such as reduction in number of flares or reduced need for other treatmentsNo meta-analysisFor 9 of 10 trials no data was available for the primary outcomes
      Primary efficacy outcome II: Short-term (within 6 wk). Changes in participant-rated or parent-rated symptoms of atopic eczema, such as pruritus (itching) or sleep loss
      • Chen X.
      • Yang M.
      • Cheng Y.
      • Liu G.J.
      • Zhang M.
      Narrow-band ultraviolet B phototherapy versus broad-band ultraviolet B or psoralen-ultraviolet A photochemotherapy for psoriasis.
      (psoriasis) (13/13)
      In abstractPrimary efficacy outcome I: Participant-rated global improvement.No meta-analysisNone of the 13 included trials addressed this primary outcome
      Primary efficacy outcome II: Percentage of participants reaching Psoriasis Area and Severity Index 75.No meta-analysisOnly 2 of 13 trials with different interventions addressed this primary outcome
      Primary efficacy outcome III: Clearance rate, defined as no lesions of psoriasis or minimal residual activity10 from 13 studies assessed this primary outcome;

      4 studies included in meta-analysis: 2 meta-analysis with each 2 studies each subgroup 1: 2/1

      Subgroup 2: 2/0
      None or only one study per intervention, once because of heterogeneity between the studies
      • Chi C.C.
      • Kirtschig G.
      • Baldo M.
      • Brackenbury F.
      • Lewis F.
      • Wojnarowska F.
      Topical interventions for genital lichen sclerosus.
      (lichen sclerosus (genital) (7/7)
      NoPrimary efficacy outcome I: Participant-rated improvement or remission of symptoms (in terms of quality of life, pain, itching, and less pain with intercourse)6 of 7 studies assessed the outcome

      2 studies included in 1 meta-analysis
      1 study did not evaluate this outcome
      Primary efficacy outcome II: Investigator-rated global degree of improvement (in terms of pallor, purpura, hyperkeratosis, ulceration, erosion, erythema, sclerosis, and scarring)7 of 7 studies assessed the outcome

      No meta-analysis

      Paper reports in text that “All seven included studies reported investigator-rated improvement of gross appearance,” however, again, variable language used in meta-analysis for this outcome
      • Eekhof J.A.
      • Van Wijk B.
      • Knuistingh Neven A.
      • van der Wouden J.C.
      Interventions for ingrowing toenails.
      (ingrowing toenails) (21/24)
      In abstract and plain language summaryPrimary efficacy outcome I: Relief of symptomsNo meta-analysis
      Primary efficacy outcome II: Recurrence

      Primary efficacy outcome III: Regrowth (including nail spicules/nail spikes)
      16 from 24 studies assessed primary outcome recurrence;

      1 meta-analyses subgroup 1: 2/1

      No meta-analysis
      • Ersser S.J.
      • Cowdell F.
      • Latter S.
      • Gardiner E.
      • Flohr C.
      • Thompson A.R.
      • et al.
      Psychological and educational interventions for atopic eczema in children.
      (atopic eczema) (9/10)
      In abstractPrimary efficacy outcome I: participant-rated global assessmentNo meta-analysisNone of the 9 included studies addressed this primary outcome
      Primary efficacy outcome II: reduction in disease severity, determined by an assessor using an objective measureNo meta-analysis

      8 of 9 studies assessed severity (6x SCORAD)
      Methodological weaknesses in the selected studies; heterogeneity of the outcome measures; and the heterogeneous nature of the interventions
      Primary efficacy outcome III: improvement in sleepNo meta-analysisNone of the 9 included studies addressed this primary outcome
      Primary efficacy outcome IV: improvement in quality of life (or reduction in distress of the child and parent)No meta-analysis

      7 from 9 studies assessed quality of life
      Methodological weaknesses in the selected studies; heterogeneity of the outcome measures; and the heterogeneous nature of the interventions
      • Kastarinen H.
      • Oksanen T.
      • Okokon E.O.
      • Kiviniemi V.V.
      • Airola K.
      • Jyrkka J.
      • et al.
      Topical anti-inflammatory agents for seborrhoeic dermatitis of the face or scalp.
      (seborrheic dermatitis (face and scalp) (35/36)
      In abstractPrimary efficacy outcome I: Total clearance (total resolution of symptoms), evaluated by an outcome assessor (expressed as a percentage of people treated)27 of 36 trials assessed the outcome total clearance

      15 trials included in 4 meta-analyses
      1 study excluded as mode of application was different from all other trials

      1 study excluded as clearance defined as ≥75%, which was less than in other included trials
      Primary efficacy outcome II: Disease severity scores for scaling, pruritus, or erythema at the end of treatment as (evaluated by participant self-report, outcome assessor, or both)28 of 36 trials assessed one of the outcome (scaling, pruritus or erythema)

      7 trials included in 3 meta-analyses
      • Kwok C.S.
      • Gibbs S.
      • Bennett C.
      • Holland R.
      • Abbott R.
      Topical treatments for cutaneous warts.
      (common warts, cutaneous warts [non-genital]) (70/85)
      No (efficacy mentioned, but no further details given)Primary efficacy outcome I: Clinical cure (defined as complete disappearance of elevated/warty skin) at end of treatment period37 of 85 assessed this primary outcome

      Meta-analysis: 35 included in meta-analysis: 10 meta-analysis plus subgroup-analysis with meta-analysis
      The heterogeneity of the trials made it difficult to perform statistical pooling of the data
      Primary efficacy outcome II + III: Participant satisfaction/ dissatisfaction/quality of lifeNo meta-analysis
      • Martin L.K.
      • Werth V.
      • Villanueva E.
      • Segall J.
      • Murrell D.F.
      Interventions for pemphigus vulgaris and pemphigus foliaceus.
      (pemphigus vulgaris and pemphigus foliaceus) (10/11)
      NoPrimary efficacy outcome I: The proportion of participants achieving remission (defined as the absence of lesions or the presence of transient new lesions that heal within 1 wk, while the person is receiving minimal therapy)3 of 11 trials evaluated the specified outcome measure

      2 trials included in meta-analysis (but with the data from only one study, because the other is not estimable)
      • Sasse A.D.
      • Sasse E.C.
      • Clark L.G.
      • Ulloa L.
      • Clark O.A.
      Chemoimmunotherapy versus chemotherapy for metastatic malignant melanoma.
      (metastatic malignant melanoma) (18/18)
      NoPrimary efficacy outcome I: Overall survival (number of participants alive at end of trial)8 of 18 trials evaluated the outcome overall survival, all 8 trials were included in a meta-analysis
      Our assessments of the trials’ risks of bias were given as “high” in 7 of 10 reviews and “unclear” in the remaining three reviews. Risks of bias were noted to be particularly high across all trials for the blinding of participants and personnel (Supplementary Table 4 online). It was also “high or unclear” in the allocation of sequence concealment, the blinding of trial outcome assessment, and other potential threats to validity.

      Lack of overlap between trial outcomes and review-sought outcomes

      Figure 2 shows that of the 1,086 outcomes identified among 220 trials, 742 (68%) were not included in the reviews. The median number of outcomes per trial was 4 (range 1–34). Of the 60 outcomes the reviews sought, 17 (28%) were not reported in any trial.
      Figure thumbnail gr2
      Figure 2Usability of trial outcomes by systematic reviews.

      Missing outcomes

      Supplementary Table 2 shows that 3 of 23 (13%) outcomes that reviews sought as primary were missing in every relevant trial. These were “participant-rated global improvement” (
      • Chen X.
      • Yang M.
      • Cheng Y.
      • Liu G.J.
      • Zhang M.
      Narrow-band ultraviolet B phototherapy versus broad-band ultraviolet B or psoralen-ultraviolet A photochemotherapy for psoriasis.
      [psoriasis]), “improvement in sleep” (
      • Ersser S.J.
      • Cowdell F.
      • Latter S.
      • Gardiner E.
      • Flohr C.
      • Thompson A.R.
      • et al.
      Psychological and educational interventions for atopic eczema in children.
      [atopic eczema]) and “participant dis/satisfaction” (
      • Kwok C.S.
      • Gibbs S.
      • Bennett C.
      • Holland R.
      • Abbott R.
      Topical treatments for cutaneous warts.
      [warts]). The mean presence in the trials of any specified review-sought primary outcome, irrespective of whether reported by trials as primary or secondary, was 45% (95% confidence interval [CI] = 32–59%).
      Fourteen of 37 (38%) secondary review outcomes were missing completely (Supplementary Table 3). The mean presence in the trials of any specified review-sought secondary outcome, irrespective of whether reported by trials as primary or secondary, was 23% (95% CI = 14–32%). The review with the most secondary trial outcomes (80%) was for pemphigus (
      • Martin L.K.
      • Werth V.
      • Villanueva E.
      • Segall J.
      • Murrell D.F.
      Interventions for pemphigus vulgaris and pemphigus foliaceus.
      ) and the review with the least (21%) was for psoriasis (
      • Chen X.
      • Yang M.
      • Cheng Y.
      • Liu G.J.
      • Zhang M.
      Narrow-band ultraviolet B phototherapy versus broad-band ultraviolet B or psoralen-ultraviolet A photochemotherapy for psoriasis.
      ).

      Lack of concordance

      Of the total 1,086 trial outcomes that this study identified, those that the Cochrane authors defined as primary were seen 215 times. Thirty-one of 215 (14%) of these were reported by the trials as primary. The outcomes that the reviews defined as secondary were present 129 times in the trials, but only reported as secondary in 16 (12%) trials.
      Of the 23 outcomes sought as primary by the reviews, 11 (48%) were not reported as such by the trials. Thirty of 37 (81%) outcomes sought as secondary by the reviews were not reported as such by the trials.
      The degree of concordance between reviews and trials agreeing on which outcomes should be primary or secondary is displayed in Figure 3. The planned meta-epidemiologic analyses for the secondary objective of the study were not possible due to this lack of data.
      Figure thumbnail gr3
      Figure 3Concordance between reviews seeking outcomes reported as primary or secondary and trials reporting them as such.
      The primary review outcome seen the most as a primary trial outcome (“reduction in disease severity as measured [objectively] by a trained assessor”) was present in 33% of trials assessed in the review by
      • Ersser S.J.
      • Cowdell F.
      • Latter S.
      • Gardiner E.
      • Flohr C.
      • Thompson A.R.
      • et al.
      Psychological and educational interventions for atopic eczema in children.
      (atopic eczema). The mean proportion of concordance between trial outcomes being sought as primary by reviews and being reported as primary by trials was 7% (95% CI = 3–12%).
      The secondary review outcome seen the most (“response rates [partial and complete]”) was reported as “secondary” in 8% of trials assessed in the review by
      • Sasse A.D.
      • Sasse E.C.
      • Clark L.G.
      • Ulloa L.
      • Clark O.A.
      Chemoimmunotherapy versus chemotherapy for metastatic malignant melanoma.
      (metastatic malignant melanoma). The mean concordance between secondary review outcomes and secondary trial outcomes was 2% (95% CI = 0–4%).

      Loss of potential data for systematic reviews

      The Cochrane authors defined the outcomes they wished to consider in their reviews. In then selecting the trials to assess, they implicitly established the maximum number of times they could have seen these outcomes. This potential figure can be derived as “number of outcomes sought by the review” multiplied by “number of trials included in the review.” Figure 4 illustrates how much of these data were indeed seen by the Cochrane reviews. Irrespective of whether trials reported outcomes as primary or secondary, the mean value of potential realized was only 35% (95% CI = 27–43%). For trial outcomes specified as primary or secondary in concordance with the reviews' requirements, the mean value of potential realized was 5% (95% CI = 2–7%).
      Figure thumbnail gr4
      Figure 4Proportions of outcome concordance between Cochrane Skin Reviews and trials included.
      The Cochrane reviewers frequently commented on the difficulties they faced due to absent or significantly heterogeneous trial outcomes and due to insufficient focus on patient-centered or quality-of-life matters (Supplementary Table 5 online). In every single Cochrane review included in this study trial, inconsistencies; lack of standardization; and insufficient reporting were identified as major impediments for summarizing research evidence.

      Outcome domains

      An example of an outcome domain is Quality of Life, which would contain any outcome or measure that assessed quality of life, irrespective of the actual instrument used. The trial outcomes considered by the reviews fell into several domains and these are shown in Supplementary Table 6 online. This displays that despite a lack of concordance for many specific outcomes, there was better agreement at the outcome domain level. For example, 27 trials into atopic eczema were assessed in
      • Bamford J.T.
      • Ray S.
      • Musekiwa A.
      • van Gool C.
      • Humphreys R.
      • Ernst E.
      Oral evening primrose oil and borage oil for eczema.
      ; Global Disease Severity and Single Symptom were domains containing the primary outcomes defined by the Cochrane review and domains containing outcomes reported in 22 (85%) and 20 (77%) of trials, respectively.

      Discussion

      In evaluating the efficacy of interventions, evidence from randomized clinical trials represents scientifically rigorous evidence from a primary study design. Typically, clinical trials are designed to test the effect of exposure to a defined intervention versus control on specific primary efficacy or safety outcome(s). The trial design, statistical analysis employed, and choice of sample size is focused on investigating the effect of an intervention on a primary outcome. The results of a trial can only be interpreted as confirmatory for this primary outcome (
      • Hanson B.
      The art of choosing sound study endpoints.
      ). Effects of the intervention on secondary trial outcomes have to be interpreted as exploratory or hypothesis generating. The high level of evidence produced by clinical trials, therefore, applies predominantly to the primary outcome(s). From a methodological perspective, pooled evidence from confirmatory trials (i.e., trials with the same primary outcome as the review) may allow more valid conclusions regarding the efficacy of interventions than pooled evidence from exploratory trials (i.e., trials with a different primary outcome than the review). Meta-epidemiologic evidence on this critical research question is missing.

      Main findings

      Our study set out to analyze the degree of overlap between trial outcomes and those sought by reviews for a broad spectrum of dermatological conditions, including psoriasis, melanoma, atopic eczema, pemphigus, seborrheic dermatitis, and cutaneous warts.
      We identified a low degree of agreement between trial and review outcomes. This could indicate a vulnerability of systematic review conclusions. As the choice of outcomes in Cochrane reviews is designed to capture the key benefits and harms of treatments, this small overlap of review/trial outcomes could also suggest that many trials are not measuring the outcomes perceived to be the most important by patients, clinicians, and other allied health care professionals.

      Missing outcomes and loss of potential data for reviews

      Our results demonstrate that a significant number of outcomes sought by reviews are missing in dermatology trials. The systematic review of any topic is impossible when the required outcomes are not present in the published trials. Indeed, for this reason, and the heterogeneity of those trial outcomes that were actually reported, no meta-analysis was possible for 11 of 23 (48%) of the primary review outcomes. This represents a sizeable loss of possible conclusions to a review and to the potential subsequent evidence-based guidelines. Ninety percent of reviews in our study were missing >50% of outcome data that they would have had if every trial had reported every outcome the reviews wanted to assess. This finding is worse than that seen in a meta-analysis of general systematic reviews (
      • Kirkham J.J.
      • Dwan K.M.
      • Altman D.G.
      • Gamble C.
      • Dodd S.
      • Smyth R.
      • et al.
      The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews.
      ,
      • Kirkham J.J.
      • Gargon E.
      • Clarke M.
      • Williamson P.R.
      Can a core outcome set improve the quality of systematic reviews?—a survey of the co-ordinating editors of Cochrane Review groups.
      ). In this, only 25% of reviews were missing >50% of potential data.

      Limited usability of research effort

      Sixty-eight percent of all trial outcomes not being included in the reviews reflects how much research effort went no further than the original trial, and leads to the question of how trialists decide which outcomes to use in their study.

      Lack of concordance

      The significant lack of concordance between reviews looking to assess specific outcomes as primary or secondary and trials investigating them as such had a substantial impact on the susbsequent reviews and meta-analyses. Review authors directly commented on this.
      Mutual agreement of core outcomes, rather than being based solely on a review author’s opinion, would particularly facilitate a relevancy informed by multiple viewpoints. This, in turn, may support better uptake and implementation by researchers of the concept.

      Risk of bias

      Outcome reporting bias, the selection for publication of a subset of the originally recorded outcome variable (
      • Williamson P.R.
      • Gamble C.
      Identification and impact of outcome selection bias in meta-analysis.
      ), was more prevalent in our study than in a meta-analysis of general systematic reviews. This found 34% of assessed reviews contained at least one trial with high suspicion of outcome reporting bias (
      • Kirkham J.J.
      • Dwan K.M.
      • Altman D.G.
      • Gamble C.
      • Dodd S.
      • Smyth R.
      • et al.
      The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews.
      , recommended by Cochrane for systematic reviewers). The corresponding figure for our study was 70%.

      Outcome domains

      There was broader concordance between the trials and reviews with regard to outcome domains than there was concordance for defined specific outcomes. This suggests that trialists and reviewers are, in general, considering similar areas of particular diseases rather than completely different aspects. As such, this implies that better concordance of specific outcomes is eminently achievable.

      Limitations

      We decided to analyze Cochrane reviews because the Cochrane Handbook makes patient representation and involvement in review teams mandatory so that the choice of outcomes chosen in these reviews can be assumed to reflect the patient perspective. The 10 systematic reviews that we assessed represented only 15% of the 69 Cochrane Skin systematic reviews published during the time period of sampling for our study. Since then, 37 more have been published.
      Our allocation of trial outcomes to systematic review outcome domains was performed iteratively. A uniform taxonomy was missing and allocation had the potential to be piecemeal.
      Whether results apply to non-Cochrane reviews is unclear. For Cochrane, authors are encouraged to decide which primary review outcomes should be measured before looking at the data and to publish their intentions in a protocol. Non-Cochrane reviews may not do this and be more data-driven, basing their choice on what is available. This could lead to better concordance but not necessarily better-evidenced reviews. Evaluating the discordance in outcomes is particularly relevant for evidence-based practice.
      Outcome identification in the primary trials was sometimes challenging, especially when the methods sections were not well developed. This difficulty may have led to under-reporting of outcomes, if they were not clearly designated as such.

      Implication for further research and perspective

      Our study found a low degree of overlap between trial and review outcomes. This could facilitate weaknesses in systematic review conclusions due to insufficient data. Core Outcomes Sets (COS) may be a solution here and they are increasingly being developed to standardize outcomes across trials and to allow cross-trial comparisons. In dermatology, the HOME (Harmonizing Outcome Measures in Eczema) initiative pioneered COS development (
      • Schmitt J.
      • Apfelbacher C.
      • Spuls P.I.
      • Thomas K.S.
      • Simpson E.L.
      • Furue M.
      • et al.
      The Harmonizing Outcome Measures for Eczema (HOME) roadmap: a methodological framework to develop core sets of outcome measurements in dermatology.
      ). Other groups have started COS development in other fields of dermatology, but none of these groups has yet implemented their COS.
      We suggest that the main effort of future research in this field should be not only the development but also the implementation of COS in dermatology. The Cochrane Skin–Core Outcome Set Initiative has recently been established to improve and standardize outcome measurement in clinical trials and to make trial evidence more usable (
      • Kottner J.
      • Jacobi L.
      • Hahnel E.
      • Alam M.
      • Balzer K.
      • Beeckman D.
      • et al.
      Core outcome sets in dermatology: report from the second meeting of the International Cochrane Skin Group Core Outcome Set Initiative.
      ,
      • Schmitt J.
      • Deckert S.
      • Alam M.
      • Apfelbacher C.
      • Barbaric J.
      • Bauer A.
      • et al.
      Report from the kick-off meeting of the Cochrane Skin Group Core Outcome Set Initiative (CSG-COUSIN).
      ). Further aims are to develop clinically relevant and patient-centered sets of dermatological trial outcomes and improve the quality and interpretability of systematic reviews. Ultimately, the objective is to facilitate dermatological research delivering a tangible clinical impact for patients through evidence-based health care.
      The implementation of COS requires high acceptability of all stakeholder groups, including key clinicians, researchers, and patient representatives; but also regulators (US Food and Drug Administration, European Medicines Agency), pharmaceutical industry, and journal editors. Journal editors have been, for example, involved in the first HOME Delphi study published in this Journal (
      • Schmitt J.
      • Langan S.
      • Stamm T.
      • Williams H.C.
      Harmonizing Outcome Measurements in Eczema Delphi panel. Core outcome domains for controlled trials and clinical recordkeeping in eczema: international multiperspective Delphi consensus process.
      ). Ideally, journal editors should encourage the application of existing COS in their guidelines for authors. Funders of clinical trials already do so in the United Kingdom and in Germany.
      The Cochrane Skin Core Outcome Set Initiative will primarily achieve this by supporting research groups as they develop a COS. Specifically, Cochrane Skin Core Outcome Set Initiative provides expert and ongoing peer-based methodological advice (e.g., regarding outcome measurement [
      • Grinich E.E.
      • Schmitt J.
      • Kuster D.
      • Spuls P.I.
      • Williams H.C.
      • Chalmers J.R.
      • et al.
      Standardized reporting of the Eczema Area and Severity Index (EASI) and the Patient-Oriented Eczema Measure (POEM): a recommendation by the Harmonising Outcome Measures for Eczema (HOME) Initiative.
      ]), a development pathway and administrative support for the lifespan of the COS projects. Its structured and experienced approach can also help reduce bias risk. Core outcomes and, crucially, measurement instruments are currently in progress for 16 disease areas (including acne, chronic spontaneous urticaria, chronic wounds, atopic and hand eczema, melanoma, rosacea, hidradenitis suppurativa, nail psoriasis, and basal cell carcinomas). The Cochrane Skin Core Outcome Set Initiative can further help to reduce the risk of bias, which in our study was also worse than in others. Specifically, the impact of selective reporting of outcomes seen in outcome reporting bias could be reduced (
      • Williamson P.R.
      • Altman D.G.
      • Blazeby J.M.
      • Clarke M.
      • Devane D.
      • Gargon E.
      • et al.
      Developing core outcome sets for clinical trials: issues to consider.
      ). This has been shown to hamper systematic reviews significantly (
      • Kirkham J.J.
      • Dwan K.M.
      • Altman D.G.
      • Gamble C.
      • Dodd S.
      • Smyth R.
      • et al.
      The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews.
      ).
      The Cochrane Skin Core Outcome Set Initiative initiative is constantly open to new members looking to develop a COS, join a current project group, use a specific COS, receive methodological advice, or join as a patient representative (www.cs-cousin.org).

      Materials and Methods

      This study was an investigation of a random sample of systematic Cochrane Skin Group Reviews and the component trials.

      Protocol and registration

      A review protocol was developed and published at PROSPERO in September 2015 (CRD42015025005).

      Data source and data collection

      A random sample of 10 systematic reviews of the Cochrane Skin Group published until January 2015 and their component trials. A random sample was drawn, using the statistical software R, from all 69 Cochrane Skin Group reviews in dermatological diseases published until January 2015. The 10 reviews included 242 primary studies (range 7–85 studies). Only trials written in English, German, Italian, or Spanish were included.

      Data extraction

      Where possible, Cochrane provided clinical trial data from the Archie database (https://archie.cochrane.org/). This consisted of data originally extracted by the systematic review team from the trials they included during their reviewing process. Independent reviewers extracted data at both trial and systematic review levels and data were double extracted.

      Review global-data extraction

      The following information was extracted from each review; whether a meta-analysis was performed; the reasons why no meta-analysis was performed as applicable, specifically “heterogeneity” in outcome measurement across trials as the reason; the number of trials included in the meta-analysis of the primary efficacy outcome of the systematic review; and the number of trials not included in the meta-analysis of the primary efficacy outcome (total and due to no assessment of primary review outcome in underlying trials).

      Trial global-data extraction

      For each trial, we extracted published data on general characteristics (year, geographical bias [assessed by the Cochrane risk of bias instrument]), outcomes (i.e. primary, secondary, not specified), and results (with respect to the primary outcome the number of patients randomized and the number of events per group).
      If outcomes were not explicitly classified as primary or secondary, we assumed an outcome was a primary outcome when sample size calculations were based on this outcome. Otherwise, the outcome was classified as “not specified.” If there was a reference to a study protocol in the trial, we also obtained this to extract the relevant data.

      Outcome extraction

      For each trial and review outcome, we extracted the outcome domain(s), measurement instrument or outcome definition, number of participants and events in both the intervention and the control group (for reviews only applicable if meta-analysis has been conducted), and whether the outcome was the primary or secondary outcome, or if there was no information in this respect.

      Protocol deviation

      We had planned to compare pooled outcomes from meta-analysis of all trials to the pooled effects of (i) component trials with the primary review outcome used as primary, (ii) secondary, or (iii) unspecified outcome with the 2-step meta-epidemiologic approach (
      • Savovic J.
      • Jones H.
      • Altman D.
      • Harris R.
      • Juni P.
      • Pildal J.
      • et al.
      Influence of reported study design characteristics on intervention effect estimates from randomised controlled trials: combined analysis of meta-epidemiological studies.
      ). The planned meta-epidemiologic analyses, however, for the secondary objective of the study became unfeasible due to lack of data.

      Data collection process

      Outcomes were extracted in duplicate independently from the original publications by two reviewers using a standardized electronic template in Microsoft Access (Microsoft, Redmond, WA). Then both data sets were compared and all areas of disagreements were resolved.

      Conflicts of Interest

      Christian Apfelbacher received consultancy fees from Dr. Wolff GmbH for consulting on patient-reported outcomes in skin disease. The remaining authors state no conflict of interest.

      Acknowledgments

      We thank the Cochrane Skin management team for providing access to Archie data. We also thank Jacqui Moreau for input into the study protocol. This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

      Supplementary Material

      References

        • Bamford J.T.
        • Ray S.
        • Musekiwa A.
        • van Gool C.
        • Humphreys R.
        • Ernst E.
        Oral evening primrose oil and borage oil for eczema.
        Cochrane Database Syst Rev. 2013; : CD004416
        • Bath-Hextall F.J.
        • Jenkinson C.
        • Humphreys R.
        • Williams H.C.
        Dietary supplements for established atopic eczema.
        Cochrane Database Syst Rev. 2012; : CD005205
        • Boers M.
        • Brooks P.
        • Strand C.V.
        • Tugwell P.
        The OMERACT filter for outcome measures in rheumatology.
        J Rheumatol. 1998; 25: 198-199
        • Boers M.
        • Kirwan J.R.
        • Tugwell P.
        The OMERACT Handbook. OMERACT.
        2014
        • Chalmers I.
        • Glasziou P.
        Avoidable waste in the production and reporting of research evidence.
        Lancet. 2009; 374: 86-89
        • Chen X.
        • Yang M.
        • Cheng Y.
        • Liu G.J.
        • Zhang M.
        Narrow-band ultraviolet B phototherapy versus broad-band ultraviolet B or psoralen-ultraviolet A photochemotherapy for psoriasis.
        Cochrane Database Syst Rev. 2013; : CD009481
        • Chi C.C.
        • Kirtschig G.
        • Baldo M.
        • Brackenbury F.
        • Lewis F.
        • Wojnarowska F.
        Topical interventions for genital lichen sclerosus.
        Cochrane Database Syst Rev. 2011; : CD008240
        • Eekhof J.A.
        • Van Wijk B.
        • Knuistingh Neven A.
        • van der Wouden J.C.
        Interventions for ingrowing toenails.
        Cochrane Database Syst Rev. 2012; : CD001541
        • Ersser S.J.
        • Cowdell F.
        • Latter S.
        • Gardiner E.
        • Flohr C.
        • Thompson A.R.
        • et al.
        Psychological and educational interventions for atopic eczema in children.
        Cochrane Database Syst Rev. 2014; : CD004054
        • Grinich E.E.
        • Schmitt J.
        • Kuster D.
        • Spuls P.I.
        • Williams H.C.
        • Chalmers J.R.
        • et al.
        Standardized reporting of the Eczema Area and Severity Index (EASI) and the Patient-Oriented Eczema Measure (POEM): a recommendation by the Harmonising Outcome Measures for Eczema (HOME) Initiative.
        Br J Dermatol. 2018; 179: 540-541
        • Hanson B.
        The art of choosing sound study endpoints.
        Injury. 2008; 39: 656-658
        • Kastarinen H.
        • Oksanen T.
        • Okokon E.O.
        • Kiviniemi V.V.
        • Airola K.
        • Jyrkka J.
        • et al.
        Topical anti-inflammatory agents for seborrhoeic dermatitis of the face or scalp.
        Cochrane Database Syst Rev. 2014; : CD009446
        • Kirkham J.J.
        • Dwan K.M.
        • Altman D.G.
        • Gamble C.
        • Dodd S.
        • Smyth R.
        • et al.
        The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews.
        BMJ. 2010; 340: c365
        • Kirkham J.J.
        • Gargon E.
        • Clarke M.
        • Williamson P.R.
        Can a core outcome set improve the quality of systematic reviews?—a survey of the co-ordinating editors of Cochrane Review groups.
        Trials. 2013; 14: 21
        • Kottner J.
        • Jacobi L.
        • Hahnel E.
        • Alam M.
        • Balzer K.
        • Beeckman D.
        • et al.
        Core outcome sets in dermatology: report from the second meeting of the International Cochrane Skin Group Core Outcome Set Initiative.
        Br J Dermatol. 2018; 178: e279-e285
        • Kwok C.S.
        • Gibbs S.
        • Bennett C.
        • Holland R.
        • Abbott R.
        Topical treatments for cutaneous warts.
        Cochrane Database Syst Rev. 2012; : CD001781
        • Martin L.K.
        • Werth V.
        • Villanueva E.
        • Segall J.
        • Murrell D.F.
        Interventions for pemphigus vulgaris and pemphigus foliaceus.
        Cochrane Database Syst Rev. 2009; : CD006263
        • Sasse A.D.
        • Sasse E.C.
        • Clark L.G.
        • Ulloa L.
        • Clark O.A.
        Chemoimmunotherapy versus chemotherapy for metastatic malignant melanoma.
        Cochrane Database Syst Rev. 2007; : CD005413
        • Savovic J.
        • Jones H.
        • Altman D.
        • Harris R.
        • Juni P.
        • Pildal J.
        • et al.
        Influence of reported study design characteristics on intervention effect estimates from randomised controlled trials: combined analysis of meta-epidemiological studies.
        Health Technol Assess. 2012; 16: 1-82
        • Schmitt J.
        • Apfelbacher C.
        • Spuls P.I.
        • Thomas K.S.
        • Simpson E.L.
        • Furue M.
        • et al.
        The Harmonizing Outcome Measures for Eczema (HOME) roadmap: a methodological framework to develop core sets of outcome measurements in dermatology.
        J Invest Dermatol. 2015; 135: 24-30
        • Schmitt J.
        • Deckert S.
        • Alam M.
        • Apfelbacher C.
        • Barbaric J.
        • Bauer A.
        • et al.
        Report from the kick-off meeting of the Cochrane Skin Group Core Outcome Set Initiative (CSG-COUSIN).
        Br J Dermatol. 2016; 174: 287-295
        • Schmitt J.
        • Langan S.
        • Stamm T.
        • Williams H.C.
        Harmonizing Outcome Measurements in Eczema Delphi panel. Core outcome domains for controlled trials and clinical recordkeeping in eczema: international multiperspective Delphi consensus process.
        J Invest Dermatol. 2011; 131: 623-630
        • Sinha I.
        • Jones L.
        • Smyth R.L.
        • Williamson P.R.
        A systematic review of studies that aim to determine which outcomes to measure in clinical trials in children.
        PLoS Med. 2008; 5: e96
        • Williamson P.R.
        • Altman D.G.
        • Blazeby J.M.
        • Clarke M.
        • Devane D.
        • Gargon E.
        • et al.
        Developing core outcome sets for clinical trials: issues to consider.
        Trials. 2012; 13: 132
        • Williamson P.R.
        • Gamble C.
        Identification and impact of outcome selection bias in meta-analysis.
        Stat Med. 2005; 24: 1547-1561