This Article  • Full text  • PDF  • Correction  • Send to a friend  • Save in My Folder  • Save to citation manager  • Permissions  Citing Articles  • Citation map  • Citing articles on HighWire  • Citing articles on ISI (8)  • Contact me when this article is cited  Related Content  • Related letters  • Similar articles in JAMA  Topic Collections  • Evidence-Based Medicine  • Statistics and Research Methods  • Review  • Alert me on articles by topic

Peter C. Gøtzsche, MD, DrMedSci; Asbjørn Hróbjartsson, MD, PhD; Katja Mari, MSc; Britta Tendal, MSc

JAMA. 2007;298:430-437.

Context  Meta-analysis of trials that have used different continuous or rating scales to record outcomes of a similar nature requires sophisticated data handling and data transformation to a uniform scale, the standardized mean difference (SMD). It is not known how reliable such meta-analyses are.

Objective  To study whether SMDs in meta-analyses are accurate.

Data Sources  Systematic review of meta-analyses published in 2004 that reported a result as an SMD, with no language restrictions. Two trials were randomly selected from each meta-analysis. We attempted to replicate the results in each meta-analysis by independently calculating SMD using Hedges adjusted g.

Data Extraction  Our primary outcome was the proportion of meta-analyses for which our result differed from that of the authors by 0.1 or more, either for the point estimate or for its confidence interval, for at least 1 of the 2 selected trials. We chose 0.1 as cut point because many commonly used treatments have an effect of 0.1 to 0.5, compared with placebo.

Results  Of the 27 meta-analyses included in this study, we could not replicate the result for at least 1 of the 2 trials within 0.1 in 10 of the meta-analyses (37%), and in 4 cases, the discrepancy was 0.6 or more for the point estimate. Common problems were erroneous number of patients, means, standard deviations, and sign for the effect estimate. In total, 17 meta-analyses (63%) had errors for at least 1 of the 2 trials examined. For the 10 meta-analyses with errors of at least 0.1, we checked the data from all the trials and conducted our own meta-analysis, using the authors' methods. Seven of these 10 meta-analyses were erroneous (70%); 1 was subsequently retracted, and in 2 a significant difference disappeared or appeared.

Conclusions  The high proportion of meta-analyses based on SMDs that show errors indicates that although the statistical process is ostensibly simple, data extraction is particularly liable to errors that can negate or even reverse the findings of the study. This has implications for researchers and implies that all readers, including journal reviewers and policy makers, should approach such meta-analyses with caution.

Author Affiliations: Nordic Cochrane Centre, Rigshospitalet, Copenhagen, Denmark.

RELATED LETTERS

Data Discrepancies in Meta-analyses That Use Standarized Mean Differences
Hugh McGuire, Melissa Edmonds, and Jonathan Price
JAMA. 2007;298(19):2261-2262.
EXTRACT | FULL TEXT  

Data Discrepancies in Meta-analyses That Use Standarized Mean Differences—Reply
Peter C. Gøtzsche and Britta Tendal
JAMA. 2007;298(19):2262-2263.
EXTRACT | FULL TEXT  

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

Using Existing Systematic Reviews in Complex Systematic Reviews
Whitlock et al.
ANN INTERN MED 2008;148:776-782.
ABSTRACT | FULL TEXT  

Effect of mechanical ventilation in the prone position on clinical outcomes in patients with acute hypoxemic respiratory failure: a systematic review and meta-analysis
Sud et al.
CMAJ 2008;178:1153-1161.
ABSTRACT | FULL TEXT  

Quo Vadis, Perioperative Beta Blockade? Are You "POISE'd" on the Brink?
London
Anesth. Analg. 2008;106:1025-1030.
FULL TEXT  

Data Discrepancies in Meta-analyses That Use Standarized Mean Differences
McGuire et al.
JAMA 2007;298:2261-2262.
FULL TEXT