 |
|
Publication Bias in Editorial Decision Making
Carin M. Olson, MD;
Drummond Rennie, MD;
Deborah Cook, MD,MSc,FRCPC;
Kay Dickersin, PhD;
Annette Flanagin, RN,MA;
Joseph W. Hogan, ScD;
Qi Zhu, MS;
Jennifer Reiling;
Brian Pace, MA
JAMA. 2002;287:2825-2828.
ABSTRACT
| |
Context Studies with positive results are more likely to be published than studies with negative results (publication bias). One reason this occurs is that authors are less likely to submit manuscripts reporting negative results to journals. There is no evidence that publication bias occurs once manuscripts have been submitted to a medical journal. We assessed whether submitted manuscripts that report results of controlled trials are more likely to be published if they report positive results.
Methods Prospective cohort study of manuscripts submitted to JAMA from February 1996 through August 1999. We classified results as positive if there was a statistically significant difference (P<.05) reported for the primary outcome. Study characteristics and indicators for quality were also appraised. We included manuscripts that reported prospective studies in which participants were assigned to an intervention or comparison group and statistical tests compared differences between groups.
Results Among 745 manuscripts, 133 (17.9%) were published: 78 (20.4%) of 383 with positive results, 51 (15.0%) of 341 with negative results, and 4 (19.0%) of 21 with unclear results. The crude relative risk for publication of studies with positive results compared with negative results was 1.36 (95% confidence interval [CI], 0.99-1.88). After being adjusted simultaneously for study characteristics and quality indicators, the odds ratio for publishing studies with positive results was 1.30 (95% CI, 0.87-1.96).
Conclusions Among submitted manuscripts, we did not find a statistically significant difference in publication rates between those with positive vs negative results.
INTRODUCTION
Publication bias refers to the greater likelihood that studies with positive results will be published.1-3 Publication bias has been demonstrated in several cohort studies that followed up protocols approved by research ethics committees,2, 4-5 ongoing trials funded by the National Institutes of Health,6 medical doctoral dissertations,7 and abstracts presented at scientific meetings.1, 8
Although such studies show that researchers are more likely to submit reports with positive results, it is less clear whether journal editors are more likely to publish them. Researchers may assume that reports of research with negative results will be rejected,9 but few researchers associated with unpublished results actually submitted a manuscript.2, 4, 10 One case-control study11 examined 100 published and 100 rejected manuscripts submitted to either of 2 Spanish medical journals and found no evidence for publication bias. However, that study may have been too small to identify a meaningful association. We are unaware of any evidence that editors are more likely to publish studies with positive results.
METHODS
Hypothesis, Design, and Setting
Our objective was to examine whether publication bias operates in editorial decision making. We hypothesized that editors preferentially publish studies with positive results. We performed a prospective cohort study to learn which manuscripts submitted to JAMA were published.
About 4000 manuscripts are submitted to JAMA annually. Each is assigned to a reviewing editor. About half are rejected after internal review and half undergo external peer review. Although the reviewing editor may reject a manuscript at any point, manuscripts are accepted only by the editor-in-chief or designated deputy. Four JAMA editors participated in this investigation; none had ultimate responsibility for accepting manuscripts at JAMA.
Inclusion Criteria
Manuscripts were included in the cohort if they reported results of a study that was prospective, assigned participants to an intervention, had at least 1 comparison group, and used a statistical test to compare differences in outcomes between groups. Such studies were typically randomized. Eligibility for inclusion was assessed in 2 steps: an editor-investigator working in the JAMA office screened all manuscripts for adherence to the first 3 criteria. Copies of manuscripts were then sent to 2 of 3 editor-investigators, who verified that the manuscript met all 4 criteria.
Data Abstraction
Data were abstracted during the editorial process so that investigators were unaware of the publication status of the manuscript when they abstracted data. Data were abstracted independently by 2 of 3 investigators who completed data forms by consensus and forwarded them to other investigators for data entry and analysis. Other investigators independently extracted information on publication status from JAMA's database and sent it directly for data entry and analysis.
Definitions
Our primary outcome was publication in JAMA. The primary independent variable was whether study results were positive, as determined by applying previously published sequential steps.12 The primary predictor of publication was significance of results. We classified results as positive if they showed a statistically significant effect (P<.05; 95% confidence interval [CI] for difference excluding 0 or 95% CI for ratio excluding 1) on the primary outcome in that study and negative if they did not. If no primary outcome was stated or discernible, we classified results based on most outcomes. Results were unclear if they could not be classified as positive or negative, typical when many outcomes were reported and an equal number were positive and negative but none was primary.
We abstracted information on study characteristics examined by others for association with publication1-2,4-6,13-17 and on objective indicators of study quality and reporting transparency2, 13 (Table 1). We also recorded the clinical topic, sex and age of study participants, and the funding sources as reported in the manuscript. If an industry supplied only drugs or devices, we did not count it as having funded the study.
|
|
|
|
Table 1. Characteristics of Controlled Trials and Their Relationship to Publication
|
|
|
Ethics
Before our investigation, JAMA editors agreed to take part in studies relating to peer review and editorial decision making. We did not tell editors the details of this investigation or request informed consent for participation, since their awareness might have influenced their publication decisions.18 For several years, JAMA's Instructions to Authors have told authors that their work might be included in a study.19 We did not tell authors about this investigation, because the standard editorial process was unchanged and the confidentiality of the author-editor relationship was maintained. The University of Washington's Human Subjects Review Committee approved the protocol by waiver of consent, under the condition that investigators inform the other editors about the study after all publication decisions had been made.
Analysis
We estimated the number of manuscripts needed for this investigation, assuming that manuscripts with positive and negative results would be submitted in equal proportions and the overall publication rate would be 16%. Using a 2-sided test for significance of .05 and power of 0.80, 708 manuscripts would be required to detect a difference between 12% and 20% publication rates. We increased the sample size to 750 to allow for manuscripts with unclear significance of results.
Proportions of studies published were examined by significance of results and other variables. Associations between independent variables and publication were estimated with relative risks (RRs) and 95% CIs. The P values were not adjusted for multiple comparisons, and P<.05 was considered statistically significant. To adjust for several variables simultaneously, we used multiple logistic regression and calculated the odds ratio (OR) as the measure of association. Data were entered into an Access (version 2.0, Microsoft Corporation, Redmond, Wash) database and analyzed with SAS software (version 6.12, SAS Institute, Cary, NC).
Awareness of the Study
Since awareness of the investigation may have influenced the decision making of the 4 JAMA editors who were also investigators, we performed additional analyses that excluded their assigned manuscripts. Editors who were not investigators were asked whether they had been aware of the investigation. They rated their awareness with a continuous scale, with 0 indicating they were unaware that any investigation was in progress, 5 indicating they knew some investigation was in progress but were unaware that it was about publication bias, and 10 indicating they knew everything about the investigation, including its hypothesis.
RESULTS
From February 1996 through August 1999, 13 569 manuscripts were submitted to JAMA; 745 met all inclusion criteria. Manuscripts were distributed among 20 JAMA editors for primary responsibility. Among the 745 manuscripts, 383 (51.4%) had positive results, 341 (45.7%) had negative results, and 21 (2.8%) had unclear significance of results.
JAMA published 133 (17.9%) of these manuscripts: 78 (20.4%) of 383 with positive results, 51 (15.0%) of 341 with negative results, and 4 (19.0%) of 21 with unclear significance of results. After studies with an unclear significance of results were excluded, those with positive results were not significantly more likely to be published (unadjusted RR, 1.36; 95% CI, 0.99-1.88). The association of other factors with publication is shown in Table 1.
After all study characteristics and quality indicators were adjusted for, an OR of 1.30 (95% CI, 0.87-1.96) was achieved for publishing manuscripts with positive vs negative results (Table 2). Being multicenter trials, enrolling in the United States, and having a sample size calculation were significantly associated with publication (Table 2).
|
|
|
|
Table 2. Logistic Regression Analysis of Characteristics Associated With Publication
|
|
|
After manuscripts for which any of the investigators were editors were excluded, the adjusted OR for publishing manuscripts with positive vs negative results was 1.32 (95% CI, 0.84-2.07). Of the 16 JAMA editors who were not investigators, 14 responded to the questionnaire about their awareness of the study. The median score was 4 (range, 0-10) and the mean score was 3.7, indicating editors were generally aware some investigation was in progress but were unaware of its hypothesis.
COMMENT
Submitted manuscripts reporting results of controlled trials are not significantly more likely to be published if they report positive results.
Studies with indicators for higher quality were more likely to be published. In a case-control study11 of submitted manuscripts, higher study quality was associated with publication. In contrast, methodologic quality of abstracts, letters, short reports,13 and meta-analyses submitted to JAMA20 was not associated with publication of full results.
We also found that studies enrolling some participants in the United States had an increased adjusted OR of publication. Previous studies21-22 found an association between journals' nationality and the national origin of the reports they publish. However, those studies did not account for differences in submissions to journals by nationality. Another study15 found that manuscripts submitted to Gastroenterology, a US journal, were more likely to be published if authors were from the United States. However, that study did not adjust for quality of the submitted manuscripts.
In our investigation, a statistically significant difference (P<.05) in outcomes between groups was used to determine a positive study. Some definitions of positive require that the significant result favor the experimental intervention13 or be beneficial.1 Another definition is that a positive result will change current thinking or the standard of care.3, 23
Researchers may interpret our results as confirming editors' publication bias, but any such bias is small compared with that demonstrated repeatedly for researchers. We found an adjusted OR of 1.30 for publication of controlled trials with positive results. A meta-analysis10 of controlled trials identified at funding or approval by an ethics committee found an OR for publication of 5.96 (95% CI, 2.33-15.22). Although authors2, 9 and others24 have assumed that editors preferentially publish manuscripts with positive results, researchers are more likely to write and submit manuscripts for studies with positive results.2, 4, 10
Editors may interpret our results as showing no evidence for publication bias. The Declaration of Helsinki25 charges authors and publishers to make available negative and positive results of investigations. Registries of clinical trials26 and online journals may facilitate the reporting of trials with negative results.27 Editors must guard against basing publication on the significance of a study's results; they should also judge manuscripts on the clinical question addressed and quality of the research methods.28-29
Assessing the influence of reviewers on editors' decisions is difficult. In one study,30 reviewers were no more likely to recommend publication of manuscripts reporting positive results.
The strengths of our investigation include its large sample size, prospective design, consideration of consecutive manuscripts submitted to a large-circulation, high-impact, general medical journal, objective inclusion criteria, data abstraction by 2 independent investigators blinded to publication status, and analysis of confounding variables.
Our results may not be generalizable to manuscripts describing studies other than controlled trials. Publication bias may affect studies of various designs differently.2, 4, 11, 31 Our results apply specifically to the editorial process at JAMA, for a specific period and set of editors; our findings may not be generalizable to specialty journals or journals with fewer submissions, fewer editors, or lower circulation.32
We accrued more manuscripts than required under the assumptions of our sample-size calculation. The difference in publication rates (5.4%) was smaller than we hypothesized (8%), decreasing power to detect a difference between groups. In addition, adjusting for covariates decreased the power to detect publication bias.33 We used a conservative 2-sided test for significance, which allows for the possibility that studies with negative results are published at a higher rate than studies with positive results; using 1-sided tests for our analysis would have given statistically significant results.
AUTHOR INFORMATION
Funding/Support: There was no external funding for this study. Funds were available from the Division of Emergency Medicine, University of Washington, Seattle, and the Father Sean O'Sullivan Research Center at St Joseph's Hospital, Hamilton, Ontario.
Acknowledgment: We thank Ethel Pinkston for administrative support, Barbara Crawley, MS, for technical support, and Radley Olson, PhD, for translation of the article by Vogel and Windeler.
Corresponding Author and Reprints: Carin M. Olson, MD, University of Washington Medical Center, Box 356123, Seattle, WA 98195 (e-mail: colson{at}u.washington.edu).
Author Affiliations: JAMA, Chicago, Ill (Drs Olson, Rennie, and Cook, Mss Flanagin and Reiling, and Mr Pace); Department of Medicine, Division of Emergency Medicine, University of Washington, Seattle (Dr Olson); Institute for Health Policy Studies, University of California, San Francisco (Dr Rennie); Departments of Medicine, Clinical Epidemiology, and Biostatistics, McMaster University, Hamilton, Ontario (Dr Cook); and Department of Community Health (Drs Dickersin and Hogan and Ms Zhu) and Center for Statistical Sciences (Dr Hogan), Brown University, Providence, RI.
REFERENCES
| |
1. Callaham ML, Wears RL, Weber EJ, Barton C, Young G. Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting JAMA. 1998;280:254-257. [published correction appears in JAMA. 1998;280:1232].
FREE FULL TEXT
2. Easterbrook P, Berlin J, Gopalan R, Matthews D. Publication bias in clinical research. Lancet. 1991;337:867-872.
FULL TEXT
|
ISI
| PUBMED
3. Olson CM. Publication bias. Acad Emerg Med. 1994;1:207-209.
PUBMED
4. Dickersin K, Min Y, Meinert C. Factors influencing publication of research results. JAMA. 1992;267:374-378.
FREE FULL TEXT
5. Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ. 1997;315:640-645.
FREE FULL TEXT
6. Dickersin K, Min Y. NIH clinical trials and publication bias. Online J Curr Clin Trials. 1993;Doc No 50:[4967 words; 53 paragraphs].
7. Vogel U, Windeler J. Factors modifying frequency of publications of clinical research results exemplified by medical dissertations [in German]. Dtsch Med Wochenschr. 2000;125:110-113.
PUBMED
8. De Bellefeuille C, Morrison C, Tannock I. The fate of abstracts submitted to a cancer meeting: factors which influence presentation and subsequent publication. Ann Oncol. 1992;3:187-191.
FREE FULL TEXT
9. Dickersin K, Chan S, Chalmers TC, et al. Publication bias and clinical trials. Control Clin Trials. 1987;8:343-353.
FULL TEXT
|
ISI
| PUBMED
10. Dickersin K, Min YI. Publication bias: the problem that won't go away. Ann N Y Acad Sci. 1993;703:135-146.
ISI
| PUBMED
11. Campillo C. Publication bias in two Spanish medical journals. Presented at: The International Congress on Biomedical Peer Review and Global Communications; September 19, 1997; Prague, Czech Republic.
12. Moher D, Dulberg C, Wells G. Statistical power, sample size, and their reporting in randomized controlled trials. JAMA. 1994;272:122-124.
FREE FULL TEXT
13. Chalmers I, Adams M, Dickersin K, et al. A cohort study of summary reports of controlled trials. JAMA. 1990;263:1401-1405.
FREE FULL TEXT
14. Gilbert J, Williams E, Lundberg G. Is there gender bias in JAMA's peer review process? JAMA. 1994;272:139-142.
FREE FULL TEXT
15. Link AM. US and non-US submissions: an analysis of reviewer bias. JAMA. 1998;280:246-247.
FREE FULL TEXT
16. Misakian AL, Bero LA. Publication bias and research on passive smoking. JAMA. 1998;280:250-253.
FREE FULL TEXT
17. Scherer R, Dickersin K, Langenberg P. Full publication of results initially reported in abstracts: a meta-analysis. JAMA. 1994;272:158-162.
FREE FULL TEXT
18. Feinstein AR. Construction, consent, and condemnation in research on peer review. J Clin Epidemiol. 1991;44:339-341.
FULL TEXT
|
ISI
| PUBMED
19. JAMA instructions for authors. JAMA. 1995;274:91.
FREE FULL TEXT
20. Stroup DF, Thacker SB, Olson CM, et al. Characteristics of meta-analyses related to acceptance for publication in a medical journal. J Clin Epidemiol. 2001;54:655-660.
FULL TEXT
|
ISI
| PUBMED
21. Ernst E, Kienbacher T. Chauvinism. Nature. 1991;352:560.
PUBMED
22. Braun T, Glanzel W, Schubert A. National publication patterns and citation impact in the multidisciplinary journals Nature and Science. Scientometrics. 1989;17:11-14.
23. Wilson A. Meta-analysis, part 2: assessing the quality of published meta-analyses. Med J Aust. 1992;156:173-187.
ISI
| PUBMED
24. Altman LK. Experts see bias in drug data. New York Times. April 19, 1997:B7, B12.
25. World Medical Association. 2000. Declaration of Helsinki. Available at: http://www.wma.net/e/policy/17-c_e.html. Accessibility verified March 26, 2002.
26. Smith R, Roberts I. An amnesty for unpublished trials. BMJ. 1997;315:622.
FREE FULL TEXT
27. Song F, Eastwood A, Gilbody S, Duley L. The role of electronic journals in reducing publication bias. Med Inform Internet Med. 1999;24:223-229.
FULL TEXT
|
ISI
| PUBMED
28. Angell M. Negative studies. N Engl J Med. 1989;321:464-466.
ISI
| PUBMED
29. Iverson C, Flanagin A, Fontanarosa PB, et al. American Medical Association Manual of Style. 9th ed. Baltimore, Md: Williams & Wilkins; 1998.
30. Abbot NC, Ernst E. Publication bias: direction of outcome less important than scientific quality. Perfusion. 1998;11:182-184.
31. Petticrew M, Gilbody S, Song F. Lost information? the fate of papers presented at the 40th Society for Social Medicine Conference. J Epidemiol Community Health. 1999;53:442-443.
ISI
| PUBMED
32. Ioannidis JP, Cappelleri JC, Sacks HS, Lau J. The relationship between study design, results, and reporting of randomized clinical trials of HIV infection. Control Clin Trials. 1997;18:431-444.
FULL TEXT
|
ISI
| PUBMED
33. Agresti A. An Introduction to Categorical Data Analysis. New York, NY: Wiley & Sons; 1996.
 CiteULike  Connotea  Del.icio.us  Digg  Reddit  Technorati  Twitter
What's this?
RELATED LETTERS
Idiopathic vs Hereditary Pancreatitis
F. Ulrich Weiss, Peter Simon, Carsten Hohoff, Klaus Peter Zimmer, Wolfram Domschke, and Markus M. Lerch
JAMA. 2003;289(8):985.
EXTRACT
| FULL TEXT
Registering Clinical Trials—Reply
Kay Dickersin and Drummond Rennie
JAMA. 2003;290(19):2546.
EXTRACT
| FULL TEXT
RELATED ARTICLE
June 5, 2002
JAMA. 2002;287(21):2875-2876.
EXTRACT
| FULL TEXT
THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES
|
Outcome Reporting in Industry-Sponsored Trials of Gabapentin for Off-Label Use
Vedula et al.
NEJM 2009;361:1963-1971.
ABSTRACT
| FULL TEXT
Important Science--It's All about the SPIN
Casadevall and Fang
Infect. Immun. 2009;77:4177-4180.
ABSTRACT
| FULL TEXT
Editorial Policies and Publication Bias: The Importance of Negative Studies
Sridharan and Greenland
Arch Intern Med 2009;169:1022-1023.
FULL TEXT
Nonscientific Factors Associated with Acceptance for Publication in The Journal of Bone and Joint Surgery (American Volume)
Okike et al.
JBJS 2008;90:2432-2437.
ABSTRACT
| FULL TEXT
Publication bias in qualitative research: what becomes of qualitative research presented at conferences?
Petticrew et al.
J. Epidemiol. Community Health 2008;62:552-554.
ABSTRACT
| FULL TEXT
Publication Bias in Orthopaedic Research: An Analysis of Scientific Factors Associated with Publication in The Journal of Bone and Joint Surgery (American Volume)
Okike et al.
JBJS 2008;90:595-601.
ABSTRACT
| FULL TEXT
Ethical Considerations in the Interpretation and Communication of Clinical Trial Results
Coultas
Proc Am Thorac Soc 2007;4:194-199.
ABSTRACT
| FULL TEXT
A Viewpoint on Evidence-based Health Informatics, Based on a Pilot Survey on Evaluation Studies in Health Care Informatics
Ammenwerth and de Keizer
J. Am. Med. Inform. Assoc. 2007;14:368-371.
ABSTRACT
| FULL TEXT
Statistical and Clinical Interpretation of Research Results
Allison
J. Am. Podiatr. Med. Assoc. 2007;97:165-170.
ABSTRACT
| FULL TEXT
Peer-reviewed publication of clinical trials completed for pediatric exclusivity.
Benjamin et al.
JAMA 2006;296:1266-1273.
ABSTRACT
| FULL TEXT
Posttraumatic Stress in Youth Experiencing Illnesses and Injuries: An Exploratory Meta-Analysis
Kahana et al.
Traumatology 2006;12:148-161.
ABSTRACT
Effect of Blinded Peer Review on Abstract Acceptance
Ross et al.
JAMA 2006;295:1675-1680.
ABSTRACT
| FULL TEXT
Bias in published cost effectiveness studies: systematic review
Bell et al.
BMJ 2006;332:699-703.
ABSTRACT
| FULL TEXT
Bias in Clinical Intervention Research
Gluud
Am J Epidemiol 2006;163:493-501.
ABSTRACT
| FULL TEXT
Sufficiency and Stability of Evidence for Public Health Interventions Using Cumulative Meta-Analysis
Muellerleile and Mullen
AJPH 2006;96:515-522.
ABSTRACT
| FULL TEXT
Peer Review and Professionalism at the Archives of Internal Medicine
Emanuel and Greenland
Arch Intern Med 2005;165:2559-2560.
FULL TEXT
The Accuracy of Diagnostic Imaging for the Assessment of Chronic Osteomyelitis: A Systematic Review and Meta-Analysis
Termaat et al.
JBJS 2005;87:2464-2471.
ABSTRACT
| FULL TEXT
Negative Results and Impact Factor: A Lesson From Neonatology
Littner et al.
Arch Pediatr Adolesc Med 2005;159:1036-1037.
ABSTRACT
| FULL TEXT
Recent Developments in Health Law
J Law Med Ethics 2005;33:384-403.
Acute Endophthalmitis Following Cataract Surgery: A Systematic Review of the Literature
Taban et al.
Arch Ophthalmol 2005;123:613-620.
ABSTRACT
| FULL TEXT
Interaction Between an 11{beta}HSD1 Gene Variant and Birth Era Modifies the Risk of Hypertension in Pima Indians
Franks et al.
Hypertension 2004;44:681-688.
ABSTRACT
| FULL TEXT
Factors Influencing the Publication of Randomized Controlled Trials in Child Health Research
Hartling et al.
Arch Pediatr Adolesc Med 2004;158:983-987.
ABSTRACT
| FULL TEXT
High-Dose Statins in Acute Coronary Syndromes: Not Just Lipid Levels
Nissen
JAMA 2004;292:1365-1367.
FULL TEXT
A Low, Rather than a High, Total Plasma Homocysteine Is an Indicator of Poor Outcome in Hemodialysis Patients
Kalantar-Zadeh et al.
J. Am. Soc. Nephrol. 2004;15:442-453.
ABSTRACT
| FULL TEXT
Registering Clinical Trials--Reply
Dickersin and Rennie
JAMA 2003;290:2546-2546.
FULL TEXT
Idiopathic vs Hereditary Pancreatitis
Weiss et al.
JAMA 2003;289:985-985.
FULL TEXT
Association Between Time Interval to Publication and Statistical Significance
Dickersin et al.
JAMA 2002;287:2829-2831.
ABSTRACT
| FULL TEXT
|
