This Article  • Full text  • PDF  • Send to a friend  • Save in My Folder  • Save to citation manager  • Permissions  Citing Articles  • Citation map  • Citing articles on HighWire  • Citing articles on Web of Science (23)  • Contact me when this article is cited  Related Content  • Similar articles in JAMA  Social Bookmarking   What's this?

Two Randomized Trials

Michael L. Callaham, MD; Robert K. Knopp, MD; E. John Gallagher, MD

JAMA. 2002;287:2781-2783.

Context  Better peer review is needed, but proven methods to improve quality are unknown. Our objective was to determine whether written feedback to reviewers improves subsequent reviews.

Methods  Eligible reviewers were randomized to intervention or control (receiving other reviewers' unscored reviews and the editor's decision letter). Study 1 (September 1998–September 2000) included reviewers with a median quality score of 3 or lower; study 2 (April 2000–January 2002), reviewers with median score of 4 or lower. Study 1 was designed with a power of 0.80 to detect a difference in score of 1; study 2, with a power of 0.80 to detect a difference of 0.5. All reviewers were at a peer-reviewed journal (Annals of Emergency Medicine). The main outcome measure was the editor's routine quality rating (1-5) of all reviews (blinded to study enrollment).

Results  For study 1, 51 reviewers were eligible and randomized and 35 had sufficient data (182 reviews) for analysis. The mean individual reviewer rating change was 0.16 (95% confidence interval [CI], -0.26 to 0.58) for control and -0.13 (-0.49 to 0.23) for intervention. For study 2, 127 reviewers were eligible and randomized, and 95 had sufficient data (324 reviews). Controls had a mean individual rating change of 0.12 (95% CI, -0.20 to 0.26) and intervention reviewers, 0.06 (-0.19 to 0.31).

Conclusions  In study 1, minimal feedback from editors on review quality had no effect on subsequent performance of poor-quality reviewers, and the trend was toward a negative effect. In study 2, feedback to average reviewers was more extensive and supportive but produced no improvement in reviewer performance. Simple written feedback to reviewers seems to be an ineffective educational tool.

Author Affiliations: Department of Emergency Medicine, University of California, San Francisco (Dr Callaham); Department of Emergency Medicine, University of Minnesota, Minneapolis (Dr Knopp); Department of Emergency Medicine, Albert Einstein College of Medicine, Bronx, NY (Dr Gallagher).

CiteULike    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter     What's this?

THIS ARTICLE HAS BEEN CITED BY OTHER ARTICLES

Balancing Uncertain Risks and Benefits in Human Subjects Research
Barke
Science Technology Human Values 2009;34:337-364.
ABSTRACT  

Eliciting and using expert opinions about dropout bias in randomized controlled trials
White et al.
Clin Trials 2007;4:125-139.
ABSTRACT  

Why do peer reviewers decline to review? A survey
Tite and Schroter
J. Epidemiol. Community Health 2007;61:9-12.
ABSTRACT | FULL TEXT  

Effects of training on quality of peer review: randomised controlled trial
Schroter et al.
BMJ 2004;328:673.
ABSTRACT | FULL TEXT  

The Peer Review Process (aka Peer Reviewology)
Yucha
Biol Res Nurs 2002;4:71-72.