Scientists - such an unpredictable bunch!
Not every good scientist is a good peer reviewer, too. That's why editors would like to be able to predict reviewer performance, before recruiting a new researcher for their journal. But what qualities in a scientist's CV raise an editor's hopes of a good review standard? Seniority? Academic rank? Training in critical appraisal? A new study shows that none of the criteria are of any use for picking the right person for the job! By Brynja Aadam-Radmanic.
(Feb 26th 2007) As most scientists regularly write papers together with co-authors having to define and check for quality in their own publications, a naive observer would expect all researchers to possess the basic skills needed for peer review, too. However, to the distress of journal editors, reading and writing papers seems to be quite a different story to scrutinising a colleague's piece of work. So how can editors find the right partners for peer review?
Editors need peer reviewers to help them decide if a submitted manuscript meets their journal's standards. They want scientists that know how novel and substantial a manuscript must be to be acceptable for publication in their periodical. They expect reviewers to check texts thoroughly within days for any flaws and weaknesses, which only an expert can detect. And, if necessary, the reviewer is expected to make suggestions, which will help improve the authors' work.
For a study published in PloS Medicine, Michael L. Callaham and John Tercier wanted to find out if there is any easily available information that could help editors to find good reviewers. They contacted all the regular reviewers from the
Annals of Emergency Medicine. A total of 306 of these experienced reviewers (71% of all those associated with the journal) completed a survey of past training and experiences, which might be expected to improve peer review skills. The reviewers had performed 2,856 reviews of 1,484 separate manuscripts over a four-year study period. During this time the journal's editors rated the quality of the reviews.
Surprisingly, most variables, including academic rank, formal training in critical appraisal or statistics, or status as principal investigator of a grant, failed to predict performance of higher-quality reviews in a multivariate analysis. The only significant predictors of quality were working in a university-operated hospital versus other teaching environments and relative youth (under ten years of experience after finishing training). Paradoxically, serving as a member of their Institutional Research Board (IRB) was associated with lower-quality scores.
However, even these findings don't help editors to select the best reviewers. The authors of the study state, "Most importantly, most of the odds ratios (ORs) were less than 2, and even when our model produced ORs of 2 or more and significant
p-values, the area under the curve was barely better than chance alone, demonstrating the lack of usefulness of these criteria in the real world."
This means, you never know what you're going to get! Editors have to continue on a trial and error basis as they have done to-date. There is no guarantee until someone has been tried and approved as a reviewer; even the best researcher can be a lousy reviewer!
As much as editors are alarmed by these results, one must be reminded of the positive aspects. Every scientist has an equal chance of being asked for a contribution to the peer review system; every researcher has an equal chance to demonstrate individual skills, to be bestowed the honour of reviewing for a prestigious journal and of receiving some extra work distracting one from one's own research...
Link:
Callaham ML, Tercier J (2007) The Relationship of Previous Training and Experience of Journal Peer Reviewers to Subsequent Review Quality. PLoS Med 4(1): e40
