Do authors know who refereed their paper? A questionnaire survey

Methods and results

Psychological Medicine is a leading international academic journal of psychiatry. For a five month period all those who submitted a manuscript to the journal were asked if they could guess the identity of the referees assigned to their paper (usually two or three), drawn from the pool of 580 available to the editors. All authors were sent a simple form asking them to write down the presumed identity of each referee and to indicate their degree of certainty on a four point scale, ranging from very uncertain (1) to certain (4). Alternatively the author could say that he or she had no idea of each referee's identity. The single page questionnaire was sent at the same time as the author was given the final decision about acceptance or rejection of the manuscript. Proportions were compared using the χ² test without Yates's correction.

A total of 135 forms were sent out and 94 received back (70%). As expected,5 non-responders were more likely than responders to have had their paper rejected (44.0% (upsilon) 7.8%, χ² = 19.9, df = 1, P<0.001). The total number of referees' reports for the 94 papers for which we received responses was 252. Of these 252 referees 15 were correctly identified (5.9%), 36 were incorrectly identified (14.3%), and in 201 (79.7%) the author had no idea of the referee's identity. Nearly all papers were reviewed by more than one referee (usually three) In four instances the author indicated the correct referee but against the wrong report. In two instances there were reasons to believe this was because of a misreading of the reference number and that the identity had been correctly guessed. If all those who had identified a referee of their paper but for the wrong report were given the benefit of the doubt then the correct number of guesses rose to 19 (7.5%).

The mean level of certainty for those who correctly identified the referee was 2.5 (lying between uncertain and fairly certain), compared with 1.8 for inaccurate guesses (between very uncertain and uncertain) (t = 2.55, df = 46, P = 0.014).

Using authors rather than referees as the denominator we found that those who correctly identified one or more referee were more likely to have had their paper accepted (χ² = 4.61, df = 1, P = 0.03).

Comment

Anyone who has ever submitted a scientific paper will no doubt be familiar with the elaborate process of intuition and detection that goes into attempting to deduce the identity of the anonymous referee who has praised or damned the paper. This study suggests that even for a specialty journal such efforts are largely unrewarding and that most referees remain anonymous.