My friend and colleague Scott‘s blog is at least partially concerned with the trials and tribulations of the tenure trail, so it occurs to me that I ought to at least write some about that myself. (You like how I just slipped that in? A lot of alliteration from anxious anchors placed in powerful posts!)
I got an email a couple of days ago from one of the top journals in information science, asking me to review a manuscript. Now, there’s no reward system in place for being a journal reviewer… I mean, being a reviewer for this journal won’t increase my chances of being published in said journal, and I sure won’t get credit on the tenure run for doing this. But I actually see it as my obligation as an academic, to do my part to contribute to the process that generates quality scholarship. So I said yes. Actually this is my second review for this particular journal, and only my second journal review ever. I was actually flattered when I was asked the first time; I took it as a sign that I’m gaining a reputation in the field, that the journal editor knew who I was and what my research area is enough to know what kind of manuscripts to send my way. Of course that was just the thin edge of the wedge… I’m on their radar now. I guess the reward for doing good work really is more work.
Now, a word about the journal reviewing process. An author submits a manuscript and the journal editor asks some reviewers to review the manuscript. The author is not told who the reviewers are, and in most cases the editor removes the author contact info from the manuscript so the reviewers don’t know who the author is either. Double-blind reviewing this is called. And it’s a good idea, too, for what I would think would be fairly obvious reasons: as an author, I get the reviewers’ comments returned to me when the editor lets me know if my paper has been accepted or rejected, and if I get a bad review, it could create bad blood among colleagues if I knew who wrote it. On the flip side, as a reviewer, knowing who the author is could influence my decision to accept or reject the paper even before I read it. In fact not only is double-blind reviewing prima facie a good idea, there’s actual empirical evidence that it’s a good idea: according to this study of double-blind versus single-blind reviewing, acceptance rates are lower and referees are more critical when the reviewer is unaware of the author’s identity.
So this journal asks me to be a reviewer, and not only is the author’s name still in the paper, the name is actually in the email asking me if I’m willing to be a reviewer! So if I, say, didn’t like the author, I could have refused to review the paper? I suppose so. But the whole point of writing this is that I do know the author, not just by reputation but personally. In fact the author is a friend; I’ve read his work many times and given him feedback on several papers pre-journal submission. I know that he does good work. So this is definitely one of those situations that double-blind reviewing was designed for: I pretty much decided immediately that unless this friend of mine wrote a real dog – which I consider highly unlikely – I’d recommend that the journal accept the manuscript. Of course I’d provide detailed feedback so that he could improve the paper before publication, but I’d do the same if he had sent me the paper before he’d submitted it to the journal.
Anyway, the upshot is that I’m disappointed in the whole journal reviewing process right now. I of course want to be published, so I suppose from that point of view having the reviewers know my name is in my favor. But I honestly believe that it’s a more fair process if it’s double-blind, plus being better for the scholarship. Would I recommend that the journal accept this paper even if I didn’t know who the author was? Probably; I’m sure it’s a great paper (I haven’t finished reading it yet). But am I biased in its favor now? You bet.
Update 1/22/05: I was talking to Scott about this post earlier this evening and he asked the very insightful question – which frankly I should have thought of myself – does the journal in question claim to use double-blind reviewing? Answer: no. I can’t find a single thing on the journal’s site about double-blind reviewing. Papers will be peer-reviewed, that’s all that’s promised.
I’ve run into similar situations where the promise of professional respect and anonymity was breached. It particularly baffles me, since the glitch does not reflect poorly on the author or the reviewer, but instead lowers my respect for the journal and its editorial staff.
That being said, academia is much less concerned with confidentiality of its faculty members compared to more sensitive topics such as corporate competitive intelligence. The stakes are lower, since the research is done, the paper is written, and no one will scoop the author on the research. I think the journals recognize the unlikelihood of major bias (more so than exists now, anyway). They also recognize that many faculty members are overwhelmed with their various responsibilities to their students, their research, their university, and their colleagues, and mistakes like this can easily happen.
Was it handled well? Of course not, but there is no penalty system in place. When you ask for volunteers, you get what you pay for. This is something academia is never going to learn, and they’re hurting themselves by taking the naive view that people perform tasks for the good of an external entity. This is only one example of many.