Peer-Reviewed Publications

(This is the continuation of a series of postings introduced on January 22, 2010.)

“Curmudge, the term ‘peer-reviewed’ has been in the news a lot during the past few months. This might be one of those extremely rare blog topics about which you know something. Would you be so kind as to enlighten me, or is this a subject—like differential equations—that you have totally forgotten?”

“I would be delighted to share my recollections, Jaded Julie. Because I had manuscripts peer reviewed between 1961 and 2002 and did a bit of reviewing myself, much of it has become hardwired.”

“Please give it a try, Old Guy. Let’s start by looking at the process from the author’s standpoint and how he or she decides what is publishable.”

“The critical guideline in this whole process, Julie, is integrity. If data from an experiment or series of experiments provides the basis for a finding or story heretofore unknown to the world or to others in your discipline, it is a candidate for publication. But one can’t cherry-pick his data. If some of the data don’t fit your story, they can’t simply be ignored; they can only be rejected if they are statistical outliers.”

“I can appreciate the importance of integrity. Doesn’t everyone have some element of vested interest in his research?”

“Vested interest comes in many flavors, Julie. Late in one’s career, the author’s delight in contributing to knowledge would represent the low end of the vested interest scale. At the far end of the scale is current work in atmospheric science. It seems that in that discipline one’s findings impact multimillion-dollar technological ventures, politics, and the fate of the industrialized world. The higher one goes on the vested interest scale, the greater the temptation to compromise one’s integrity.”

“So back to the guy in the lab. The author writes his paper, including data to support his hypothesis (remember, no fudging or cherry-picking) and enough experimental detail to permit someone to duplicate the work, and sends it off to a journal for publication, right?”

“Exactly. Often he must wait many months—up to a year—for the peer review process to transpire.”

“Am I correct, Curmudge, that publication in a peer-reviewed journal has traditionally been the ‘gold standard’ for scientific knowledge?”

“You are essentially correct, Jaded Julie. Remember, however, that the ultimate tests are (1) one’s work must be independently reproducible by someone else, and (2) any forecasts must turn out to be true. Nevertheless, despite all of its warts and blemishes and potential for hanky-panky, the peer review process has worked pretty well. Because my area was not highly controversial and was populated by a limited number of scientists, peer review was not very onerous for either authors or reviewers.”

“Curmudge, what’s that ‘potential for hanky-panky’ that you mentioned earlier?”

“Here’s an example. Consider a topic about which there are two schools of thought and in which editors and reviewers have a vested interest. Remember that peer review is a single-blind process in which the reviewers know the author but not vice-versa.”

“I think I can see what’s coming, Curmudge. If the editor’s position on the topic is ’A’ and the author is advocating position ‘B’, the editor can reject the paper out of hand or send it to reviewers that share position ‘A’ who will recommend rejection. I feel sorry for the poor author. He’s been shot down, but he doesn’t know by whom. And potential reviewers who are interested in position ‘B’ may wonder why they never receive any manuscripts to review. ”

“That’s it, Julie. If the author can’t find another journal whose reviewers will at least consider evidence of position ‘B’, he and position ‘B’ are effectively suppressed. This would appear to be ‘political’ science, i.e., science for the purpose of politics or vested interests. My speculation is that this might represent the untimely and unjustified demise of traditional peer review.”

“So how do we summarize the lessons of these last four postings?”

“It’s pretty simple, Julie. Be wary of politicians talking about science, of people confusing effect with cause, of scientists with a vested interest using complex mathematical models that can’t be experimentally verified, and of researchers who suppress publication of findings that don’t agree with their own.”