Inside Higher Ed: Kill peer review or reform it?

“Blind peer review is dead. It just doesn’t know it yet.” That’s the way Aaron J. Barlow, an associate professor of English at the College of Technology of the City University of New York, summed up his views on the future of the traditional way of deciding whose work gets published in the humanities.

Professor Barlow did not dispute that most of the top journals in the humanities continue to select papers this way. But speaking in Seattle at a session of the annual meeting of the Modern Language Association, he argued that technology has so changed the ability of scholars to share their findings that it is only a matter of time before people rise up against the conventions of traditional journal publishing.

While others on the panel and in the audience argued for a reformed peer review as preferable to Professor Barlow’s vision of smashing the enterprise, and some questioned the practicality of simply walking away from peer review immediately, the idea that the system needs radical change was not challenged. Professor Barlow said that the system might have been justified once when old-style publishing put a significant limit on the quantity of scholarship that could be shared. But in a new era, he said, the justifications were gone. (Reflecting the new technology era, Barlow and one other panellist spoke via Skype to an audience that included two tables and wireless internet access to allow bloggers, Twitter users and journalists to write about the proceedings as they were taking place.)

To many knowing nods in the room, Professor Barlow argued that the traditional system of blind peer review – in which submissions are sent off to reviewers, whose judgements then determine whether papers are accepted, with no direct communication with authors – had serious problems with fairness. He said that the system rewards “conformity” and allows for considerable bias.

He described a recent experience in which he was recruited by “a prestigious venue” to review a paper that related in some ways to research he had done. Professor Barlow’s work was not mentioned anywhere in the piece, and he realised that the journal editor figured that he would be annoyed by the omission. And although he was, Professor Barlow said he did not feel that assigning the piece to him to review was fair to the author. “It was a set-up. The editor didn’t want a positive review, so the burden of rejection was passed on to someone the author would not know.”

He refused to go along and declined to review the paper when he realised what was going on. This sort of “corruption” is common, he said.

Professor Barlow has a long publishing record, so his frustrations with the system cannot be chalked up to being unable to get his ideas out there. But he said that when one of his papers was rejected recently, he simply published it on his blog directly, where comments have come in from fans and foes of his work.

“I love the editorial process” when comments result in a piece becoming better, he said, and digital publishing allows this to happen easily. But traditional peer review simply delays publication and leaves decision-making “in the dark”. Peer review – in the sense that people will comment on work and a consensus may emerge that a given paper is important or not – does not need to take place prior to publication, he said.

“We don’t need the bottleneck or the corruption,” he said. The only reason blind peer review survives is that “we have made appearance in peer-reviewed journals the standard” for tenure and promotion decisions. That will change over time, he predicted, and then the traditional system will collapse.

Peer review plus

While Professor Barlow noted the ability of digital publishing to bypass peer review, the idea of an intense, collaborative process for selecting pieces and improving them came at the session from the editor of Kairos, an online journal on rhetoric and technology that publishes work prepared for the web. Kairos has become an influential journal, but Cheryl Ball, the editor and an associate professor of English at Illinois State University, discussed how frustrating it was that people assume that an online journal must not have peer review. “Ignorance about digital scholarship” means that she must constantly explain the journal, she said.

Kairos uses a three-stage review process. First, editors decide if a submission makes sense for a review. Then, the entire editorial board discusses the submission (online) for two weeks before reaching a consensus that is communicated to the author with detailed letters from the board. (Board members’ identities are public, so there is no secrecy about who reviews pieces.) Then, if appropriate, someone is assigned to work with the author to coach him or her on how to improve the piece before publication.

As Professor Ball described the process, thousands of words are written about submissions, and lengthy discussions take place – all to figure out the best content for the journal. But there are no secret reviewers, and the coaching process allows for a collaborative effort to prepare a final version, not someone guessing about how to handle a “revise and resubmit” letter.

The process is quite detailed, but also allows for individual consideration of editorial board members’ concerns and of authors’ approaches, Professor Ball said. “Peer reviewers don’t need rubrics. They need good ways to communicate,” she said. Along those lines, Kairos is updating its tools for editorial board consideration of pieces to allow for synchronous chat, the use of electronic “sticky notes” and other ways to help authors not only with words, but also with digital graphics and illustrations.

Learning from law reviews

Allen Mendenhall, a PhD student at Auburn University who is also a blogger and a lawyer, suggested that humanities journals could take some lessons from law reviews. Mr Mendenhall is well aware of (and agrees with) many criticisms of law reviews, and in particular of the reliance for decisions on law students who may not know much about the areas of scholarship they are evaluating.

But he offered law reviews as an example of how a new web service could challenge the traditional ways of doing things. Many law reviews now use ExpressO to allow authors to submit a paper to multiple law journals at the same time. Once a journal accepts a piece, the author has a set time to reply – and during that time can notify other law reviews that participate of the chance to accept the piece on an expedited basis, in which case the author will place the piece there.

“The author is rushing journals the way college students rush a fraternity or sorority,” he said.

Obviously this system deviates in all kinds of ways from the norms of humanities scholarship, Mr Mendenhall said, in that most journals expect to be the only place considering a piece. But he argued that this system forces journals to stop sitting on pieces. “Everyone is competing, and that speeds up the publication process,” he said.

Recently, Mr Mendenhall had four pieces published in journals – one through ExpressO and three through the traditional peer-review system. The traditionally vetted pieces appeared seven months, nine months and two years after he submitted the articles. The ExpressO article appeared two and a half months after he submitted it.

A speedier process, he said, helps scholarship by getting ideas out there. But it also helps junior faculty members – and that is a legitimate reason to consider changes, he said. “Why should we wait months or years for a response?” he said. “Speed can help untenured professors add to their CVs and build a reputation. It’s more power for authors.”