Advertisement
Advertisement
Advertisement
Peer review teeters as experts struggle with burden of work
21 May 2009
System threatened because academics lack time to assess research grant proposals. Zoe Corbyn writes
The peer-review system for allocating research grants is on the brink of breakdown, it was warned this week, as research councils struggle to find enough suitable academics willing to review applications.
Research councils say they have to approach many more scholars than they did in the past to find the number and calibre of reviewers needed because the experts are saying they are too busy to undertake the work.
The research councils' grant system relies on experts agreeing to take time away from their own research and teaching, usually without payment, to review research proposals in return for others doing the same.
But evidence gathered by Times Higher Education reveals serious difficulties.
"We are teetering on the brink of the system breaking down," said Nigel Brown, former director of science and technology at the Biotechnology and Biological Sciences Research Council and now vice-principal of the University of Edinburgh. "It is not impossible, but it is getting quite difficult. People are under a lot of pressure in their day jobs, and it is just easier to say 'no'."
The rise in the number of grant applications has been met with a fall in the number of academics willing to participate in the system.
Both the BBSRC and the Engineering and Physical Sciences Research Council report that they have experienced a rise in the number of academics refusing to take part in peer review. Similar reports have come from the Medical Research Council.
This is despite the fact that most research councils have some form of "peer-review college" to provide a ready pool of experts.
"The system is creaking under the demand people are putting on it," said Alf Game, the BBSRC's deputy director of research, innovation and skills.
Dr Game said a BBSRC proposal needed at least two referees - ideally three - before it went before a panel. He said that whereas a few years ago this was achieved by approaching four people, it now took six or seven.
"It is not unknown to go to 12 people to get two responses," he said. "Very occasionally, we can't even get two."
The EPSRC said that its success rate in getting academics to review work was about 50 per cent - a drop from the 65 or 70 per cent it enjoyed three years ago.
It also said that the speed of responses to peer-review requests was a problem even though there was a monetary incentive for undertaking and returning peer reviews on time.
The BBSRC also reported that it was facing a problem with the quality of reviews it received.
"We simply can't get reviewers to be critical enough to give proposals enough differentiation," Dr Game said.
He said the research council may move in future to introduce sanctions for academics who are in receipt of BBSRC grants but decline to take part in peer review. "Sanctions are quite likely if it continues to become more difficult."
Not every council seems to be struggling in the same way. The peer-review college introduced in 2003 by the Natural Environment Research Council to improve falling response rates delivers an almost 100 per cent hit rate, said Anne McFarlane, its research funding manager.
However, Nerc's system is straining under other problems. A review published earlier this year showed that Nerc researchers were not confident that members of the peer-review college were qualified enough to identify the highest-quality proposals. Biases in the system were also identified.
Figures compiled last year by Times Higher Education showed a 13 per cent rise in the number of funding applications to all research councils in the 2007-08 financial year compared with the year before. Last year, the EPSRC received 4,758 applications for 1,442 awards. The BBSRC got 1,983 applications for 581 awards.
It can be hard to find reviewers for large grants to multiple participants because everyone who could examine the proposal is involved in it, Professor Brown said.
zoe.corbyn@tsleducation.com
How two academics deal with the deluge of requests
Research councils may bemoan a lack of peer reviewers, but overworked academics struggle to balance their "community service" duties with a growing burden of teaching, research and administration tasks.
One is Claudio Stern, a professor of anatomy and developmental biology at University College London who is also an editor of the journal Mechanism of Development.
"The whole system is suffering as a result of overstretching," he said. "I try not to say 'no', but I can't accept everything. It depends on when the requests come ... there are times when I get dozens a day."
There is no alternative to peer review, he said. He would like to see academics better rewarded, but he is aware that such a move would mean cuts to research.
One of his big complaints about the system is that commercial publishers are making huge profits by using academics "free for everything". Another is that some scientists are not as committed to the system as others. "It is rather unfair," he said.
More than ten years ago, when he received a bundle of 12 grant proposals from a research council for a single funding round, John Gray made a stand.
Already overburdened, the head of the department of plant science at the University of Cambridge refused the requests and instituted a new personal rule: if the Biotechnology and Biological Sciences Research Council wanted his help, it would simply have to pick up the phone and ask him. Should it continue to send out proposals electronically, he would do them if he could; but if he could not, he would not, and the BBSRC would not get a reply.
"They refused point blank and that is how it has been ever since," he said. "I guess I have reviewed only two or three applications since."
He believes that the BBSRC has lost its personal touch. "We just get things fired off to us and are expected to drop everything and do them. They say they have not got time to telephone, but I suspect that if they phoned reviewers first, they would get a better response ... it would save them a lot of hassle."
Readers' comments
A simple rule of thumb. A typical proposal gets 5 reviews. Therefore for every proposal that I submit I should be reviewing 5. If I don't I am parasitizing the system. Thus I should be reviewing 10 per year. People who refuse to review should be denied the right to apply for funding. You can erect similar rules for peer review of publications, external examining etc.
The problem with sanctions is that they would force people (especially those whose expertise is highly specialised) to review proposals they do not feel competent to assess. You might get more reviews, but that is not worth much if most of the reviewers tick the "low confidence" box.
I think the point made of those refusing to review should be altered to those consistantly refusing to review over the course of one year. If they never consider themselves competent perhaps the system is wrong as it is not targetting the correct people; perhaps the experts who have no time to reivew need to suggest a deputy peer reviewer who they think is competent.
As an editor of a journal I can also report we find it more and more difficult to get people to review papers. I guess everyone is stressed writing papers in order to survive, even though no one or few will read them, as well as busy writing grant applications most of which will not be funded,
It is simple academics shoud stop working for free and start charging professional rates. Why should I spend a day reviewing a paper (for free) when I can earn substantial money consulting?? Same with grant applications there should be professional rates of pay for reviewers and then you will get professional reviews. We have enough worrk - teaching admin and research to do and so any extra work should be paid for by the journals and the grant awarding bodies.
The system is of course being stressed by many other factors , including the proliferation of bureacractic procedures within Universities. As one example Universities have in general increased the number of personal chairs other the last 20 years so now they have to have different grades of Professors. This of course requires internal committees and external letters from other professors (usually at the senior end) to support the cases for promotion with from professorial band to another. Characteristically the requests come with tight deadlines. The end result is more unproductive work. Writing a few letters a year for colleagues is not much of course, but there are myriads of such new tasks have emerged, so collectively this means less time to teach, research and participate in perr review.
Two further considerations based on the decline in success rates for grants submitted to the major funding bodies: 1. Several colleagues stopped reviewing grants for a research council when they discovered that grants they had scored as "outstanding" were not funded. Recently a major charity cancelled two funding rounds without much notice. I am pretty sure that some grants will have been reviewed before this cancellation. Reviewers will feel that they wasted their effort. 2. People are less inclined to review for a funding body, if it keeps rejecting their applications. That is human nature.
"so now they have to have different grades of Professors. ..to support the cases for promotion .. from professorial band to another. " Is this common elsewhere? I've never heard of different grades of professor except for the US-type nomenclature which doesn't seem to be what Steve is referring to.
Two further considerations based on the decline in success rates for grants submitted to the major funding bodies: 1. Several colleagues stopped reviewing grants for a research council when they discovered that grants they had scored as "outstanding" were not funded. Recently a major charity cancelled two funding rounds without much notice. I am pretty sure that some grants will have been reviewed before this cancellation. Reviewers will feel that they wasted their effort. 2. People are less inclined to review for a funding body, if it keeps rejecting their applications. That is human nature.
Interview committees focus on publications that we have written. Seldom do they bother to ask 'what publications have you studied recently'? The ramifications of RAE exercises and the like are focused on what has been written, rather than what has been assimilated. As for the question of reviewing - in my experience few people, if any, are interested in this vitally important area of work. I find that when I encounter a sound paper it becomes the focus of my attention for days if not weeks - I read and re-read. My point is that we are moving towards a situation when more and more people are talking but fewer and fewer people have time to listen. Reviewing is a casualty of this, and I am aghast as a reviewer when I am asked the question (paraphrasing) "Have you worked extensively through the mathematics of this publication?" My response is that I have endeavoured to do so,but often time constraints force me to assume accuracy. We are I believe failing to appreciate the time that is needed to properly understand the views expressed by our peers. The bean counters are driving the process and as, for example, scientists, we all too frequently defer to the dictates of the bean counters. .
Reviewers of research grants should be paid. The costs of this should be born by the applicant in the form a of a fee charged for the submission of the grant application (£1000 would cover it). The need to pay this fee would encourage applicants and their institutions to think more carefully before submitting proposals and serve to both improve the quality and reduce the volume of applications. Much of this latter is driven by the repeated revision and resubmission of mediocre proposals.
Being invited to review a proposal on behalf of colleagues in the Arts and Humanities is a privilege. It promotes a greater sense of understanding across disciplines than the repeated focus on Department or subject-disciplinary requirements. It is work undertaken on behalf of one's colleagues and should be undertaken for that reason alone. A fee paid by applicants and to reviewers is a dreadful idea and anyway would never compensate for the time allocated to both activities.
I still don't understand whether peers are paid for their reviews or not. And if they are, how much do they get per review, or is it some sort of mutual assistance where everyone check what the other and get his pay back
Peers only get paid in the House of Lords, and by a few publishers.
There is a good alternative called aceplace. Since peer review is biased towards manual viewing of publication lists, an automatic system work just as well.