RAE reviewers accused of going over the top on scores

Concerns about 'grade inflation' as panels 'put too many people in the 4* box'. Zoe Corbyn reports

Concerns have been raised that reviewers for the research assessment exercise have been "too generous" in awarding the top grade.

The results from the 2008 RAE, published on 18 December, said that 17 per cent of all research submitted by more than 50,000 researchers was "world leading" (4*). Some 150 of the 159 institutions included in the exercise were shown to have at least 5 per cent of world-class activity in at least one department.

Initial celebration of the results as a clear sign of the UK's international research excellence has been replaced by complaints about "grade inflation" and warnings that this could affect the distribution of funding.

Observers have expressed concerns about what they say are the "very high" percentages of 4* work in certain subject areas. Some peer-review panels in arts-based subjects, as well as in business and economics, seem to have been more generous in awarding the top grade than those marking sciences or languages.

In communication, culture and media studies, for example, the highest-scoring department saw 65 per cent of its research activity graded 4*. Dance and music have similarly high figures.

However, the top-scoring physics department saw just 25 per cent of its research graded 4*.

Top departments in bioscience-related subjects, where the UK is considered to excel, hover between 30 and 40 per cent for 4* research activity. In languages, the departments with the highest percentages of 4* research activity are grouped in a similar bracket.

"It would have been better if all the subjects had used the top category (4*) less," said Andrew Oswald, professor of economics at the University of Warwick. "The whole point of the categorisation from 1* to 4* was to try to separate out work that was on the edge of winning Nobel prizes, and to get people to understand the difference between good work and absolutely breathtaking, iconoclastic work. I think that a general problem with the RAE is that all panels have put too many people into that 4* box."

Eric Thomas, vice-chancellor of the University of Bristol, said that although the 17 per cent figure for 4* overall was "probably" about right, he was concerned that several panels had been too generous in their interpretation of 4*.

"Some results show more than 50 per cent 4* in some institutions in particular units of assessment," he said. "The grade 4* was deliberately described as world-leading research which I would have thought of as a minority, not a majority," he said.

Departments were graded by 67 subject subpanels moderated by 15 main panels, which also met to ensure consistency in how the grades (of 4* for world-leading research through to 1* for nationally recognised research) were interpreted.

The Higher Education Funding Council for England, which ran the RAE, has said that the results across panels are supposed to be "broadly comparable".

But one panel member, who asked not to be named, agreed that it had not been "too hard" to get a 4*.

"Is it the Nobel prizewinning paper or something that is much more common but nevertheless drives the field forward? ... In the end we decided that it was something that happened to good people every year," he said.

• The RAE's exposure of pockets of research excellence across a range of teaching-intensive institutions could lead to extensive poaching of staff by the more powerful research institutions, the sector was warned this week.

This RAE provided, for the first time, a "research profile" for each department rather than a single summative judgment. The profiles highlighted pockets of excellence in departments that, overall, are not world class.

One senior university executive commented: "It is the other side of the exercise. You identify the very good universities so people want to be in them, but ... any identification of clever people means the market will (naturally) look to them. When we are looking to hire at whatever level, we want someone good so we look around to see where they are."

zoe.corbyn@tsleducation.com.

Already registered?

Sign in now if you are already registered or a current subscriber. Or subscribe for unrestricted access to our digital editions and iPad and iPhone app.

Register to continue  

You've enjoyed reading five THE articles this month. Register now to get five more, or subscribe for unrestricted access.

Most Commented

  • Man measuring bar graphs with tape measure

An Elsevier analysis explores the viability of a ‘smarter and cheaper’ model

  • David Willetts

The former universities minister discusses the reforms that reshaped higher education and his first steps into academia

  • Man holding a box filled with work-related items

Refusal by John Allen to obey instruction from manager at Queen Mary University of London led to his sacking, tribunal rules

  • A black and white crowd scene with a few people highlighted

What are the key issues local union branches are dealing with, and how do they manage relationships with institutions in what many activists argue is an increasingly confrontational environment?

  • Muslim woman at graduation ceremony, Barbican, London

Sector called on to embrace faith-related concerns in intellectual debates