Advertisement
Advertisement
Advertisement
RAE reviewers accused of going over the top on scores
1 January 2009
Concerns about 'grade inflation' as panels 'put too many people in the 4* box'. Zoe Corbyn reports
Concerns have been raised that reviewers for the research assessment exercise have been "too generous" in awarding the top grade.
The results from the 2008 RAE, published on 18 December, said that 17 per cent of all research submitted by more than 50,000 researchers was "world leading" (4*). Some 150 of the 159 institutions included in the exercise were shown to have at least 5 per cent of world-class activity in at least one department.
Initial celebration of the results as a clear sign of the UK's international research excellence has been replaced by complaints about "grade inflation" and warnings that this could affect the distribution of funding.
Observers have expressed concerns about what they say are the "very high" percentages of 4* work in certain subject areas. Some peer-review panels in arts-based subjects, as well as in business and economics, seem to have been more generous in awarding the top grade than those marking sciences or languages.
In communication, culture and media studies, for example, the highest-scoring department saw 65 per cent of its research activity graded 4*. Dance and music have similarly high figures.
However, the top-scoring physics department saw just 25 per cent of its research graded 4*.
Top departments in bioscience-related subjects, where the UK is considered to excel, hover between 30 and 40 per cent for 4* research activity. In languages, the departments with the highest percentages of 4* research activity are grouped in a similar bracket.
"It would have been better if all the subjects had used the top category (4*) less," said Andrew Oswald, professor of economics at the University of Warwick. "The whole point of the categorisation from 1* to 4* was to try to separate out work that was on the edge of winning Nobel prizes, and to get people to understand the difference between good work and absolutely breathtaking, iconoclastic work. I think that a general problem with the RAE is that all panels have put too many people into that 4* box."
Eric Thomas, vice-chancellor of the University of Bristol, said that although the 17 per cent figure for 4* overall was "probably" about right, he was concerned that several panels had been too generous in their interpretation of 4*.
"Some results show more than 50 per cent 4* in some institutions in particular units of assessment," he said. "The grade 4* was deliberately described as world-leading research which I would have thought of as a minority, not a majority," he said.
Departments were graded by 67 subject subpanels moderated by 15 main panels, which also met to ensure consistency in how the grades (of 4* for world-leading research through to 1* for nationally recognised research) were interpreted.
The Higher Education Funding Council for England, which ran the RAE, has said that the results across panels are supposed to be "broadly comparable".
But one panel member, who asked not to be named, agreed that it had not been "too hard" to get a 4*.
"Is it the Nobel prizewinning paper or something that is much more common but nevertheless drives the field forward? ... In the end we decided that it was something that happened to good people every year," he said.
• The RAE's exposure of pockets of research excellence across a range of teaching-intensive institutions could lead to extensive poaching of staff by the more powerful research institutions, the sector was warned this week.
This RAE provided, for the first time, a "research profile" for each department rather than a single summative judgment. The profiles highlighted pockets of excellence in departments that, overall, are not world class.
One senior university executive commented: "It is the other side of the exercise. You identify the very good universities so people want to be in them, but ... any identification of clever people means the market will (naturally) look to them. When we are looking to hire at whatever level, we want someone good so we look around to see where they are."
zoe.corbyn@tsleducation.com.
Readers' comments
A good point - how would one recognise international excellence in such a subjective and self-serving field as "media studies"? The simplest solution is for HEFCE to withdraw all funding from such areas and concentrate on the STEM subjects on which our economic future depends.
I am an academic in a department of foreign languages. In my subject, about 20 universities have achieved 4*, which should show the "excellence" of UK universities within this field. However, hardly any British scholar, within my subject, is known in the rest of Europe or in the US. How can we be doing so much world-leading research if most of us are virtually unknown by the truly great and world-leading scholars in Europe and the US. Perhaps becasue British scholars focus on topics the rest of the world consider unimportant -- such as film studies, cultural studies, gender studies, and present-day novelists. when it comes to "difficult" topics, such as the Middle Ages, the Renaissance, Romanticism, Existentialism, very little is being done in the UK. Which brings to mind what one of my students said to me recently: "I am taking English and reading great literature, the Romantics, and then I you are making us read these bland 1990s novels".
I feel that the RAE is simply "an internal, pat on the back, self-congratulatory" exercise. In the end, it a waste of time and money and measures nothing. World-class research needs a world-wide measure/indicator either through journal publications, number of Nobel Prize winners, or at least the potential benefit to society, science, technology and engineering.
Anybody who has been on a grant-awarding body will be aware of the tendency of colleagues in some, relatively small or marginal, sub-fields to think that almost anything done in their sub-field by anybody, anywhere, must be terrific. I guess it must make them feel better, and reinforces their sense that what they do matters. One is not obliged to accept these self-evaluations. Presumably HEFCE is capable of 'aiming off' to allow for the evident tendency of some panels to have been more generous to their peers than others may have been? Otherwise, panels' behaviour will have directed a larger share of a static (in real terms, probably declining) amount of QR money to subject areas with a looser interpretation of international quality (whatever that is -- my personal interpretation of these criteria is just Very Good, OK, Not Much Good, and No B***** Good -- 17% VG sounds reasonable. And why should anybody within the HE sector object to, or be surprised by, grade inflation in research ratings, when we contemplate what has happened to the degree and other standards that we control?). The answer is surely for HEFCE simply to make sure that this doesn't happen. Whatever algorithm it ends up employing, one crucial ingredient is the basic 'quantum' for a subject area -- the sum of money flowing towards, say, 1 FTE of 3* work. So the effect of, for example, the Culture & Media Studies Panel's generosity would not be that CMS got more public funding -- simply that a 4* wasn't worth as much, in cash, as it might have been if they'd been as discriminating as the Physicists. There's also a simple answer for league-table compilers (because RAE grades are only valuable in two currencies -- QR money, and league-table rankings). However the various panels may have applied what were (supposedly) common criteria (how could they have been common criteria, given their lack of objectivity or verifiability, or indeed standardization?), we end up with a list graded from best to worst, so all that matters, in reputational terms, is where you end up, relative to all the rest.
Alex's solution neatly encapsulates the emerging role of universities in bolstering a failing market economy. (A good point - how would one recognise international excellence in such a subjective and self-serving field as "media studies"? The simplest solution is for HEFCE to withdraw all funding from such areas and concentrate on the STEM subjects on which our economic future depends.) So that academia could be fully market-serving, I presume? Come to that why not close down all departments not fixated on industry?
its funny this only comes from a professor at Warwick" and one at "bristol" maybe they are going to feel the pinch..credit crunchy 2008 heading towards 2009. so called light weights climbing up the RAE rankings.in my opinion this is just the old boys club paranoid about funding. like with any other evaluation about grading or ranking there will be some criticism..but the RAE truly does reflect the fact too much money has been going to the wrong places and some places deserve more funding.
RAE grade inflation has less to do with the environment of so-called continuous improvement than with the conflict involved in being a panel member at all. Individual members may be nominated by subject associations; chairs and vice-chairs have in the past been answerable to the same bodies in open feedback sessions. There is no blame attached to the people concerned, but asking them to be both completely objective and to represent the interests of a subject community is clearly problematic. It's another example of the way an exercise designed to identify international excellence started from a parochial set of assumptions.
Who cares if grades were inflated? The RAE is about allocating money to British institutions. It has absolutely nothing to do with establishing the international strength of the UK's research. Tomorrow China may conduct their version of RAE and "prove" that 80% of all their research is "world leading" (maybe they'll give their researchers 5*)... It's absurd that some people seem to believe that a UK study of UK institutions somehow can show anything about how these institutions are doing compared to institutions not included in the study.
Don't blame other countries for UK's incompetence in research. Whether other countries inflate their do it or not does not change the fact that the RAE is just a useless exercise.