Advertisement
Advertisement
Advertisement
It's evolution, not revolution for REF
24 September 2009
Continuities with RAE, but impact to equal one quarter of assessment scores. Zoë Corbyn reports
The impact of academics' work on the wider world will determine one quarter of universities' overall score in the forthcoming research excellence framework, with the number of assessment sub-panels to be cut by more than half.
These major changes to the way research is assessed were set out by the Higher Education Funding Council for England this week.
The REF, which will replace the research assessment exercise, will put particular stress on the economic and societal benefits of research, and will also see the greater use of metrics to inform peer review and more consistency among panels.
Hefce's 56-page consultation, the academy's final formal chance to comment on the REF, comes 18 months after it was first proposed that a mix of metrics based on citations, research income and postgraduate student numbers would determine the allocation of nearly £2 billion in annual quality-related research funding from 2014.
However, the plan that has emerged, which follows numerous studies, a bibliometrics trial and intense discussions with the sector, looks very different from the original proposals.
It resembles an evolution of the peer-review-based RAE more than a revolution based on metrics. Hefce suggests that significantly reducing the burden of peer assessment through the use of research citations while retaining robust judgments on quality are all but incompatible aims.
"While we remain concerned to reduce the burden of the assessment, we believe we have exhausted the main options for any radically different alternative approach (to the RAE)," the document says.
Output, impact, environment
The consultation proposes an assessment regime based on three elements: "output quality" (weighted at 60 per cent); "impact" (25 per cent); and "environment" (15 per cent).
The impact measure is introduced as a distinct element of the research assessment for the first time, in line with the Government's so-called "economic-impact" agenda.
As in the RAE, "output quality" will continue to be based on peer review, referred to as "expert review" in the new model.
In a change from the previous system, however, citation data will be available to "supplement and inform" the assessment of various science subjects (see "REF at a glance", right). Other subject sub-panels will decide whether they want to use such data.
"Impact" will be judged by the sub-panels only for high-quality research. Impact must be evident during the REF assessment period, although the research could have been carried out ten to 15 years before. It will be assessed using departmental case studies - one for every five to ten staff submitted - and an impact statement from departments.
The "environment" measure will consider factors including research income, the number of postgraduate research students and their completion rates. Panels will decide whether bigger departments with "critical mass" should receive special recognition.
Hefce's draft plan is to apply the same 60:25:15 weighting across all units of assessment (UoAs). This will lead to "greater simplicity and comparability across the exercise" and reduce the scope for "tactical decision-making about which UoA to submit work to", it says.
In a controversial move, the number of UoAs and sub-panels is to be cut from 67 to 30, and the number of main assessment panels from 15 to four. "We believe that the proposed breadth and scale of coverage ... is manageable," Hefce says. "We are committed to substantially reducing the number of UoAs and do not wish to consider arguments for retaining comparatively small discrete UoAs."
A single sub-panel will assess outputs, environment and impact, with "specialist advisers" helping on outputs and "research users" doing so on impact. Here the role of academics will be downplayed.
Roughly the same number of experts will be involved in the assessment as in RAE 2008, David Sweeney, Hefce's director of research, said. The consultation, published on 23 September, asks for feedback on whether the number of outputs submitted per person should be reduced from four to three, and whether certain "substantive outputs", such as monographs, should be "double-weighted".
As in the RAE, the selection of staff and their outputs will be left to institutions, and it is proposed that the "profile-based" assessment pioneered in RAE 2008 will continue.
The consultation also proposes honing the descriptions of "world-leading" (4*) and "internationally excellent" (3*) research used in RAE 2008 to achieve "greater discrimination" at the top end of the scale.
It also seeks to resolve the issue of whether the proportion of eligible staff submitted should be made public, with Hefce altering procedures to ensure it will be.
"We recognise that some institutions feel strongly that they should be able to demonstrate their research intensity," the document says.
We're all ears
Mr Sweeney said Hefce was seeking a "genuine consultation" and was in "listening mode". He added that its intention remains to "identify excellence wherever it is found".
Bahram Bekhradnia, director of the Higher Education Policy Institute, said it was a "great relief" that Hefce was proposing a peer-review process informed by metrics.
"Although the proposals have a new name, they are a development of the RAE," he said. However, he suggested it was "surprising and perhaps unwise" that as much as 25 per cent of overall scores would be attributed to impact.
"It would have been wiser not to give so much weight to what is effectively an experiment," he said.
He described the reduction in the number of sub-panels as "understandable and brave", but said it would be "damaging to the credibility of the exercise if large numbers thought that they were being judged by people without the competence to do so".
The proposals were also welcomed by Research Councils UK.
"We support Hefce's position that there is a role for quantitative evidence, including citation information ... but believe that peer review should still form the backbone of any assessment of research excellence," an RCUK statement says.
The consultation closes on 16 December.
zoe.corbyn@tsleducation.com
REF AT A GLANCE
- Assessment to be based on three elements: research output (60 per cent); impact (25 per cent); and environment (15 per cent).
- Citation data to be used to inform expert review in units of assessment covering the medical, health, biological, physical and computer sciences, along with psychology and engineering.
- A two-tier structure is planned, with 30 sub-panels (one for each unit of assessment) working under the guidance of four main panels.
Times Higher Education is sponsoring a major conference on the REF with Hepi on 14 October. For further information: www.hepi.ac.uk
Readers' comments
What does it mean to "identify excellence wherever it is found"? I mean, if it's already found, then... BBC should look at universities for the inspiration to produce another Keeping Up Appearances.
It is all rubbish anyway, causing enormous amounts of paperwork and of no relevance; Research by its very nature is supposed to be generated by the staff appointed in University by their expertise at degree and then they research as part of their paid work to include lectures unless the work is design studio, hospital (clinical) or they are a Reader. Research Asst Fellow is a separate institute arrangement. Nobody has the right to investigate research of University staff except in critical sectors, or vast funded sectors, and supervisors supervise they do not dictate. Especially not political lobby groups external to University. Institutions deal with undergraduate courses to professional practice requirements. All the country is doing is producing vast loads of documents as the Press (TV) is trying to make research private company funded, high flying public image, isolated high cost unit, directed rather than innovative and productive, agency controlled, where any old press journalist can put their nose and make work for themselves publishing it. Coast BBC . as they have been for forty years. We do the work, other people try to make money for themselves and get elected using it ... if this country got to grips with ***required*** work programmes outside University employment as designated funded and rates and tax paid we would make some progress. Who is going to decide what is socially acceptable ... Gordon Hyden Peter Haggett The Ecologist Barack Obama ... politics politics and urban congestion, they did not want the job, nobody wants it except the press during election time which has been going on for four years to 2010. Soon be time for another USA whistle stop start and the banks stole all the finances anyway. Go write about soccer and catwalk fashion and let us actually do some work.... Or ! accept RAE is a political white elephant in a Kenyan desert and should never have been, as Malvinas 1982. We actually have important work to do, not form filling for unemployed executive Civil Servants leftovers to make power bases for the press to start another destruction on behalf of Hitler, who is dead. Waiting for a Messiah of the New Revolutionary Academic success i.e. how to sit around and not do any work as established already while the country sinks under concrete. Steel windmills around and around the same old Tokyo summit since 1946. And long may we live in peace without it. Mike Stagg BScWales hydrology soil Geology MSc I did 1969 2009 ... they are all gone !!!!!
Michael Dennis Stagg: I agree about the RAE being a waste of time, not sure about the rest of what you are saying though...
The last 100 years were a relentless war against the doer. The war to make everyone on the planet a stupid employee. Even Voltaire and Rousseau were in private life dissolute wicked cheaters, the predecessors of yappies and hippies. No wonder, there is nobody now to shout: OUT! Still, someone must shout: OUT!
The key problem with the REF proposal is not the proposed use of metrics (although there are problems with using metrics in this way) but the introduction of impact as a massive contributor to overall research evaluation. This overlooks the intrinsic value of excellent research, which has nothing to do any economic uses that it may or may not have. The two greatest research contributions (arguably) of the modern age, namely Newton's work on mechanics and Einstein's on relativity, had little or no discernable short-term impact.
And do you think they don't realise what they doing? Sure, they do. But they intentionally ignored all previous knowledge and experience of academia to boost mediocrity and add, totally ridiculously, the importance to themselves. Surely, this is revolution. Thousands of people know what they are seeing, but they fear for their jobs, not understanding that it's practically the last opportunity to speak out - revolutions are difficult to stop later. And the situation is even worse: young generation takes this garbage seriously, old people die. The knowledge accumulated for millenia is not there anymore. This is the strategy of "Change" - go slow because quick revolutions have all failed, preach the Change to the youth and have patience. Move the converted youth to the top. Enroll all into the system. Don't let anyone stay stay behind, God forbid, with their own business.
Let me suggest an alternative view. Ignoring impact gives little credit to those people who do applied research. It is not second class just because it is applied. Some of the RAE panels downgraded grant/contract money that came from Industry. Can I suggest that rather than lamenting a perceived loss of influence that you rather see this as broadening the metrics. And yes, before anybody says anything I know the difference between applied research and a bit of contract income that is just technical. And as for the RAE. UK PLC is spending 1.2 billion on the research component of the HEFCE grant. How should that money be disbursed? If you can think of a better way then please enlighten us. Yes the RAE/REF is a flawed beast but so are the alternatives and some are worse than others.
and what exactly is wrong with expecting research funded by the taxpayer to demonstrate some evidence of public good? The problem is how you define 'impact'... and I'm sure we'll all be working creatively on that problem from this point in...
I would like to point out that Fourier's and Laplace's work in statistical mathematics and calculus proved almost useless until over 150 years later, when Claude Shannon's masters thesis coupled with the work of Harry Nyquist finally illuminated to us the use of discrete data to describe continuous processes without loss of information. Not terribly enthused about the REF - impact assessments are often useless tools, for the reasons outlined.
Damien and Paul. Yes I could also quote examples of work that was ignored for years. It all depends on what Impact is? I simply believe that "Impact" is a widening measure and that has to be good. In the last RAE, imagine the paper that was published that improved some industrial process. Perhaps it was an average sort of paper. In reality it saved the company millions, reduced CO2 emissions by some huge amount and saved jobs. Now I actually know some work that was published with that profile. I would say that that was high impact and as worthy as some Nature/Science paper. Thus I do not see impact as robbing one field, I see it as an act that broadens the field. Anyway, come on folks, we are used to the goalposts changing, it adds to the challenge and keeps us on our toes right!
Academic - we are used to the goalposts changing, I just resent them being moved by QUANGOs, Civil Servants and Politicians.
Damien. The RAE/REF will decide how to disburse 1.2 billion of taxpayers money. One could argue that in a democracy that Politicians and their administrative aides (civil service) have the electoral right to shift the posts where they want them. The alternative is to lose the dual support system and chuck all the money into the Research Councils. I am not sure that I like that solution very much.
Fourier, Laplace, Newton, Eisntein .. sure. Perhaps in place of evidence of impact academic could provide evidence that they are in the same league as those worthy names.
Dr. Truth. Your argument seems to be that a small number of excellent academics in the past were unencumbered by the RAE or the need to list impact and so nobody should have to do it now. Can you spot the flaw in your argument. I would point also out the folk that you list were not funded by the public purse to the extent that we are. If you want HEFCE money then you have to play by their rules. Simple isn't it? If you dont want public money go and work for the private sector.
academic: 'I simply believe that "Impact" is a widening measure and that has to be good'! Do you work for HEFCE?
Nope. I just believe that basic research and applied research are both valuable. I have known many academics, including some RAE panel members who looked down on research if it was even remotely applied (even if it was well cited). I just see that the implementation of this scheme offers some hope. And before anyone asks, some of my work is mildly applied but most of it is not. So Whippet how would you disburse the HEFCE cash then?
Why would anyone spend a lifetime working hard for little money, if they did not think their research had an 'impact'? The real issue is how to define that impact. Most university academics' work, beyond teaching, is essentially dedicated to making the research community they inhabit function - providing the prose that justifies the publication of academic journals that three people read. But we too seldom explicitly justify the existence of those same communities. Can I suggest we look a bit harder at the purpose of all our hard thought and hard work (even when read by only three people), and put a bit of effort into re-affirming the role of academics and intellectuals in making society as a whole work. In other words and for myself, I look at 'impact' and its place in the REF as an opportunity to justify the work I do to that broader population that pays my (and your) salary. If we can't make a compelling case, if all the well-educated and smart people employed by our universities can't make this small argument effectively, then we don't deserve the money in the first place.
Well put Tim.
Tim Hitchcock - your suggestion is admirable, but the central point is surely that when it comes to justifying 'impact' no matter how well you put your case, the assessors are unlikely ever to agree? You then open up the questions of equivalence - the DSP research student, developing code which might immediately have a wealth generating impact in the private sector, versus the obsessive history student who manages in five years to illuminate details which were otherwise lost but, conversely, cannot hope to generate wealth by substantiating such truths. I cannot honestly say that the former is more 'valuable' than the latter simply because its economic impact is noteworthy. Who can? There have been countless instances where hugely important pieces of work have gone without citation for decades, only to have their importance rediscovered by a researcher who accidentally picks up the subject thread all those years later. There is some nonsensical research being undertaken, mostly by the lower level institutions who are punching for the quick publicity rather than attempting to expand and extend the limits of human understanding. Clearly academics cannot always be left to themselves to pursue whatever they see fit, but the REF does not seem to have introduced a particularly happy medium according to my understanding of it.
To academic: you comment about the flaw in my comment is flawed. Can you spot the flaw?
Dr. Truth..perhaps you should visit Dr. Grammer-Checker!
I am concerned about the need to assess 'impact'. I think it should be for HEFCE to demonstrate how their assessment system would have ranked work by Newton, Einstein etc, given that much of this work did not have practical or economic application in the short-term (or even medium term).
@Fearing the worst - indeed, or ask them to assess how long it took to discover the application of the Laplace transform and how this would have faired in their 'impact' assessment.
Ummm.... I wonder what the impact of this 'impact' component to the new REF will be. My guess would be that it will be negligible, simply because it would face considerable opposition if it were to be otherwise. So as time progresses and we approach the REF things will be much as before. More money wasted, a total waste of everyone's time. @academic: I would remove HEFCE, concentrate all the research funding in the top thirty universities and make the rest teaching only universities, simple as that.