Structural adjustments

It has now been more than three years since the Treasury made an announcement, unexpected to everyone but a handful of vice-chancellors, that sent shock waves through the academy.

After years of hearing universities complain about the burden imposed by the research assessment exercise, the game-playing it caused and the way it impeded interdisciplinary and user-focused research, in March 2006 the Government told the sector that RAE 2008 would be the last. In its place would be a simple numerical system for the evaluation of research quality and the distribution of the funding councils' block grants for quality-related research (QR) funding. Because there was a strong correlation between research-grant income and research quality (in the science subjects at least), this relationship would be placed at the heart of a new "metrics-based" system. Peer review would be gone, as would the bureaucracy and all the attendant problems of the RAE.

Fast-forward to today, and with the dust settling on RAE 2008, across the sector minds are increasingly focusing on what the system scheduled to replace the RAE from 2014 - the research excellence framework - will look like. An expert advisory group of more than 100 people, including many academics fresh from sitting on RAE panels, have been drafted in by the Higher Education Funding Council for England to help shape the proposals; bibliometrics (the number of times academics' work is cited by peers) is being assessed to see how it could be used in the new system; workshops have been held to discuss topics such as the "accountability burden" and "user-valued research"; a collection of consultants have reported and there are more reports in the pipeline; and there is plenty of informal dialogue with various sector groups. The final proposals are due to be published for consultation in autumn.

The stakes are high. Not only because the REF will drive academics' behaviour in the years ahead, but because some believe that the QR pot faces the threat of being subsumed into the research council pot - the other arm of the so-called dual support system for research. Unless Hefce can produce a model for distributing QR cash that has the sector's confidence, it could be in a weak position to defend the funding stream against efforts to shift it across to research councils and the newly created Technology Strategy Board - in effect a research council for industry - where it is both easier to see where the money is being spent and where the Government can exert greater strategic control over it.

Of course, the new system looks very different from the Treasury's initial ill-conceived vision that had the sector up in arms, and which was passed to Hefce at the end of 2006 to try to sort out. The notion of using research income as the main proxy for quality was rejected after an initial government consultation showed little appetite for the approach and much enthusiasm for retaining at least some peer review. The next iteration, a "basket of metrics" for the sciences - consisting of bibliometrics, research grant income and research student numbers, with a separate light-touch peer review for the arts and humanities subjects - also received a decidedly lukewarm response.

For the past year, however, there has been in place a vision that more seem happier with: a single system that straddles both the sciences and arts and humanities subjects, in which different disciplines will use a combination of "expert review" and unspecified "quantitative indicators ... including bibliometrics where appropriate". The importance of research income is much reduced in this approach.

It is the details of the new assessment regime that are now being finessed by Hefce, and what are emerging as the three main elements are "quality of research outputs", "research environment" and "social and economic impact of research" (see boxes). But in this process, many observers detect something remarkably like an evolution of the RAE - "the RAE with numbers" - rather than a radical new metrics system. "We see the REF as a further development of what we do now, not something completely new," Hefce states in material that it has presented in various meetings.

"We are going to have something that looks very much like the RAE, with some numbers attached to it as well," says Bahram Bekhradnia, director of the Higher Education Policy Institute and an architect of earlier assessment exercises. "The best reform (to the RAE) is for metrics to be available more systematically to the panels to use in their peer-review judgments, and it looks like it is moving very firmly in that direction. We are on track for a sensible result."

The development process has been followed closely by Geoffrey Crossick, warden of Goldsmiths, University of London, who recently chaired one of Hefce's workshops. "I think the outcome is not going to be as different from the RAE as we might have expected even a year ago," he predicts. "As someone who has always been worried that a bibliometric or general metrics-driven research-evaluation system would be damaging, I am feeling much more comfortable with the general direction. It does not mean that we are there yet, but the sector is effectively working with Hefce now, which is really very good."

Both Bekhradnia and Crossick credit Hefce for its effective mediation between the Government and the sector, which they believe has moved the original proposals to something more broadly acceptable. "It would be a brave Treasury person who told Hefce that it had to go back and do something similar to the first attempt," says Bekhradnia. One vice-chancellor who wishes to remain anonymous sums up the sector's view as: "Everyone is happy as long as the words 'peer review' are there."

But not everyone is happy. There is an alternative view that Hefce is allowing the academy to ride roughshod over it. Eric Thomas, the vice-chancellor of the University of Bristol and head of the research policy committee for Universities UK, the vice-chancellors' body, says his personal position has always been clear. He believes that the RAE has "had its day" and what is needed is a simple, auditable, unbureaucratic formula-based system, with research-grant income and research student numbers serving as the important metrics - in other words, something more akin to the Treasury's original proposal.

"What I don't want is a mutated RAE by another name ... I would not be happy with something that is basically a reconstructed RAE with all the attendant bureaucracy and distortion of planning that occurs with that," Thomas says, hinting that the knives will be out if what Hefce presents in the autumn does not pass muster.

Michael Sterling, who has just retired as vice-chancellor of the University of Birmingham, also advocates abandoning rather than rejigging the RAE. He believes that a wide basket of metrics offers a more objective and efficient approach than a peer-review system.

"There is a real worry that Hefce will just repeat the RAE because it is better the devil you know," Sterling says, adding he would like to see how RAE 2008 scores correlate with some of the proposed metrics. (An earlier plan to shadow the 2008 RAE with a metrics exercise did not take place.)

David Sweeney is Hefce's director of research, innovation and skills. Doubtless aware of the political sensitivities for the organisation of being seen to be merely nudging the RAE forward rather than offering something new, he shrugs off the comparisons.

How different will the REF be from the RAE? "I think it is in the perception of the beholder. It clearly is different. It will involve bibliometrics and the consideration of research impact. Some will perceive that as radically different. It is a robust approach that takes into account as much robust quantitative information as we can," Sweeney says.

But he also stresses that Hefce is "building on the RAE" as opposed to undertaking a complete "step change" in direction. "There is an acceptance that the RAE was a robust process," he says.

Yet, if the REF is an evolution of the RAE - as the majority in the sector appear to favour - rather than a revolution, it raises some very big questions.

Not least is what this move might mean for research concentration, which is emerging as a key issue in the debate about QR funding. "Everything I hear (from the Government) is about concentration, and the RAE is thus seen as a blip," says the vice-chancellor.

Largely because it provided "research profiles" of departments rather than the single summative grades of previous exercises, one of the most surprising results of RAE 2008 was the identification of hotspots of excellent research at teaching-led universities. As a consequence, the QR pot has been distributed more widely, reversing the trend of the past 20 years for funding to become concentrated in a handful of research-intensive universities.

In the wake of RAE 2008, the number of universities that account for 75 per cent of mainstream QR income has increased from 22 to 26.

Yet the Government appears not to want to see this diversification continue. Speaking to Times Higher Education earlier this year, John Denham, the Universities Secretary, said that while it is important that RAE 2008 recognises pockets of excellence, he did not believe that their growth under the REF was the "right strategy" for delivering a world-class research system. Indeed, a level of concentration "perhaps not dissimilar" to what existed before RAE 2008 is his preference, he said.

Yet it is far from obvious how the REF, as it is shaping up, could help drive this agenda. While it is easy to see how a system with a strong emphasis on metrics, such as research grant income and research student numbers (already highly concentrated in the research-intensive universities) might easily deliver more concentration, it is less easy to see how the system that Hefce is developing will do so. "Everything I read about the REF makes me think it is going to result in less concentration," the vice-chancellor says.

For a start, the use of profiles, which allowed "pockets of excellence" to shine, looks set to continue. The Government's insistence that the social and economic impact of research must also count under the REF could also work against concentration.

"You could find", Bekhradnia notes, "that different universities are scoring better on impact and utility of research than the research-intensive universities." It would not be the first time Hefce has been left trying to reconcile contradictory wishes from the Government, he adds.

"The way (Hefce) would have to (engineer the system) is for the REF to, in some way, focus on collaboration between pockets and centres of excellence," the vice-chancellor says.

Another concern is how the administrative burden of the RAE will be reduced under the new system. Ministers listened to a lot of complaints on this issue in the run-up to the 2006 announcement, Thomas notes.

John Rogers is director of research and enterprise at the University of Stirling, one of the institutions participating in the bibliometrics pilot. He says there is "no confidence" as yet from pilot institutions that the REF will be a lower cost or a less-onerous system to operate. Hefce is working with institutions to identify opportunities for streamlining, says Sweeney.

Nonetheless, if the sector wants a robust system that has the academy's confidence, is there really any scope to cut the administrative load?

"The burden under the REF won't reduce, and if anything it may even increase," Bekhradnia says. "If the assessment is going to be fair and comprehensive and sophisticated, and used to distribute billions of pounds, it has got to be rigorous and it is unlikely to be light touch."

He notes that universities' burden under the RAE grew largely because of changes that academics and institutions had requested to try to make the system fairer, from rules covering early-career researchers to those addressing academic transfers. Bekhradnia also points out that the cost of assessing work under the RAE is actually far cheaper than it is under the research council system.

PA Consulting recently reviewed the costs of RAE 2008 for Hefce. After factoring in a range of activities from academics' time to software costs, it estimates the total cost to the sector at £47 million. Adding in the £12 million cost to Hefce of administrating the exercise, the total bill for RAE 2008 stands at a mere £60 million - the equivalent of 1 per cent of the £6 billion in research funding that it will distribute over its lifetime. Compare this with the research councils, whose administration costs are about 4 per cent of their total budget.

For Crossick, the burden under the REF is "not going to be massively reduced, if at all" because of what the academic community has been saying about "getting it right and doing it properly". He believes that the sector, having decided the exercise must be done "longhand" because a full-blown metrics system is not acceptable, must bite the bullet and support the result.

"We have to say that if the burden is not significantly reduced, we are prepared to carry it ... that (no reduction) is OK because we want the more robust approach that comes with it. What would really be dangerous is if we said we want it to be robust, but we don't want the burden." In that case, he warns, the Government may throw up its hands in exasperation and go back to the original Treasury ideas. Hefce, Crossick believes, seems prepared to back a system where the burdens aren't substantially reduced, but only as long as it has the support of both academics and administrators.

Yet, even within such a system, there may be ways to simplify things. Crossick argues that there needs to be less power given to panels and more decisions taken centrally. "One lesson from RAE 2008 is that we had too much proliferation of diversity for its own sake in subpanels, in some quite important areas, where it should not have existed. We need to stop the proliferation of quirky ways of doing things by panels for no good reason, and we need more uniformity than we had last time."

It is a valid point, but it may not mesh with the views of academics, many of whom see their disciplines as so unique that they are determined to resist what they see as too much standardisation.

Nigel Vincent, Mont Follick professor of comparative philology at the University of Manchester, is an expert advisory group member who was one of the 15 main panel chairs in RAE 2008. He argues that to carry the confidence of the community, panels must be given a considerable say in defining the criteria that are used to judge the research in their subjects. "If the panels have the kind of latitude that they had in RAE, then we are OK," he says.

Another sensitive area for Hefce is how to tackle game-playing. The strategic and tactical ploys that institutions employed to try to maximise their RAE scores have caused much concern, with most frustrations focused around staff selection.

Yet is a reduction in game-playing really a likely outcome? Surely, say observers, institutions will find ways to boost their scores, whatever the system.

A recent Hefce workshop on the accountability burden came to just such a conclusion: that if institutions were obliged to enter "all staff" in the REF, a new set of practices would be developed around staff contracts and achieve the same selectivity many universities exercised in RAE 2008. Moreover, workshop participants concluded, it would also lead to more low-quality work having to be assessed, doubtless to little avail, as it would be unlikely to produce any QR funding.

Crossick argues that game-playing is a pejorative term, and something that "by definition" no one likes. "To call it game-playing seems to me to somehow to miss the point. Any research evaluation system will shape the behaviours of institutions and departments. That is why we have it. The argument is how we make sure we shape the right kinds of behaviours."

It is these politically sensitive issues that will likely play as great a role as academics' wishes in shaping Hefce's autumn proposals for the RAE's replacement.

IN DEVELOPMENT: EMERGENT FEATURES AND UNRESOLVED QUESTIONS

Although the detail of the research excellence framework is still being worked out, some key points are becoming clear.

- The big picture emerging from the Higher Education Funding Council for England is that the REF will consist of three elements of assessment: quality of research outputs, the research environment and the social and economic impact of research.

- An assessment based on profiles showing the proportion of a department's work judged to fall into specific categories and used for the first time in RAE 2008 is likely to remain after it proved popular among experts.

- The dominant element in the new system will be the quality of research outputs as judged by expert panels of academics and research users. Bibliometrics (the number of times academics' work is cited by peers) will inform expert review in the disciplines where they are found to be robust. Disciplines where the bibliometrics are not robust will use expert review alone. Hefce will probably determine which subjects are fit for bibliometrics.

- The difference between "expert review" and "peer review" merely reflects the inclusion of more users on the REF panels than in the RAE.

- Of the other elements, the approach to assessing the research environment would in essence be carried over from the one used in RAE 2008, with panels again making use of metrics indicators such as research-grant income and postgraduate student numbers.

- The inclusion of a new social and economic impact element is a response to the Government's determination that the REF should better evaluate the contribution of research to the economy, society and public policymaking (see box page 36).

- The "esteem" element from RAE 2008 is to be scrapped. Few experts felt it was particularly useful.

Many practical questions remain unanswered. Key decisions include:

- whether all staff should be included or whether selection should continue. There is no clear consensus, but there seems to be a general feeling that there is "no real alternative" to staff selection;

- whether papers should be credited to institutions or to individuals. Again, there is no clear consensus, but many think papers are better credited to authors so as to promote mobility;

- how many panels are needed. Experts say there is "limited scope" for a move to fewer panels, and current suggestions include "no less than 50 to 60" with a two-tier panel structure remaining;

- how many outputs researchers should be required to submit. Some suggest giving panels scope to vary the number from the current four to a range of perhaps two to six;

- what should be the frequency of assessment. Most experts argue that five to six years is appropriate;

- how much authority panels should be granted. Whether it is the panels or Hefce that will determine how the three elements are combined in each subject is still to be decided;

- how bibliometrics can be combined with expert review and how impact can be best assessed. These will be considered by Hefce's experts over another two rounds of meetings.

COUNTDOWN TO THE REF

• March-June 2009: Conduct bibliometrics pilot and develop proposals
• Autumn 2009: Consultation
• Spring 2010: Announce decisions on the REF
• 2013: Complete first full REF exercise
• 2014: Research funding fully driven by the REF for all disciplines

Source: Higher Education Funding Council for England

RESULT? THE ASSESSMENT OF SOCIAL AND ECONOMIC IMPACT

One of the academy's greatest concerns is how social and economic impact, the most novel element in the REF, will be evaluated.

David Sweeney, the director of research and innovation at Hefce, says the REF will assess impact in a way similar to the RAE, but it will be more explicit.

Departments will provide a narrative briefing, like they did for the RAE, and panels will have available to them a range of metrics indicators that they can use to "inform their expert judgments".

"We have a basket of metric information that is routinely collected, but we don't expect it to be robust enough to measure impact on its own" because of problems of time lag and attribution, he says. "We do not anticipate that a metrics-based approach can capture a sufficient range of impacts."

As in the RAE, rather than each individual output having to demonstrate impact, the portfolio of work based in a department will be assessed.

Although the message may reassure academics, the REF still has a long way to go, and industry appears keen for hard metrics to be employed.

"It is important to have a narrative, but it is metrics that incentivise true collaborations with industry," says Jackie Hunter, senior vice-president for science environment development at GlaxoSmithKline and a member of Hefce's expert advisory group.

She cites numerical indicators, such as joint authorship and exchanges with industry, as those she would like to see used in the REF.

Alison Hodge, university partnerships director at QinetiQ, is also part of Hefce's expert group. She believes that academics have yet to take impact seriously enough, hiding - as they do - behind arguments so that it takes too long to gauge and that one can never know at the outset what will be produced. There is, she says, "scope and need" for blue-skies research, but much research is in the "muddy middle" - neither really world-leading and novel or really focusing on national needs.

Hodge believes that this "muddy middle" should be directed to national priorities. This can be achieved by rewarding academics for working with industry "much more seriously" than has been done in the past - or is proposed by Hefce, at the moment. "Narrative can be anything from the most trivial short statement to the Encyclopaedia Britannica. I think it needs some metrics ... they do tend to drive behaviour," she says.

Malcolm Grant, provost of University College London, chairman of the Russell Group of research-intensive universities and a member of Hefce's board, provides the riposte. He argues that a simplistic application of impact could cause a "significant distortion" to the REF's emphasis on research quality, potentially threatening the UK's globally competitive research profile.

"We have got to be sure that we don't get ourselves into a trap in which impact becomes the major driver of funding," he says. "I don't think people are asking our peers and competitors in China or the US about the impact of their work as a precondition to making grants."

THE FULL COUNT: HOW BIBLIOMETRICS MIGHT WORK

The use of bibliometrics to inform expert review is a feature that distinguishes the REF from the RAE.

So far, the only country to use bibliometrics to determine public research funding is the Netherlands, which has seen a steep rise in the number of publications and citations as a result. Australia is examining its use as part of a metrics-heavy Excellence in Research for Australia system it is developing.

In the UK, 22 universities have been taking part in a pilot scheme, conducted by the data-analysis firm Evidence Ltd, to work out how bibliometrics could be applied. While results are yet to be published, the picture emerging is one of technical feasibility - but there will be a choice to be made between a top-down or bottom-up approach.

After initial set-up costs, the top-down approach would be quick, easy and cheap. All the papers produced at a particular university - as recorded in either the Web of Science and/or Scopus citation databases - would be extracted and broken down into their disciplines. The journal in which a paper is published would determine which discipline it counts in. Journal lists would be the same across the sector, allowing like-for-like comparisons between institutions.

A problem with this approach is that while it may give information about the discipline, it does not necessarily provide a fair picture of what researchers in a particular department are actually doing, publishing as they do in all sorts of journals outside the set that would necessarily define their subjects.

The full range of this activity can be captured via bibliometrics, but it requires a bottom-up approach. Similar to what happens under the RAE, institutions would produce a list of departments' publications - in all journals. These publications would then be used for the citation assessment. This approach would offer a more precise picture of a department, but it is more onerous and likely to require intense validation by universities.

The question as to which approach to take is not technical but philosophical: should the REF, like the RAE, continue to assess researchers within departments or take a leap into the unknown and look at subjects as defined by journals? Plus, how much of a burden is the sector willing to bear?