REF consultation: academe's concerns

Sciences and humanities must not be judged differently, and peer review and timetabling need reassessment, organisations say. Zoe Corbyn reports

February 14, 2008

Organisations responding to a consultation on the system to replace the research assessment exercise have made it clear that they are not happy with the proposed division between the way science and humanities subjects are to be judged.

This, and worries that the timetable for introducing the new system is too tight, have emerged as the overwhelming concerns that unite the many groups responding to the consultation on the new research excellence framework, which closed this week.

Subject divide

The Higher Education Funding Council for England has argued that it should first develop the REF for the Stem disciplines - science, technology, engineering and medicine - using "metrics" on research income, postgraduate student numbers and the number of times academics' published papers are cited by their peers, to judge research quality. This system is to be piloted this year for introduction in 2010.

For the other disciplines - arts, humanities, social sciences and statistics - Hefce proposes "light-touch peer review" informed by metrics, which will be introduced from 2013, although it will not begin significant work on this system until 2009.

"We have some concern that the science/non-science split as currently proposed will have implications for research that straddles the divide," a spokesman from Universities UK said. "We would like to see more evidence developed in this (non-science) area and more of a continuum approach taken ... This may require speeding up development of the non-science approach."

The 1994 Group, which represents small research-intensive universities, said: "We are concerned by the current lack of attention in the consultation document to non-(science) subjects and feel that this may prove damaging unless addressed very soon."

"It is a fundamental principle for the operation of any research assessment and funding system that it operates for all subject areas," added its submission.

Wendy Piatt, director-general of the Russell Group, which represents large research-intensive universities, said: "We are keen to emphasise that there should be no sharp division in the sector between Stem and non-Stem subjects."

A spokeswoman at the Royal Society said its response is expected to say that it "remains to be convinced" that the proposed division between science and non- science subjects is appropriate.

The British Academy is expected to respond that it doesn't "believe such rigid delineation is helpful".

Timing

The responses also show clear pressure on Hefce not to introduce the REF for science disciplines so soon.

UUK called the timetable "very tight" and asked for "more time to pilot and test the new approach". Both the Russell Group and 1994 Group submissions called for an extension of the existing timetable by at least 12 months.

Stephen Smith, chair of the 1994 Group, said there was a danger of "planning blight" if decisions on how non-science subjects were to be assessed were not bought forward.

The role of peer review

An analysis of the responses also shows a continued divide over the role of peer review in the new system for assessing the science disciplines.

Many learned societies seemed to want to retain peer review at least in some measure to counter the shortcomings of citations, but UUK, the Russell Group and 1994 Group appeared broadly content with the "basket of metrics" proposed - which would see peer review replaced with six "peer panels" to oversee how the metrics were combined to judge research quality in each subject.

"Research assessment should include a light-touch peer review," said the submission from the Royal Academy of Engineering, which highlighted "significant problems" with the current proposed metrics system for engineering, given that the subject does not use citations to the extent others do.

The Royal Society spokeswoman said that its submission is expected to argue that there remains a need for peer review panels informed by quantitative and qualitative indicators.

The Royal Astronomical Society's response urged Hefce to "recognise the shortcomings of (citations) data and require a panel of practitioners to carry out ... 'light-touch' peer review".

"Before Hefce adopts a bibliometric indicator, the Government needs to assess if it wants the scientific community to try to maximise citations," said Nick Dusic, the director of the Campaign for Science and Engineering. "The REF should retain an element of peer review to minimise any undesirable effects of metrics and to assess research outputs that cannot be quantified with a metric."

New measures

Also evident among the submissions is a desire to include additional metrics to measure the quality of research that does not necessarily get cited by academics but is of benefit to the economy or society.

"We need to explore further how the economic and social value of research can be factored into any new system," said Les Ebdon, chair of Million+, which represents post-92 universities.

"Indicators for applied and user-focused research present a real challenge," a UUK spokesman said. "Hefce should not shy away from this."

The Confederation of British Industry's submission said: "There are a number of ... indicators for the impact of user-valued research that should be incorporated."

The Royal Society spokeswoman said that its response is expected to argue that "the overall university funding system should recognise excellence in all kinds of research including interdisciplinary and user-focused research as well as broader research activities such as public engagement and providing advice on policymaking".

Within the humanities arena, Hefce is perhaps likely to take heart from the fact that the community appears at least to have refrained from an out-of-hand dismissal of the role of metrics - such as citations.

The British Academy said the new system "should be informed by and supported by metrics", but it also highlighted that for the vast majority of subjects no "credible proxy" was yet in existence.

The Academy of Social Science's submission said: "We see metrics as informing judgments that should be primarily based on peer review ... The academy is not in favour of 'light-touch' peer review if it means superficial peer review."

Both submissions from the Political Studies Association and the Council for Higher Education in Art and Design called for more information on what 'light-touch peer review' means.

zoe.corbyn@tsleducation.com

Views on the research excellence framework

Universities UK

"We have some concern that the science/non-science split as currently proposed will have implications for research that straddles the divide ... Indicators for applied and user-focused research present a real challenge. Hefce should not shy away from this, and we believe further evidence is required, and further developments will require input from the user communities. The timeframe for the development and implementation of REF is very tight."

Russell Group

"The robustness of proposals is considered to be more important than the proposed timescale ... as a result (we) are calling for an extension of the existing timetable by at least 12 months ... There should be no sharp division in the sector between Stem and non-Stem subjects."

1994 Group

"The RAE has allowed reliable comparisons to be made between subject units, institutions, and countries. It is essential that this aspect is preserved ... It is vital to ensure that the framework is the same for Stem and non-Stem subjects to discourage any division of higher education into 'two communities'."

Million+

"The real challenge will be to make sure that any framework captures innovation, value for the economy in its widest sense and incentivises potential."

Royal Academy of Engineering

As currently proposed, (the REF) would have dramatic effects on the way in which engineering research is carried out in the UK ... The Academy would suggest all research assessment should include a light-touch peer review."

Register to continue

Why register?

  • Registration is free and only takes a moment
  • Once registered, you can read 3 articles a month
  • Sign up for our newsletter
Register
Please Login or Register to read this article.

Sponsored