Advertisement
Advertisement
Advertisement
Hefce backs off citations in favour of peer review in REF
18 June 2009
Research assessments in hard sciences will now be 'informed' by bibliometrics. Zoe Corbyn writes
The use of citations to determine the quality of academic work in the hard sciences is to be abandoned in favour of peer review in the new system being designed to replace the research assessment exercise.
However, information about the number of citations a scholar's work accrues could be provided to assessment panels to help "inform" their judgments in a range of subjects.
At a conference on the forthcoming Research Excellence Framework (REF) this week, the Higher Education Funding Council for England sketched out how it intends to assess the quality of research outputs in the system, which will determine the allocation of £1.6 billion of annual research funding from 2014.
"We just don't think bibliometrics are sufficiently mature at this stage to be used in a formulaic way or, indeed, to replace expert review," said Graeme Rosenberg, Hefce's REF project manager. "However, there is still scope for bibliometrics to inform the assessment process."
The announcement follows an interim report on the REF bibliometrics pilot exercise.
Although all decisions are yet to go to the Hefce board ahead of a consultation paper on the new system due later this year, the council told about 300 conference delegates that it was effectively sidelining the use of citations - a measurement of the number of times academics' work is cited by their peers - as the main determinant of research quality in the remaining hard science subjects for which it thought their use was possible. Instead, it is to draw up a list of subjects where it is appropriate for citations to inform peer review.
Dr Rosenberg also outlined Hefce's preferred model for assessing research quality. As in the RAE, institutions would continue to select which staff and which of up to four of their outputs are submitted.
The first exercise would cover work published between 2008 and 2012, and data on citations accrued for the papers in the same period would also go to panels.
"Panels might look at (citations for) each paper or (citations for) the whole submission. We are still working through the details," Dr Rosenberg told Times Higher Education.
Putting citation data for each paper before panels would be a radical departure from earlier plans, which until now have mooted citation profiles for institutions only in broad subject areas.
Charles Oppenheim, professor of information science at Loughborough University, attended the conference. He said it was now likely that pressure on scholars to increase their citations would grow.
"The new game to play is citations. One of the games, without any doubt, is that senior staff will encourage academics to cite within the institution," he said.
Dr Rosenberg also discussed how Hefce intended to assess the social and economic impact of research. The process would take place at a departmental level and would be based on a narrative backed by indicators and exemplars, he said.
The first REF would assess impact made between 2008 and 2012, he said, although the research itself could have been carried out earlier.
zoe.corbyn@tsleducation.com.
Readers' comments
Can anybody supply any evidence of any research of any merit which has been helped by RAEs and RAFs?
So finally common sense prevails. But I would now like to know exactly which stupid, thoughtless person, blinded by the New Labour mantra of "evidence-based this that and the other", first proposed the hair-brained idea to use citations?? Time for some journalistic digging, I think. This person must be exposed, as they have effectively wasted a huge amount of the time and energy of HEFCE and indeed the academics who actively opposed the idea.
I worry about the biases inherent in the RAE peer review system. I am sure that it is effective in rank-ordering the research of UK departments within a discipline. However, across disciplines there appears to wide variation in average RAE rankings. It therefore seems possible that some RAE subject panels had an inaccurate view of the international importance of the UK research in their field. It would be interesting if THES could correlate average RAE rankings across RAE subjects with the number of top-50 internationally ranked departments in those subjects.
It's probably alright that instead of scrapping panel rankings altogether and hard-wiring the outcome to metrics, the new REF will continuing doing rankings and metrics in parallel, using the metrics as advisory rather than binding. That's fine; it will give the metrics a better chance to be cross-validated against peer judgment (though the hybrid metric-influenced rankings of the new REF will not be as independent a criterion against which to validate metrics as the RAE rankings were, when they were not influenced by metrics). The important thing is to make the battery of candidate metrics as broad and rich as possible. It is true that metrics today are still relatively sparse, but with the growth of open access and a rich variety of web-based metrics emerging therefrom, the power and scope of metrics will now grow and grow. About the possibility of abuse: Yes, one can abuse individual metrics. Downloads are the easiest to abuse. But genuine downloads generate genuine citations, and the correlation is there and can be measured. There are other intercorrelations in multiple metric profiles too. There are endogamy/exogamy metrics: Self-citations, co-author citations, author-circle citations, same-institution citations, same-journal citations. With these, anomalies and abuses can be detected, named and shamed. Multiple metrics create a pattern, a profile. If you artificially manipulate one of them (say, downloads, or citing others in your institution) it will be detectable as a deviation from the normal profile. Once a few of these abuses are prominently exposed and shamed, that will create a strong deterrent against trying such tricks, since the objective is the exact opposite: to increase one's prestige, not to tarnish it. And unlike (some) individual metrics, multiple metric profiles are almost impossible to manipulate jointly: Try writing software to generate bogus downloads of your work looking as if they all come from different IPs the world over, and then try to generate the non-institutional citations that would normally be the correlate of such high downloads. Even that 2-metric trick is not easy to accomplish! Stevan Harnad University of Southampton
REPLY TO RICHARD HALL: ON EXPOSING THE CULPRIT -- Harnad, S. (2001) Research access, impact and assessment. Times Higher Education Supplement 1487: p. 16. http://cogprints.org/1683/