RAE table will be shaken by use of journal rankings

11 September 2008

Major upsets to the existing hierarchy of research excellence can be expected when the results of the research assessment exercise (RAE) are released in December, according to an academic helping to judge the submissions.

Ray Paul, an emeritus professor at Brunel University, has warned that when trying to select the best research papers to submit to the 2008 RAE, some university departments had relied too heavily on the perceived prestige of the journals the research was published in. Such rankings do not equate to research quality, he said.

Professor Paul told Times Higher Education: "The results that come out in December may not be what (departments) expect. They may have got themselves a whole pile of publications in the top-ranked journals but find themselves not top ranked."

Professor Paul, who sits on the RAE panels for business and management studies and for library and information management, uses a forthcoming opinion piece in the European Journal of Information Systems to argue against the use of journal rankings to select the best papers.

The RAE rules prevent assessment panels from using journal rankings to judge quality, requiring them instead to use peer review. But Professor Paul said it was "common knowledge" that universities had used ranked lists of journals to decide which papers to submit to the RAE.

In the opinion piece, Professor Paul cites the example of an information-systems journal that appears "near the top of the table" but that has a "striking" lack of correspondence with quality.

"The reason for this is that the journal, with a very large circulation, has concentrated on making the research results clear to its wider and varied readership, and therefore wants little in the paper about the research content. So when assessing such a paper for research quality - there is hardly any.

"It would appear to me that journal league tables are to do with promotion procedures and rites of passage in academia, not quality research," he adds.

Professor Paul's article belies growing divisions within the academic community over the use of journal rankings. A recent study, reported in Times Higher Education, argued that rankings of political science journals could be used to indicate the quality of publications and should be used in the RAE.

Professor Paul's article, Measuring Research Quality: The United Kingdom Government's Research Assessment Exercise, is due to appear online this month. He includes the disclaimer that his comments are meant to apply generally. "Any inference anyone makes from what I write here about how (my) subpanels are operating is entirely the product of that person's imagination," he writes. Professor Paul is also an editor of the journal.

zoe.corbyn@tsleducation.com

THE LEADING COMPUTER SCIENTIST RECRUITED BY BIRMINGHAM

Kalyanmoy Deb is one of the world's leading computer scientists, with a string of research papers and awards to his name.

So it was a major coup for the University of Birmingham when he joined its School of Computing last autumn - just weeks before the deadline for submitting academics' work for judgment in the 2008 research assessment exercise.

But this week it emerged that Professor Deb has left Birmingham, having been on unpaid leave from the moment of his appointment in October 2007 until his resignation nine months later in July. The revelation of his short tenure has raised questions about the contribution he may have made to Birmingham's forthcoming RAE score and about the quite legitimate scope universities have to boost their RAE entries.

The university declined to confirm whether or not Professor Deb's work had been submitted to the RAE, citing confidentiality. But in a statement to Times Higher Education, it confirmed that he had joined on 1 October 2007 - a month before the date by which academics had to be in post to be included in the university's RAE submission.

On joining, he was given immediate unpaid leave of absence to fulfil a pre-existing research commitment at Helsinki School of Economics. He was due to come to Birmingham on 18 August, but he resigned on 25 July.

The university said that Professor Deb had met with the Birmingham's vice-chancellor in May and was "fully committed" to his post.

It added that his appointment had followed a "robust electoral-board system" and that the computer-science submission to the RAE followed the university's RAE code of practice and met the eligibility criteria of the Higher Education Funding Council for England (Hefce).

"All matters relating to the university's RAE submissions and all audit questions raised by Hefce have been fully addressed, as far as the university is aware, to Hefce's satisfaction," it said.

Under the RAE rules, there is no minimum time academics must be in post at a university for their research to be included in the RAE.

One university head of department said that, if boosting the RAE score lay behind the appointment of Professor Deb, it "showed a serious error of judgment" on the part of the department. Sally Hunt, general secretary of the University and College Union, said it was unsurprising that, in general, "underfunded" universities felt they had to "exploit all possible avenues and loopholes in order to keep income levels up".

Professor Deb could not be contacted as Times Higher Education went to press.

zoe.corbyn@tsleducation.com.