Advertisement
Advertisement
Advertisement
Research intelligence - A measure of humanities?
5 November 2009
A European project aims to build a citations approach that works outside the sciences. Zoë Corbyn reports
It is widely recognised that when it comes to measuring performance in the humanities and social sciences, citations - the number of times an academic's work is cited by their peers - work either poorly or not at all.
For a start, the main bibliometric databases that tally up citation counts - Thomson Reuters' Web of Science and Elsevier's Scopus - consist mainly of international journals written in English, with scant regard for regional and non-English titles, although the companies are working on this.
Another problem is that, as opposed to the sciences where journal papers are the norm, many non-science scholars write books instead.
It comes as no surprise that in the forthcoming research excellence framework, arts, humanities and many social science subject panels are not even expected to ask that citation data be made available to help guide assessments. However, a new European project could change all that.
The project, sponsored by four national funding agencies - the UK's Economic and Social Research Council, the French National Research Agency (ANR), the German Research Foundation (DFG) and the Netherlands Organisation for Scientific Research (NWO) - is investigating whether it is possible to develop a database capable of capturing a wider range of research outputs from the humanities and social sciences. It could then be used to determine the academic impact of the work, for example by counting citations.
The group overseeing the project is led by Ben Martin, professor in the Science and Technology Policy Research (SPRU) unit at the University of Sussex. A draft document outlining the idea, Towards a bibliometrics database for the social sciences and humanities - a European scoping project, was presented to scholars for the first time at a meeting in Berlin last month.
The project is a response to governmental pressure for more accountability in research funding, Professor Martin said. But it also aims to provide an alternative to the European Science Foundation's much-loathed European Reference Index for the Humanities (ERIH), which has caused a storm in the academy by categorising journals in three classes (A, B and C) depending on their global significance.
This has created a de facto hierarchy that some institutions and funders are said to be unofficially using to decide promotions and grants.
That the ERIH also encroaches on social science journals appears to be the main reason for the ESRC's desire to develop another approach.
Professor Martin said that funders wanted to explore an alternative to the ERIH and investigate "whether in principle it might be possible to construct a more useful and realistic database".
This would include not only international and regional journal articles, but also books and chapters aimed at both academic and general audiences, "grey literature" such as policy reports, and even non-published output such as performance.
Works included would probably have to be peer-reviewed, which would pose problems given that some top humanities journals rely on editorial judgment, and book publishers and journals could be graded according to quality. The project could be rolled out in phases, with journals first, followed by books and then the other elements.
No easy task
The hurdles to success, however, are huge, and Professor Martin stressed that nothing has been set in motion yet. The group plans to present recommendations to the funders at the end of the year, one of which is likely to call for a pilot to test the scheme's feasibility.
But it is unclear whether the benefits of implementation would outweigh the costs.
Professor Martin said: "It would be perhaps five to ten years before we have usable, reliable results."
Opposition from academics also seems to be brewing. Judi Loach, director of Cardiff University's Graduate School in Humanities, who attended the Berlin meeting on behalf of A-Hug, a group that brings together the UK's learned societies in the arts and humanities, described the proposals as "potentially more dangerous than the ERIH".
"No one in the humanities agrees we need bibliometrics. We don't think it is appropriate to our disciplines," she said.
"The worry is that (the database) will be easier to slide through than the ERIH because the proposals have more academic rigour."
She also lamented the lack of academic consultation, the failure to involve the Arts and Humanities Research Council, and the choice of the SPRU to lead the project.
"(SPRU) has nothing to do with the humanities," she said.
zoe.corbyn@tsleducation.com
www.sussex.ac.uk/Units/spru/esf/.
Readers' comments
My subject is not even listed in the ERIH, so I figure in this alternative universe I get all the citations, all the money and all the kudos. That's nice. Thank you, ESF. Now, yes, good news Professor Martin. Indeed, let's get back to reality ESF!
The ESF *cannot* get back to reality. What it is proposing is yet another audit system for research. However, that project, like all of its kind, is doomed from the beginning, because it is misconceived in its very essence. "Within a *closed* system, audit detects errors and fraud through sampling information and cross-checking it for *inconsistencies* when compared with established organisational and practice criteria [via descriptors, indicators and the like]" (Bruce Charlton). That is, university "auditing has many analogies with financial auditing. But instead of monitoring money flows in a closed system to detect financial fraud, ['performance measurement'] audit samples information in order to monitor compliance to a system." Ultimately this has *nothing* to do with reality - it is rather a matter of: if it doesn't fit *our* descriptors, imposed from above, it doesn't exist. There is no point - on the contrary - in trying to tweak or "improve" such systems. That is not their purpose! The reason why the managerial strata who have now taken control of academic life need to enforce audit is to impose their power over the academics. The managers normally have, by definition, no understanding of content (maths, theology, physical chemistry, medieval history or whatever). So they define "desired" outcomes (like HEFCE’s economic impact) in advance and cast them into "objective" form (a bit like, say, assessing the true quality of a musician's violin-playing by measuring numbers of notes played per minute and maximum decibels produced, plus "satisfaction scores" among the audience) - that is, the managers take their policy cue from their political and business masters, and then impose a scheme to audit the work produced by academics in order to determine whether there are inconsistencies (see above) between those "desired" outcomes and research results. The ESF scheme is nothing other than a new technical twist - in this case a "bibliometrics database" purportedly adapted to the humanities - in the quest for audit instruments to facilitate this ultra-reactionary goal (and to sell these instruments as "new and improved" and in this case even "European" ... presumably in the hope that researchers will "buy the story" - and therefore at last accept audit vassalage as such!). That is why the sub-title to the article is so dangerously ambiguous: "A European project aims to build a citations approach that works outside the sciences." Did you say: "works"? In fact, if it "works", it is likely to be a bigger catastrophe than what preceded it.