Nobel laureates join fight against impact-based research funding

I came to this country to devote a large part of my life to do basic research in Astrophysics. After 13 years, considering the management of the new council (STFC), I see no other option than leave this country, as many of my colleagues have already done.

I completely agree with Ramakrishnan, Kroto, Walker, Hunt, Roberts, Josephson and others and will be signing the petition. The science that has had the biggest long-term impact in the UK and the rest of the world has been basic research where at the outset of the research project the potential impact of the work was unclear. There are so many examples of this it is incontrovertible. This type of initiative will mean some of the countries best scientists will not get funded over lesser science that has some apparent mmediate impact. Short-term and long-term British science will suffer. Expect another Brain Drain soon.

I've no problem with this and neither should anyone who has done good quality work. The underlying science (outstanding papers) will make up at least 60% of the profile. Environment (funding, postgrads, esteem) will probably end up somewhere between 15-25%. So somewhere between 25-15% of an REF profile will go to demonstrating that the research described was in someway valuable. I've no problem with being accountable to the people who fund my research in this way. It's long overdue.

@Carl May: You spectacularly miss the point. Define "valuable". Valuable to whom? Valuable on what time scale? You also use the argument trotted out with tedious regularity by the research councils: those who criticise the impact agenda are of course ivory tower academics who do not want to be made accountable to the taxpayer. There's insufficient room here to discuss the gaping holes in this frustrating - and, frankly, insulting - argument. I recommend that you visit "http://www.csc.liv.ac.uk/~leslie/impact/impact.html" to read the counter arguments. (P.S. Over 2000 academics have now signed up to the UCU petition; nearly 2000 have signed up to a similar ARHC-related petition at "http://petitions.number10.gov.uk/REFandimpact/").

@Philip Moriarty I haven't miss the point at all. I've merely written something you don't agree with. As a profession we were pretty quick to criticize the RAE because focusing on papers and grants didn't take account of the wider impacts of our work. Now we have an evaluation exercise that does exactly that, and a small part of the REF for any unit of assessment will be about impact case studies. I just don't think it is a big deal to be asked to do this.

@Carl May. Thanks for responding - the more debate we can have about this, the better. I don't know which discipline/research area you work in but I'm a physicist. It is both scientifically and *economically* entirely flawed to expect academic science to deliver economic impact on the time scales proposed in the REF "consultation" document.The link I provided in my comment above lists a set of papers which describe the key arguments in some detail (and also includes a link to the responses of UK unversities to the 2006/2007 RCUK "consultation" (...cough...) on the incorporation of economic impact criteria into peer review. These make for interesting reading). HEFCE and RCUK are attempting to radically alter the ethos of university research, abandoning core academic principles in the process. That mightn't be a "big deal" for you. It clearly *is* a big deal for the 2,162 people who have signed the UCU petition to date, the 1989 (to date) who have signed the petition by James Ladyman at the Number10.gov website, and the 2294 who signed the petition "to promote discovery in UK science". Best wishes, Philip. [P.S. A plea to the THE editors: Please, please enable line breaks and HTML tags for online comments!]

I overdo the use of ironic quotes around the word "consultation" in the comment above. Apologies for the sledgehammer sarcasm. Nevertheless, the recent Cambridge review of primary education gets it spot on when it describes what it calls "the empty rituals of consultation". RCUK ignored very negative feedback from UK universities stemming from its consultation on peer review in 2006/2007. Why should we think that the current HEFCE REF consultation - for which many academics are investing a large amount of time in preparing responses - will treat negative feedback any differently? (The Cambridge review led, of course, to a sea-change in the Government's thinking on primary school education. Ooops, there goes that sledgehammer again...)

We should be careful not to throw out the baby with the bath water. Impact analysis as a principle for allocating research funds is respectable. Choosing the proper measures of impact, however, is critical as the exchange of views between May and Moriarty show. We should get the measures right before we jam this impact approach down everyone's throat. For instance, in social science, the proper way to assess impact could be greater efficiency or greater effectiveness or both in some piece of research. Other criteria might be more suitable elsewhere such as physics. But uppermost in our minds must be our duty to the taxpayer. We must accept the fact we need to allocate dollars (or pounds or euros) in some rational way. We should not do it simply on the basis of a person's "reputation" (which may discriminate unfairly against new blood) or number of articles (uninterpretable and likely to discriminate against women and foreign born scholars).

Every country is complaining that top scientists are leaving. Although no one knows where exactly they are going, the reasons behind the movement are clear. In the first place it is money, a lot of it. Then, stands tremendous growth of journal publications. Such factors as ten fold increase in governmental concern about science and growth of universities are also important. These new trends have prompted the governments to pay more attention to science and increase funding.

@Philip Moriarty First of all I stand on the general principle that it's not an unreasonable requirement for me or you to have to show that our work has some impact or value. I was an early signatory of the Ladyman petition because it seemed to me that 'economic' impact was a short sighted model for some of the reasons that you and others suggest. Fortunately, economists won't be judging the impact cases studies that departments submit. It will be members of the same panel that evaluate the underlying science and the question of environment/esteem. My guess is that they will do so with the same attention to context and constraints that they give to published works and grant income. The deal here is that accountability is to your peers, not to the taxpayer. Departments will have to choose their impact case studies very carefully, and selectivity will be the order of the day. RAE and REF are exercises that do the job of concentrating money and as Robert L Fisher points out above, we do actually need mechanisms to do that. The more that we focus attention on the REF, of course, the less we consider the underlying problems. The nationalization of university teaching and research under new labour, the assault on institutional autonomy, the degradation of the professoriate, and the corporatization of university life are all evidence of a much more fundamental set of threats to the academy.

It's really interesting to see a debate between two internationally acknowledged and distinguished scholars - Philip Moriarty in physics and Carl May in social medicine. It's also really unusual to see people using their real names on the THES website . @Carl May you are right about the principle of accountability. @Philip Moriarty you are right that accoutnability needs to be contextualised properly. But let's not forget that by the time we actually get to the REF we are going to be in a very different world. Public spending cuts will have decimated research across the HE sector, there will almost certainly be a conservative government, and some universities will have simply dropped research altogether as they seek to survive.

We are talking about the Nobel prize winners signing the document, although I have very little time for the UCU which is useless. The impact value of research by scientists like Venkatraman in whatever measure that one can use is well established in a number of areas if one knows about his research work. Fisher's argument is hence is just an argument in its own sake. It will be really a loss to Camridge and the UK if Venkatraman decides to move to universities like Harvard.

Most Brit lecturers and professors aren't nobel prize winners, aren't that good, and aren't going anywhere. They'll have to live with it.

To be of any use at all, the people who assess potential impact will need to be capable of forecasting at above chance levels which projects will and won't have positive economic impact. How will their ability to do this be determined? The research evidence on economic and political forecasting shows that people generally, and experts specifically, are pretty dismal at forecasting, especially for the longer-term future. Their lack of accuracy goes hand in hand with overconfidence about their ability. Exemplary research on this topic is given in Philip Tetlock's book "Expert political judgment: How good is it? How can we know"? --- incidentally,a book with a dust jacket recommendation from a Nobel prize winner. Just this week it has been reported that the UK economy is still in recession, despite the average forecast of City economists that we would now be in a period of growth. To quote Ashley Seager in the Guardian (Friday 23rd Oct): "The average forecast from economists in the Square Mile was that the economy had expanded by 0.2% in the July to September period. But in fact, according to the ONS's preliminary figures, it contracted by 0.4% - the sixth drop in a row. That's a big forecasting error, really big". So I wonder, where is the evidence that anyone is capable of accurately predicting which research projects will lead to positive economic impact, especially when those impacts are further into the future?

@Carl May. Thanks for your considered response. You state that "... it's not an unreasonable requirement for me or you to have to show that our work has some impact or value." OK. Let's cut to the crux of the matter. Of the following examples, which has more impact and, importantly, which provides more value to the taxpayer and is ultimately of more benefit to mankind? (A) A university-industry partnership involving Procter and Gamble (or BAE Systems or any other multinational corporation of your choice) which provides large increases in efficiency of a commercial process and which has the potential to lead to a strong increase in market share for that company; or (B) Fundamental research into an (apparently) esoteric aspect of quantum mechanics/cosmology/particle physics which has the potential to provide key insights into the nature of the "fabric" of the Universe. (contd...)

..contd. @Carl May. Leaving aside the fact that there are countless examples of apparently esoteric research having a huge commercial spin-off (MRI; giant magnetoresistance; fibre-optics; the laser was considered "a solution looking for a problem" for a long time...), there are fundamental difficulties with the impact agenda which are *equally* as important as the "underlying problems" you list. You may be familiar with the work of John Ziman, a condensed matter physicist-turned-sociologist. Ziman pointed out the key importance of **disinterestedness** in academic research. If academic scientists are being driven - directly or indirectly - to consider the "bottom line" then that ultimately erodes taxpayers' trust in the academy. The impact agenda is a cornerstone of the creeping corporatisation of academia which you quite rightly criticise.

@Carl May. (Finally for now!). You state " It will be members of the same panel that evaluate the underlying science and the question of environment/esteem. My guess is that they will do so with the same attention to context and constraints that they give to published works and grant income." You have a great deal more confidence in the reliability/robustness of the REF impact assessment process than I have! On the UCU petition website there's a link to a letter to HEFCE's Chief Executive which points out that only three of the fourteen members of the REF Impact Pilot Exercise Steering Group are from a higher education institution. Best wishes, Philip.

@Philip Moriarty - also finally. Neither of us are likely to change our views on the fundamental question of how public accountability should be framed. Although I'm not impressed by the RAE or REF, and would rather devote my time to other things, I just don't think we live in a world where we have an automatic claim on resources. Without labouring the point too much, I don't and I never have. I'm a social scientist who works in medical health research -- I do both basic and applied research -- and my work is funded by a mix of medical research charities (e.g. Diabetes UK, Arthritis Research Campaign), research councils (ESRC, MRC, EPSRC), and government agencies in the UK (NIHR, Scottish CSO) and overseas (NIH, SSRC). All of these expect me to be able to communicate the importance and utility of my work in clearly to non-experts. Some of them expect this because they, in turn, need to persuade people that it is worth standing on the street rattling collecting tins in the rain to bring the money in to fund research. Others expect me to do it because they need to enrol the support of citizens as well as policy-makers in securing consent to spend money on research when their are other important and competing claims on the public purse. All expect me to be able to explain the importance and utility of my work in lay terms so that their scientific committees can make reasonable decisions about how to distribute funds between large numbers of competing - and often superbly designed and thought through - studies. Of course, it's not just the people who fund my work to whom I am accountable, but the participants in my studies - patients and professionals - need to have confidence that I am asking them to expend their time and resources for good reason. Big corporations rarely come into it, not because they have a bottom line, but because they often seek to protect it by restricting what people like me can say and publish. So perhaps I live in a professional world that is characterised by massive hyper-accountability, where statements about impact and value are so ubiquitous and normalized that I think they are no big deal because I do them every day. I don't think that this means that I have seemed disrespectful or dismissive of your views, but if it looks that way then I am very sorry, because it seems to me that you and I share some common commitments to the integrity of the sciences and humanities and to the importance of defending them against bute politics. [[NEW PARAGRAPH]] I'm likely to be REF co-ordinator for my Institute and so I will have to write the case studies that will show the wider impact of my and my colleagues' work - so I have been thinking about this a lot. In the end these case studies will account for between 15% to 25% of our research profile. We have five years to select impact case studies and to make sure we can continue to demonstrate impact. During this period we will all have to think hard about how to play the game to maximize advantage. One of the things that bothers me less and less each time I go through the RAE, is the robustness and reliability of the procedures involved. They are what they are. But most importantly, the panels are made up of people like us, which means that their judgements about the underlying science, esteem/environment and impact, will be those of practitioners of their disciplines. Some of my research has been about how professionals and policy-makers evaluate and employ evidence to underpin clinical practice. Even when the underlying politics of evidence is irrational (and sometimes wrong-headed) they seem to struggle to be fair and sensible, and to make reasonable judgements on the basis of the evidence that's put to them. That they do so in conditions of constraint is undeniable, and it's also undeniable that they work against a background of rules and normative expectations that represent political interests. My observation of RAE panels is that their members seem to try to work in a similar way, and it's interesting that groups assessing evidence in situations characterised by a lack of transparency often seem to work even harder to do the right thing. I don't believe that we live in anywhere near the best of all possible worlds, but I also don't believe that we live in the worst. On second reading this is a bit of a sermon. It's not intended to read that way! Best wishes to you too. Carl

I'm afraid that Carl May's comments really ought to be read as a kind of symptom of the odd and confused thoughts provoked when systems of institutional destruction are imposed upon (in this case university) organizations - thoughts based however upon a false premise: that the need for an RAE or an REF or something similar (even if "better framed") follows from the fact that "we live in a world where we [do not] have an automatic claim on resources". That's not the point at all! The point is that *all* these audit exercises are fake from alpha to omega. Bruce Charlton, Michael Power and many others have demonstrated time and again that the "dishonesty" at the heart of such systems has "left a legacy of institutionalised lies". So the public is not being informed by these audit exercises but entirely misled. Thus they cannot be any legitimate basis for distributing resources. On the contrary, they exist for one principal reason: because (Charlton) they allow the managers to take power from the professionals (the academics), since the managers ultimately control the audit procedures. The managers, in turn, are (willingly or not, it doesn't make any difference) serving their masters: the profit-makers. The notion that any kind of auditing procedure is connected to us "being accountable to the people who fund [our] research” is massively naive. "The conflation of 'accountability' with 'responsibility' ... has arisen because modern managers equate a proper organization with an auditable organization. Organizations must be made auditable - at any price. Traditional 'collegial' organizations of professionals - such as schools, universities, or hospitals - are regarded as intrinsically flawed - since managers do know and cannot control what is going on in them. But by its concentration upon systems and procedures, [audit - like RAE or REF] provides a mechanism for quantifying almost any aspect of organizational performance that can be given a name - 'quality', 'equity', 'access' or whatever the current buzz word happens to be". And some people, even academic colleagues, really believe the hype! "This wildly improbable belief is [now] the norm" (Charlton again). Theodor W. Adorno said it: capitalism is barbarism. It is now destroying academia too.

@ernest smith. Read between the lines. (A) It doesn't to me as though Carl May is odd or confused. It looks to me as though he is a realist who understands how research funding works. If you google him you will discover that he's one of the most successful public health sociologists in history. It looks to me as if he understands the rules of the game and really does know what he's talking about. (B) Not only does it look like he understands what research funding is about, he also understands how research assessments work and isn't afraid of them. I wish he was going to do the REF for our school. If he did, it wouldn't matter to me that the REF was a 'sham' or that capitlism = barbarism, I'd be sure that we'd get a result. (C) He used his own name and he wrote respectfully about his opponent's views. Definitely not normal for THES website. Love it! Come to our place Carl, sort it out for us.

One has great respect for Carl May and confidence that he would, if he had the chance, do his best to take the sting out of the effect of the REF on particular departments. And that is just what I was talking about: the difference between the fate of particular departments - the symptoms of the disease - and an analysis of what the rise of the audit culture as such reveals, an analysis which requires more than good will or a knowledge of the details of how a given avatar of audititis works. Having already largely lost the battle against the transformation of their universities into skills factories, researchers are indeed often clinging on to their remaining ramparts, taking the position: "It wouldn't matter to me if capitalism = barbarism, I'd be sure that *we'd* get a result". So the new system has won: "we" are now looking at our own navels, fighting - while all around us crumbles - to keep our own local departments or projects going for perhaps another five years, by - as one commentator on a similar piece put it recently - spouting the kind of bullshit which the visitation demands. That may be a sensible, if desperate, tactic in the short-term. But it also contributes towards long-term disaster. So, since Carl May is indeed one of the most successful public health sociologists, I'll cite just once more one of the most successful evolutionary psychiatrists, Bruce Charlton, from the Oxford Magazine, 2009, entitled "A culture of corruption in academia": "There is a big question over whether academia has within itself sufficient resolution to nourish individual academics in their difficult task of devotion to truth. What we need is moral courage. But there is a severe and chronic shortage of this commodity in modern British universities. I suspect that the secular and this-worldly Zeitgeist of the modern university operates on such a here-and-now, worldly, pragmatic and utilitarian ethical basis as utterly to lack moral resources for the job I have in mind... So will it happen? – will there be a Great Awakening to truth in academia? Frankly, I doubt it; and we will probably continue to see the world of scholarship degenerate towards being merely a mask for the pursuit of other interests."

@SteveK: I very much share your respect for Carl May's approach to online debate - his responses to my comments above, despite our differences in opinion, have certainly been thought-provoking. (I also suspect that Carl and I may well have rather more in common ideologically than initially appears to be the case from the exchange above). Unfortunately, however, your comment (B) above neatly sums up a major problem at the core of academia today: "If he did, it wouldn't matter to me that the REF was a "sham"...I'd be sure that we'd get a result". In other words, it doesn't matter if a process/procedure/system is entirely flawed and/or dishonest - what matters is that we "play the game" as well as we can. At a recent debate at the British Library ("Don and Dusted?") Claire Fox of the Institute of Ideas asked why the academy is now so supine. Your pithy comment goes a long way to answering this question. The idea that universities/academics should "speak truth unto power" is now depressingly seen as an anachronism.

I submitted my comment only to find that Ernest Smith (preceding comment) had posted similar sentiments while I was writing!

@Steve K. You seem to have united us all! It matters very much if the RAE is a sham. I don't like it, I can think of lots of reasons why it is a bad idea, and so forth. I would rather not do it. Further, as both Philip and I, and others above, have argued above, there are some tendencies in higher education policy over the past two decades that have led to the effective nationalisation of the whole sector and its subordination as a a policy instrument. This has had all kinds of bad outcomes for us as a profession. But that's not what this debate was about - it was about whether impact case studies were a useful addition to the REF, and then, whether making clear the value of science, humanities, and arts was a good thing. I have argued yes to both. But there is another reason that some of the key battles that Ernest Smith describes have been lost. This is that some of our colleagues have believed that it was beneath their dignity to communicate the value of their work to others, or even to speak in a language that their fellow citizens can understand. It must be understood, Steven K, that the value of our work is not always self-evident. Responding to this is nothing to do with moral corruption, or cowardly subordination to managerialism, or a supine profession. It's about fully participating, as citizens, in a democracy. End of sermon. PS I'm not the most successful public health sociologist in history. That's a silly thing to say.

Philip this link is dead http://www.csc.liv.ac.uk/~leslie/impact/impact.html. If you have another that takes us to the counter arguments, would you mind posting it please? I work in English which will have a much harder time demonstrating its social and economic impact than a science subject. It is possible to show how some work affects the English community and it is possible to show how that work filters into teaching at school and university level but who can say that that is a 'benefit'? We use the word research in English but in fact we are concerned with an exploration of values and any democracy that seeks to limit such exploration by imposing easily measurable outcomes on subjects like mine is betraying its own nature. Also, I don't buy the idea about being accountable to the taxpayer-at least not in any simplistic sense. I am a taxpayer, and I wishe the government would spend my money on schools not weapons, on helping commiunities not bankers. But I am only one voice. And each taxpayer has their own voice. So how can anyone presume to speak for all? We have seen where short termism has led in business and now we are being told to apply that model to our work. Crazy. REF is not about being accountable, it's about being controlled. We are being set up. To disagree looks as if we do not want to be accountable, as if we do not want transparency (neither does the government in relation to its own activities but let that pass ) when in fact we are trying to preserve the integrity of our own disciplines and the freedom to think and explore. Democracy as well as universities will be the poorer if we all have to demonstrate 'impact'.

@Gary Day the one thing that my three years as an English undergraduate taught me was that 'research' meant that students and teaching weren't important. Most of my tutorials and lectures were done by the most junior lecturers and PhD students. The profs were all off doing their 'research'. Do I have a better grasp on poetry and literature because of it. No. You can't say it was a benefit because it wasn't. No-one looked too closely in the emperors wardrobe, and no-one asked if it was worth the money. So as a former English student I'm glad that you have to justify yourselves. Some of you can't demonstrate 'impact' because you've had none. You thought you could go off and do whatever you pleased at the expense of people like me. I have had to pay for this out of a student loan and income tax. Now you have to show it was worth it.

I would have to agree with Carl May's comment that a significant problem in academia is the failure of academics to communicate the value of their research. A few months ago I observed a publications workshop at a prominent UK university. It was attended by late-stage PhD students, and one exercise was for them to talk for three minutes on the subject of their research, what made it unique, and why it would be of interest to others. Now, I believe that at least some of them were doing worthwhile research, but you wouldn't have known it from what they were saying. The problem was that, over a number of years of study, none of them had ever been asked to explain their work and its value to anyone outside of their field (and, I suspect for quite a few of them, not even that). This is, of course, a single anecdotal account, but I suspect that it is broadly reprsentative of the consideration that many academics have given to engaging with an audience wider than their own small subject area.

Shelley I

Shelley I appreciate that you may have been served badly but that is not the point at issue here, or at least not directly. As it happens, I agree with you that there can be a problem in some institutions where teaching is left to hard-pressed postgraduates while professors pursue their own research. But that assumes the professors would teach you better, which may not always be the case. I think things could be organised better but the problem is that funding depends partly on research (more so than when i was an undergraduate) and so there is a real drive towards research, sometimes, maybe often, at the expense of teaching when, at the same time there are more students going to university who need more intensive teaching. And because students are 'customers' they demand the best value for money. The situation is very complex and there are many competing demands not to mention an ongoing transformation of the culture of HE. But the point of this discussion, as I understand it, is whether research can be judged in terms of its social and economic impact. My point is that some subjects lend themselves more readily to this than others though even the ones that do will suffer as a result. Those that can't demonstrate said impact may disappear. And if you think that researchers should be accountable in this way then you are helping to perpetuate the very thing you complain of; that academics care more about their research than their students.

@Gary Williams. Your email makes an excellent case for tight managerial control of universities and lecturers. I'll give you the benefit of the doubt. Tell me about some scholarship done in the last five years in your field, English, that really has added 'to the common wealth'. Define it how you like. Just point me to the articles, I can find copies and read them. or even better, define it in Philip Morairty's terms, "better understanding the fabric of the universe." Again, define universe how you like. But DON'T YOU DARE try to imply that it's all just too difficult for us girls to grasp. Complicated indeed!

First my name isn't Williams, second I have neither said nor implied that 'girls' to use your term, find the current state of HE difficult to grasp. And my point was that it's very difficult to demonstrate how English makes a social or economic impact. since it is concerned with the exploration of values: it raises questions about life, society and ultimate ends. All airy fairy no doubt but a society in which there is no space to ask such questions is, in my opinion, the poorer.

You can't think of one, can you?

leavis influenced the teaching of English in schools for at least a geneneration, (he wasn't a researcher), literary theory transformed the subject at university and now school level, all the work done on women's writing has transformed the teaching of literature as has research on post-colonial writing. I could go on.

You're really not selling it, Gary. So, apparently the best thing that can be said about research in English, is that it has an effect upon how Engllish is taught as a subject. And why exactly should we care...?

I am not trying to 'sell' English. My point is that it can't be packaged. If you want intellectual life to be modelled on business principles then I suggest you take a more critical look at business principles. Yes business delivers some good things but it has a dark side: exploitation, pollution, damage to the environment, inequality, treating people as units of production, etc.

Gary, Shelley asked you as to what research in your field added to the common wealth, and you were unable to do so. It's nothing to do with modelling life on business principles, but showing that the study of English has some value and isn't just well-paid navel-gazing. I think that your vapid and unhelpful comments just show the lack of worth of your subject.

Interesting debate. Similar moves afoot in Australia. An earlier audit scheme was cancelled with the end of the Howard government, but is now back in a different form (the ERA). It is deeply flawed because it cannot recognise a seminal piece of work in a poorly ranked journal. As usual, Australia is mimicing schemes developed elsewhere. {}It may be helpful to separate measurement of 'research' impact of academics, which is something rather difficult to quantify, from 'intent'. SOme research is borne of a desire to unravel a puzzle. Other studies try to counter previous ones, or build upon them. Yet others are because the author has something they really want to say. SOme research is borne of pure ambition - to get famous or to get promoted in your field.{} What should be recognised, in my view,is the avoidance of ambition and self-promotion. When working in that mode, research is less about getting in the top journals, being well published, and winning accolades and advancement. It is about responding to the needs of OTHERS, not yourself. I do not care if this is proximate, distant or virtual others - and 'others' can be non-human too. I am a little influenced by Peter Singer here, who would take a utilitarian line on impact, I suspect, possibly looking for utility maximization from academic lives? {} PS on this basis, the most useful thing I did while being a university academic was set up bicycle campaigns and research on improving bicycle use in London. The paper in Science takes a second place!

[As the THE online comments section doesn't allow line breaks/paragraphs (for perhaps obvious reasons!), I'm going to post a number of separate comments in response to some of those above]. Carl, you argue that demonstrating the impact/importance of our work to the wider public is effectively an imperative given that we are publicly funded. I have a lot of time for this argument but with a number of important qualifications. Yes, I'm a publicly funded scientist and therefore I am duty bound to carry out research for the public good. The key (and thorny) question is, of course, what is "for the public good"? One can look at this from both an "egalitarian" perspective or, importantly, from an economic perspective. Is driving scientists to do near-market R&D - as opposed to curiosity-driven, far-from-market research - in the public good? (And no matter how much RCUK and HEFCE protest otherwise, this is a key driver underpinning the impact "agenda"). I haven't room here to put forward the arguments in detail - see my comment below to Gary Day for links to relevant papers. In addition, science unfortunately does not proceed via a democratic process...

Gary, apologies for that dead link. I don't quite know why it's suddenly no longer online - I'll chase this up and post a revised link if possible. In the meantime, my particular perspective on the impact issue is described in, for example, an article entitled "The Economic Impact Fallacy" in the June 2009 issue of Physics World and in Nature Nanotech. 3 60 (2008) ("Reclaiming academic from post-academia"). However, these pale against Sheldon Lee Glashow's inspiring speech "Can Science Save the World" (http://www.hypothesis.it/nobel/nobel99/eng/pro/pro_2_1_1.htm) and an excellent paper published by the League of European Research Universities last year and entitled "What are universities for?" [http://www.leru.org/file.php?type=download&id=1323]. There's also lots of discussion and debate at the Prometheus blog ("UK Criterion 2?"; http://sciencepolicy.colorado.edu/prometheus/uk-criterion-2-4955) and at David Colquhoun's excellent "Improbable Science" blog [http://www.dcscience.net/?p=1108]. You'll find good alignment between the arguments you put forward in your comment and those detailed in the various articles/online sources listed above. Best wishes, Philip.

@Gary/Unimpressed/Shelley: Hang on a minute. Research in English isn't just about helping people to know how to go about teaching English. It is and has always been about gaining a better understanding of how language works (or doesn't) in documents (texts, if you like) which typically demonstrate the deployment of language at a particularly demanding level. Research into (= understanding of) how this works and the effects it creates is of immediate importance to anyone who uses the English language. The investigation of howlanguage functions also develops our awareness of how language can be manipulated and misused -- and it thereby helps to protect civil society (more broadly conceived) against (say) the limited world-view of the "business model" and its shrewd but partisan rhetoric. Incidentally, I'm not an English specialist: I just struggle with how to make sense of medieval French.

As someone with a keen interest in public engagement and outreach, I can certainly appreciate your argument regarding communication with those outside one's research area. But why should the ability to communicate to a broader audience necessarily be used as an indicator/discriminator of the quality of the research? If, for example, a mathematician/physicist develops an entirely new and ground-breaking interpretation of quantum mechanics, why should the quality/ importance/impact of that research be dependent on the researcher's ability to communicate the concepts to those with no training in maths,physics, or science? (Indeed, the concepts, at least initially, may be outside the grasp of only a few experts in the area).

Oops. Final line of the the previous comment should be "Indeed, the concepts, at least initially, may well be outside the grasp of all but a few experts in the area". Apologies.

I think you are missing my ponit. The ability to communicate to a broader audience isn't an indicator of the quality of ressearch, but remains an essential function of academia. You shouldn't need to have detailed understanding of a particular field to understand the reasons why particular work might be carried out in that field. For instance, I have a couple of friends who are geneticists and, occasionally, they will explain to me what they are currently working on, and why it is of interest. And, I can understand these explanations, despite having given up science at GCSE level. Now, this isn't universally true, and it may not be possible to explain to the educated layperson the ramifications of the latest research in complex mathematics, but I believe that this is the exception rather than the norm. Ultimately, encouraging public engagement with academic research is a good thing, likely to engender more public support for academics and their work.

"Impact" (like the descriptors misleadingly called "quality" and "excellence") is NOT used by audit procedures like the REF in anything like the ordinary sense of the word: it is a special instrument for further shoring up the quantitative control by the (non-scientific) stratum of administrators (and behind them, government and big business) over academe. Its introduction into the auditing procedures for the universities should be discussed in these terms, not in relation to the quite different question of whether scientists (or scholars) should try to communicate their results or thoughts to the general public and so on. The latter has nothing to do with the managerialist strategy, except sometimes as one of its fake legitimations. As one blogger put it: '"Impact" is a new element and its introduction is in line with the government’s agenda to pump research funds into things which will generate wealth' - that is to say, in most cases, increased private profit. It is one important element in a sustained campaign to transform the very essence of the university, to rid it of its "anomalous" autonomy - anomalous in a one-dimensional world in which all values are being reduced to a single, quantifiable economic criterion. There isn't really much more to say than that. The rest is just the obviously complex detail of how this assault is to be carried through in particular places and cases.

And to quote: Jules Janick writes: "The impact factor concept has been bought, hook-line-and-sinker, by administrators worldwide .... Sadly, the impact factor can be gamed" - and of course it is and will be, en masse, turning university research exercises into a cynical exercise in manipulated number crunching: at institutions once though to be dedicated to the quest for truth! But generally the result will presumably be as illustrated by the letter Janick reproduces: * To: Abbot Franz Cyrill Napp, Augustinian Monastery of Brno - From: Bishop A.E. Schaffgotsche, Brno - Re: Gregor Mendel -February 1, 1868 - The Archbishop of Prague has determined that monastery funds for the construction of a greenhouse have been used to support a research project concerning peas of Gregor Mendel, a member of your order, that may reflect on the effects of the study of science on the spiritual calling of the monastery. As a consequence, we have opened up an investigation to determine the *value and impact* of this research in two ways: peer review and a citation evaluation. We sent a paper entitled Versuche über Pflanzen-Hybriden (Experiments on Plant Hybrids) published in Verhandlungen des naturforschenden Vereines in Brünn 4:3-47, 1866 to the eminent Botany Professor Karl Wilhelm von Nägeli, who informed us that he had already received correspondence with Mendel about this topic. Professor Nägeli was unimpressed with the research but admitted he could not spare the time to read the entire document. He had suggested that Mendel should cease working with horticultural crops and investigate Hieracium (hawkweed), a truly botanical species. We further tested the significance of the pea work through a two year citation analysis and found that the impact factor (derived from number of citations of the paper) had a value of zero. It has never been cited at all. We conclude that the lack of citations confirms the opinion of Professor Nägeli. In view of the poor review and low impact statement we suggest that Dr. Mendel ceases all research in this area. We strongly urge Father Mendel to find a better use of his time and we suggest administration.

Thanks for that link Philip and the other references. Very useful

For every Mendel there is a Newton or a Hubble, whose science is seen to revolutionise the world and whose work is celebrated straight away. There are also many thousands of lecturers and professors who do good work that never sets the world on fire, but which by its very accumulation makes the advance of higher learning possible. My partner is one of them. Like some of the correspondents here, my partner accepts that it is important to communicate the meaning and value of one's work. But like Gary Day, she also fears that the impact of her work is hard to state, and harder still to measure. Underneath the fragile carapace of all this protest, there is - I think - an enormous well of doubt. The sad truth is that Shelley's assertions (they don't deserve to be called arguments) carry some weight. Not all disciplines are equal, and not all researchers within them are equal either. Over the past thirty years I have seen at first hand the explosion of research that cannot be justified on academic grounds, let alone on grounds of public value. An endless succession of manuscripts pass through our study in which a glance at the abstract alone is sufficient to tell that the work reported is trivial or useless, or simply incomprehensible. I have tremendous sympathy with students' like Shelley who have been denied the chance to learn from subject experts in the service of an enterprise that seems to have accomplished only the destruction of trees. I thought Dr Day's responses were very weak. Moriarty and May both seem to be decent men who care very much for their work and colleagues. I salute them both.

@Gary Day: The dead link you pointed out in a comment above - and which was certainly dead yesterday - has now sprung back to life. The website is entitled "The Danger of Assessing Research by Economic Impact" and was put together by Leslie Ann Goldberg (Liverpool). The URL is http://www.csc.liv.ac.uk/~leslie/impact/impact.html . Best wishes, Philip.

It would be interesting to know what economists have to say about this. Does anyone know?

I don't mind if people disagree with me but I find it frustrating if they, simply assert that what I am saying is 'weak', 'vapid' 'foolish' etc. while hiding behind a mask of anonymity. My points are first that teaching English does not lend itself to being measured in terms of its social and economic impact. But it does not follow from that the subject is worthless. If it was, why would so many students want to study it? I can't speak for others but I teach English because it introduces students to writers and their ideas, because it introduces them to aspects of culture and history with which they are not familiar, because it helps them to think about the strengths and limitations of tradition, because it helps them to understand others' point of view, because it helps them to appreciate the intricacy and complexity of a work of art, because it helps them develop their own voice, because it teaches them to be scrupulous and vigilant in the use of words, because it is a medium for exploring more general questions about themselves and their society and so on. The fact that this list is by no means exhaustive and that others would have very different arguments for studying literature shows that it cannot be reduced to some measurable outcome. Discovering that the sun has rain may never have any social or economic impact but there is a joy and wonder in learning about the solar system that is its own justification.

Thank you again Philip. Much appreciated.

@Berg. You state above: "It would be interesting to know what economists have to say about this. Does anyone know?" The following is a quote from Andrew Oswald (University of Warwick), a leading professor of economics: " I do not know a single economics professor who supports the view from the research councils that research funding proposals should be evaluated by their economic impact. " [http://www.independent.co.uk/news/education/higher/andrew-oswald-we-should-charge-more-in-tuition-fees-1767607.html] ---------- (New paragraph) ---------. It's hardly surprising, however, that economists have no confidence in RCUK/HEFCE/BIS's economic impact agenda. The imposition of economic impact criteria in peer review and research assessment stems not from sound economic arguments but from political ideology. --------- [I would also recommend searching for the work of Richard R Nelson, an economist at Columbia. I'm a physicist with zero training/background in economics and I found Nelson's papers to be extremely readable and models of clarity. See, for example, "The market economy and the scientific commons", Research Policy 33 455 (2004)].

As someone who agonised over whether to pursue an English or Physics degree at university - I went to school in Ireland and therefore took Leaving Certificate, rather than A-level, examinations -I very much enjoyed reading your comments on the value of studying/teaching English at degree level. Although I'm now a physicist, your reasoning "resonates" with my motivations for studying/teaching/doing research in physics. There is indeed, as you say, "a joy and wonder in learning about the solar system that is its own justification". However, academic scientists have now got so used to "playing the game" that we have avoided putting forward these important cultural arguments. -------------------- If I can quote again from the interview with Andrew Oswald which I link to in the previous comment, "At the moment, a lot of people in universities appear willing to say silly things because they are trendy and therefore will get us short-term approbation. That is not a safe strategy, because universities are in the knowledge business, and if people do not wish to hear that, then we just have to keep explaining calmly why we are, and why we are, and why we are". Best wishes, Philip.

I find myself in two minds about this. In truth, much of the comment on this thread sounds like a lot of academics with chips on their shoulders enjoying having a go. That is not to say that Impact per se is a good thing. The problem is that it lends itself rather well to some subjects which are, 'close to the market,' (in the terms of the consultation). Engineering perhaps or medicine. Something like pure mathematics or English will however have an impact that is essentially academic. The Consultation does suggest that this will be accounted for, but I simply can not see any way of dis aggregating the contribution of one discipline from that of another. Taken to the extreme, should (say) mathematicians go and do engineering research? I see no reason why funding should not at some level be accounted for. Indeed, surely this is something that universities should be looking at anyway? So yes - there are real difficulties in impact and ones that look, to my mind, likely to discriminate against some disciplines. But, 'I'm an academic and I don't like this and I'm feeling undermined,' is not really much of an argument.

Apologies. 'I see no reason why funding should not at some level be accounted for' Should read 'I see no reason why IMPACTS should not at some level be accounted for.'

Irrespective of discipline, it does seem to me that this all hinges on whether the measurement is retrospective or prospective. The research councils have almost inevitably gone for a cautiously prospective approach -- they have to because they're funding up front, into the future. But here, the AHRC allows a fair amount of latitude in what impact might be. But the REF looks to be heading for pseudo-retrospective: pseudo-, because the time-scales just arent't long enough for anything except the most applied research to show up. I'm not sure actually that the REF consultation really allows for academic impact. "Cultural" seems also to have been elided as the document advances.

David Trotter - The consultation specifically excludes academic impact. You are correct, I think, about timescales. In some disciplines the impact of research will be quick - medicine I suppose. In something like maths the timescales are decades. In any case, why should current research funding be determined by what a department was doing a decade ago, which seems to be the timescale set out. Having said this, I do struggle with the idea that impact is somehow not a legitimate concern. It is.

@Dara: "In truth, much of the comment on this thread sounds like a lot of academics with chips on their shoulders enjoying having a go". That rather misrepresents the debate and discussion above. It appears that arguments based on the societal importance of curiosity-driven/exploratory research don't sway you, so perhaps you might take time to consider the economic arguments (rather than taking statements from the research councils/HEFCE at face value). There has been a huge amount of debate and literature in the US on the impact of the Bahy-Dole act of 1980 and its effects on academic research and the University-Industry complex. Much of this literature is relevant to the current drive by RCUK/HEFCE to impose economic impact criteria in peer review and research assessment. I recommend that you read, for example, "Universities Inc. : The Corporate Corruption of Higher Education", Jennifer Washburn; "Science for Sale", Daniel S. Greenberg; and/or the work of economists and social scientists such as Richard R Nelson, Paul David, and John Ziman before you characterise the impact debate (here or elsewhere) as arising from academics "with chips on their shoulders". Philip.

Ooops. That should be "Bayh-Dole" and not "Bahy-Dole" in the preceding comment. Apologies. Philip

Philip Moriarty - Without wanting to misrepresent you. It appears that you have a pretty good line is just dismissing people who disagree with you with catch all phrases like, 'you miss the point, and,'that rather misrepresents...' I was in fact very specific about the value of research that has no obvious societal output - my comments on pure mathematics were very specific on that point. It saddens me that you seem to have made your mind up that people who disagree with you are intrinsically ignorant and hold hostile and absolutist views designed to visit misery on you personally. I was also very specific about my quibbles with impact - you seem to ignore them perhaps because you made your mind up before writing? Perhaps you might take time to consider the views of those who need to go and make a case to taxpayers for a few billion n these difficult times? Sure, as I say above, impact poses great issues and the consultation (which I assume you have actually read?) skates over them. That is not to say that impact is the non-issue you want it to be. As to chips on shoulders, the rather disappointing and high-handed way you try to turn me into a bit of a whipping boy speaks volumes about your shoulder. I have already put far too many words on here, so I will leave the last word to you.

@Dara: I agree that impact is a legitimate and indeed necessary consideration but it has to be framed and it has to be measured in ways that do not demolish the activity ostensibly being measured. Babies and bathwater. As Philip Moriarty points out, there's a fair amount of evidence that this is pretty hard to achieve. It is simply jaw-dropping that the REF proposes allocating 25% of funding to a (for the moment) nebulous and potentially counter-scientific measure.

@Dara: I have not "made up my mind" that those who disagree with me are "intrinsically ignorant". That certainly is a gross misrepresentation of my position! The point I was trying to make is that it is unfair to write off the debate/discussion here thus far as being due to "a lot of academics with chips on their shoulders". I don't think that's fair on *any* of us who have posted comments here. I apologise for the "robust" nature of my response to you - I certainly did not want to make you feel like a "whipping boy" - but your comment re. "chips on shoulders" - like a recent statement about the "naivete" of academics from the chief executive of HEFCE - is unhelpful. I also did not ignore your quibbles about impact - the literature I cited above directly addresses the points you made. And, yes, of course I have read the consultation!

Although I am not an economist, I am a program evaluator by training and experience and I find the debate about "impact" somewhat confused so far. One can certainly apply "impact" as a criterion where it is a valid measure of the worth of the research. For instance, one could easily measure the impact of a study of disparity in salaries by gender of academics by looking at any steps taken by universities, government etc. to rectify that disparity. One can also apply other criteria to justify research and the objective should be to pick valid criteria. Professor Moriarty, as I understand his reasoning, sees impact as silly for fundamental research. The impact will not be known, as he rightly argues, for perhaps years, even decades after something is discovered. However, when looking at fundamental research, it is certainly possible to consider such things as possible implications of findings and the value of those implications. This is a "benefit-cost" assessment as economists might consider it. IBM rightly considered that challenging Bardeen-Cooper-Schieffer theory of superconductivity would be very profitable if it could find high temperature super conductors. The U.S. Government certainly calculated the potential value of an effective ABM shield in the face of open skepticism among many scientists sharing the point of view of those opposed to such research as impractical and likely to accelerate the arms race. Other criteria are available to justify research as well. Without pretending to be a physicist, although I am an experienced contract administrator, I can say that knowing Philip Moriarty is convinced that something is worth doing theoretically in physics would favorably predispose me to such a project because Philip Moriarty has a good track record in these areas and the likelihood of something worthwhile emerging is high. I might be less eager to support someone saying the same thing as Professor Moriarty without that credential. I think that is human nature. English literature research is a bit more chalenging to evaluate but we have good ways of doing that actually such as concordance of views statistics. This is a technique whereby independent raters' opinions are assessed to see their concordance. (see any stat book on Kendall's Test of Concordance or "W" statistic). The bootom line is: we should not be afraid of impact when it is relevant. However, we should not be make it the end-all be all of evaluation either. I presume the assessment tools that result from this debate will reflect that sensible approach.

Who is the "One" in "One can certainly apply ‘impact’ as a criterion..."? (from our "program evaluator" - I assume that the spelling means that he works in the United States, making money from "Qualityspeak" projects, "benchmarking" and other audit operations imposed on various categories of professionals). So I suppose that the "One" who can "apply impact" as a criterion is the Manager who is now, on behalf of his political and business bosses, wrenching control of the universities away from the academics and scientists. Yes, "One" can apply any criterion, as nutty as you like, or as refined: the point is that, *whatever* the criteria applied, academic freedom will have been destroyed. The most tempting counter-argument in this connexion is that, since the public pays, research carried out at the universities should be "transparent" and "accountable". But these are in fact formalized and artificial managerialist categories which bear no relation to real life: the public is not informed by the RAE and REF (and the like), but misled. Philip Tagg wrote some time ago, objecting not to the use of particular audit criteria (like the latest modish notion called "impact") but to the imposition of neo-managerialist audits as such; for * audit as such is demonstrably flawed and totally inadequate as a means of assessing the value of university teaching and research; * audit perverts the meaning of words denoting laudable principles of human behaviour; * audit prevents innovation, perverts research and fosters dishonesty, cynicism and hypocrisy.