Advertisement
Advertisement
Advertisement
Lancaster produces alternative RAE 2008 rankings
19 December 2008
Institutions’ selectivity in submitting staff to research assessment exercise should be taken into account, says 1994 Group member
The University of Lancaster has produced its own research assessment exercise results league table, which takes into account how selective institutions have been in choosing staff to have their work judged.
The move indicates how determined some universities are to have their “research intensity” recognised, after the Higher Education Statistics Agency (Hesa) was unable to provide alongside this week’s RAE results figures that would have shown the proportion of research-active staff each university had excluded from the RAE. The results would have exposed institutions that had hidden weak researchers from scrutiny.
On the alternative league table it devised, Lancaster, a member of the 1994 Group of smaller research-intensive institutions, occupies ninth position when specialist institutions are excluded. The universities of Edinburgh, St Andrews and Bristol also move up.
The new ranking exercise comes as a number of institutions are now choosing to reveal how selective they have been.
In a statement on the University of Oxford’s RAE results, John Hood, the vice-chancellor, said the university had submitted “over 85 per cent” of its academics for assessment (http://tinyurl.com/4kuj5h).
Meanwhile, the School of Oriental and African Studies, University of London, said it had entered “a little over 92 per cent” of its eligible staff ( http://tinyurl.com/3qrpew). It said it revealed the figure because “league-table compilers have given data that includes teaching-only staff, thereby reducing our figure to 65 per cent”.
Brian Cantor, vice-chancellor of the University of York, argued strongly for the inclusion of the “intensity” measure in the run-up to the release of the results in Times Higher Education. He revealed that York had a 91 per cent staff submission rate in the 2008 RAE.
The Times Higher Education Table of Excellence presents contextual information on the indicative proportion of RAE-eligible staff submitted for assessment by institutions, but does not use it in devising rankings.
The figure is calculated by dividing the total number of staff that an institution submits to the RAE by the number of academic staff at the institution within the grades “professors”, “senior lecturers and researchers” and “lecturers”, according to the latest published data available from Hesa. Times Higher Education has also provided detailed caveats on how the data should be interpreted.
Lancaster’s table presents the same information on the indicative proportion of RAE-eligible staff submitted as the Times Higher Education’s table, but takes the process a step further by multiplying the figure by universities’ average scores to get a “modified” average score, which it then uses to rank institutions.
In Lancaster’s table, institutions in which the indicative proportion of RAE-eligible staff submitted is greater than 100 per cent (including Lancaster) are capped at 100 per cent, although Lancaster noted that its actual submission rate was 90 per cent.
“It is an issue of transparency,” Paul Graves, Lancaster’s director of governance and planning, told Times Higher Education. “We accept that there are a variety of ways that submission rates might be interpreted or used, but we think it an important principle that the RAE results should [include them].”
Further figures indicating how selective some 20 universities were in submitting staff are also known from surveys conducted by Times Higher Education in late 2007.
Among research-intensive universities, submission levels are known for Royal Holloway, University of London (where 91 per cent of eligible staff were submitted) and the University of Sussex (86 per cent).
Among the teaching-intensive universities, final submission figures are known for the University of Chichester (18.5 per cent), Thames Valley University (11 per cent), Keele University (about 50 per cent), Canterbury Christ Church University (16 per cent), Northumbria University (18 per cent), Glasgow Caledonian University (25 per cent), City University London (62 per cent), the University of Salford (45 per cent), Manchester Metropolitan University (31 per cent) and the University of Wolverhampton (18 per cent).
Planned submission rates were provided to Times Higher Education by Oxford Brookes University (33 per cent), Birmingham City University (11 per cent), Kingston University (26 per cent), the University of Derby (9 per cent), Brunel University (87 per cent), the University of Lincoln (35-40 per cent) and the University of Plymouth (40-50 per cent).
But greatest interest is focused on the top of the table, with many in the sector keen to know whether the larger research-intensive institutions have been more or less selective than their smaller counterparts in who they submitted to the RAE.
The University of Lancaster’s league table can be viewed at:
Previous Times Higher Education stories on the proportion of staff submitted:
Exclusions from RAE see steep rise
Teaching institutions draw on smaller cohort for RAE
Readers' comments
Oh come on you Lancastrians, no sour grapes please - have you been learning from the Labour Party that anything you like can be spun to the positive. How was it - 1st in UK for Physics - well done - joint 2nd in UK for Design, Art, Music and Theatre , and in Bio-medicine - well at least Bio-medicine is useful to mankind and to advancing knowledge and understanding. And what happened to your Law, Politics, and European Studies submissions - not quite so hot were they - but I guess they will all bask in the reflected glory of your few well deserved results. But as to trying to manipulate the rules of the game after the stable door has been closed is just a bit late guys. In fact, lets try a different average across all your RAE submissions averaged by the number of staff you submitted - try it - you won;t find yourself in the 'top' grouping any more - but then, of course, that would just be sour grapes on my part in suggesting a different spin ...
I fully agree that this new table is the result of cynical spin. Unlike York there were no preemptive calls from Lancaster to include intensity in the ratings. It was only after Lancaster manipulated the numbers that it decided a great injustice had been done. <p>Incidently the Lancaster table places it 9th; it was in the 10th row because the table heading was in the 1st row. Thus according to THE "institution" came 1st and Cambridge came 2nd. <p>Having said that, based on my own predjudices, I have to say that I believe the Lancaster table to be more accurate. A certain Colchester based institution is certainly in a more credible position.
Red roses or white? As an 'umble Lancaster research associate, I am astonished to find educated grown men and women squabbling in such an unseemly fashion about a septennial exercise which produces the higher education equivalent of the school exam league tables. Christmas can be a stressful time in the academic year - please all go home and have a rest over the festive season, and let's start 2009 anew. Best wishes to one and all.
I think David Flowershott's shot's wide of the mark. There has to be an adjustment for intensity, or the figures really don't mean anything in ranking terms (do you really think Warwick's law research output is lower than Oxford Brookes' and Swansea's?). The Lancaster table provides a ranking, rather than raw data, and that's what most of us will be most interested in (apart from the money, and we'll have to wait to learn about that). Laurie Taylor points the reason why, in this week's "Poppletonian" - Poppleton leaps 123 places, by submitting only one person in each unit of assessment. From the point of view of assessing each university's/department's research output, if percentages are going to be used (ie, so many percent 4-star, so many 3-star and so on) they are only meaningful if you know the percentage of staff submitted. <p>It is a fair point Tom Hope makes that York pre-emptively called for intensity and Lancaster didn't, but doesn't the speed of response (those calculations clearly weren't done in five minutes on the back of a fag packet), and past performance, indicate that Lancaster must always have been concerned about the importance of adjusting for intensity? It looks as though while York was (rightly) banging on about what wasn't going to happen, Lancaster was getting on with ensuring it did happen, if only unofficially. <p>As a Lancaster lawyer, I'm not the right person to respond to Flowershott's snipe (and I write in a personal capacity), but I would point out that we submitted 22 out of 24 staff, and the two who weren't submitted are on teaching-only contracts. So that's 100% return of those whose contracts include research, including a number of members are early career. Since the Lancaster law score was as good as Warwick's, I think we've done pretty creditably.
Is it true that Cambridge and Oxford had a huge increase of staff FTEs (by some 600-800) in 2007? If this is not the case, is the HESA data (2006-7) reliable, or is there a better explanation for the sudden increase of staff FTEs in some universities?
The fact that this modified table has been produced by an institution that gains 11 places by the modification makes it about as impressive as those modified olympics medal tables adjusted by some combination of GDP and population size that always put Australia as number 1 and are invariably compiled by Australians.
PS- just googled David Flowershott - no results. Non-existent, or just research inactive?
I tried - in a simplistic way - to do something similar for my discpline of Computer Science (UoA23). You can see the results at www.wmin.ac.uk/~courtes/uoa23rank.html. My method multuplies the scores for each rank by the number of FTE staff in order to try to give more weight to institutions that submit more staff - the difference is remarkable. What can't be done though is to weight each classification by the number of staff whose work falls under that classification.
I agree, there is a degree of cynicism here which can favour some and not others. Large departments either suffer for submitting a proportionally lower number of staff or because unless you are an Oxbridge, having a whole department at tip top quality is going to be very tough indeed. Glasgow (32) Kings College London (37) Newastle (39) Queen's Belfast (44) and Cardiff (46) are all Russell Group examples here. If we compare a department with 50 staff against one with 5, it'd only take three researchers to hit the 4 grade versus 30 in the former for the results to bear equal, it'd only take 4 submissions versus 40 for the relative percentages submitted to add up. The relative strength of insitutions such as Lancaster, Essex, Sussex and Aberyswyth which come nowhere close to competing with a King's or a Glasgow in funding is a clear example of this. <p>To make the point that I made in another thread, large Universities don't spread the money among staff equally anyway, so they'd be inclined to submit as much as possible for as big a pot as possible and take the hit on average quality. If the lion's share will be going to a small percentage of researching staff anyway, it'd suit them down to the ground to have this flexibility. To give a standing example of this, Imperial College sumitted 5 members of staff at a 3.2 average for one area, Edinburgh 62 at a 2.65 average for the same. Since institutions all have their strengths and weaknesses, of which the multi-faculty organisations are most prone to exposure of the latter, then its unsurprising some results have come out the way they have. When the funding is diced up, however, I'm sure some Universities ranked outside the top 30 in Lancaster's table will be sitting pretty in and around the top 10 when it comes to a share of the pot. <p>To this end, the most impressive results are UCL, Edinburgh and Manchester who have all hit high grades using both methods despite submitting a huge volume of researchers. While we can argue all day if students really benefit from the staff on purely research contracts, it's safe to say they'll do very well next year when the cash is handed out.
Once again, a farce of epic proportions. <p>I think we should be congratulating some of the post-92 universities on their 'better performance' this time round. By better performance, I mean that it's getting harder for rubbish lists like this to ignore their significant research output. And I say good luck to them. <p>I look forward to printing off the list and using it as toilet paper when mine runs out over Christmas.
We can complain as much as we want. We can invent new tables in ways that improve our performance. None of it matters. Lancaster clearly chose to submit too many staff. Also, "staff on teaching contracts" are called lecturers- its a disingenious way of hiding non research active staff. Not a very British approach at all - bad sports!
there are 2 points about this table. First, it was a good idea. If you were sending your kid to a university, or considering where to go to do a PhD, you'd want to know what percentage of their notionally research-active staff were in the RAE2008 submission. In fact that would probably be one of the two most important metrics you'd want to know about. In that respect the point is well made by this table. However it was a flawed table. This is obvious because, according to the table apparently Oxford submitted 157% of its FTE to the RAE. Even Lancaster itself apparently submitted 106%. With that level of inaccuracy, applying rankings is fairly meaningless.
Let's be clear what this data means. For a University scoring 100% on the ratio of raefte submitted to academic staff numbers in HESA, that is probably achieved by submitting around 80% of the academic staff on T and R contracts with the other 20% of the submitted numbers being eligible independent researchers not in the HESA academic staff category. Oxbridge has a vast army of college fellows partly funded out of QR, that's how you get a number like 157%.