Advertisement
Advertisement
Advertisement
IUSS reaction: ‘Quality control’ is the problem, not the solution
3 August 2009
More of the same won’t allow us to reform the British system effectively, writes Florian Bieber
No research assessment exercise, research excellence framework, external examiners, double marking or moderation. What may sound like a dream to many academics (myself included) is the worst nightmare for higher education administrators and the Innovation, Universities, Science and Skills Committee, to judge from its report on the state of universities that was published on 2 August.
The report recommends more quality controls, more standardisation and a greater role for external examiners and the Quality Assurance Agency. However, one might ask how America’s Ivy League universities manage without much of this? The reality is that rather than boosting the quality of university education, the logic of quality control is a major source of the problems that bedevil the UK sector.
Part of this lies in the apparent confusion between quality control and standardisation. While standards can secure a minimum of quality, they can also stifle the variation, creativity and maximum quality so essential to higher education.
There are four dynamics at work here:
a) standardisation runs contrary to the logic of quality-based differentiation
b) quality control often leads to increased workloads with few benefits
c) the discussion about widening participation is not linked with quality
d) and finally, the debate is insular.
The select committee’s report – indicative of much thinking about higher education – laments the lack of uniform standards across the sector. The authors appear infuriated that the vice-chancellors of the universities of Oxford and Oxford Brookes cannot answer the “simple question of whether students obtaining first-class honours degrees at different universities had attained the same intellectual standards”.
Herein lies standardisation’s fundamental logical flaw: you cannot have better universities without worse ones.
In brief, not every university can be Oxford or Harvard. It will always be difficult to compare a graduate from a lower-rated university programme with one from a top institution, even if the degree has the same name. However, there is nothing wrong with that. Some universities will always be better, meaning that their degrees cannot be identical to others. Trying to impose a uniform standard is likely to result in a drive towards the lowest common denominator rather than the highest level of quality.
In addition, the system of external examiners, double marking and moderation is more often than not a time-consuming waste of academics’ time. These standardisation tools have in-built disincentives that often result in everybody involved going through the motions of upholding standards, spending valuable time that might otherwise be used to increase the number of contact hours with students, the low number of which the report laments. More effective complaints mechanisms, as seen in the better US universities, are likely to be fairer than a bureaucratised system based on a fundamental distrust of the judgment of teaching staff.
Another key tension that is not sufficiently acknowledged in the report is the conflict between widening participation (that is, bringing students from disadvantaged backgrounds into the university system) and quality. Many such students will excel and enrich the sector; but at the same time many will pose a challenge to maintaining certain standards of education. This is not to say it is not worthwhile, but it is potentially a trade-off that must be confronted, something best done at a much earlier stage of the education process.
Finally, the select committee report draws on a visit to the US – but not a single European country – and recommends the community college model for the UK (clearly the authors did not visit many community colleges).
The report, in common with much of the debate on higher education reform, is very insular. For instance, the Bologna reform process, which is among the key tools for creating a European higher education space, is mentioned only in the footnotes. Indeed, this reflects the lack of debate on how the UK can integrate and maybe even learn from the experience of other European countries. No doubt the challenges that the often very hierarchical university systems in the rest of Europe face are greater than those in the UK, but this does not mean that nothing can be learnt from them.
More importantly, we are already part of a European academic space through research and exchange programmes such as Erasmus and the European Union’s Framework programmes, and it is time to stop ignoring this when it comes to reform. Although the US higher education sector is often even less aware of the world beyond its doors than ours, the fact that its universities are very diverse (from Deep Springs College in the Californian desert with 26 students to Harvard with about 20,000) allows for more creative learning.
In short, to improve the UK’s universities, we must stop recommending more of the same. Instead, thinking outside the box and looking harder beyond our shores might be a good starting point.
Florian Bieber is a lecturer in East European politics at the University of Kent. He spent the spring as Luigi Einaudi chair for European and international studies at Cornell University, New York.
Readers' comments
"Standardisation runs contrary to the logic of quality-based differentiation" - not necessarily! And why is it a flaw to accept that "you cannot have better universities without worse ones"? And while it may not be a simple question, it is surely not an unreasonable question to ask "whether students obtaining first-class honours degrees at different universities (have) attained the same intellectual standards”. And, although I have no faith that we have any existing systems that can in any way assure the standards of our current practices, we already accept the fact that a far higher percentage of students get a higher degree classification at universities like Oxford than at others. Surely all the committee is recommending is that such differentiation should be evidence-based using rigorous systems of comparison, and thereby justified and deserved.
One of the problems with "quality control" is that the word quality does not mean what you think it should mean. Instead, "quality control" merely checks that you have bueaucratic mechanisms in place to check "quality", and in practice that is never about actual teaching quality, but about lists of aims and objectives, check boxes, and lists of buzzwords. The idea that more of this will improve quality is lunacy, since the existing processes have had - at best - no effect on actual teaching quality.
No William, you are right, we should leave teaching alone completely and go back to a 70s model - if you want to get pissed at lunchtime and lose assignments then you should jolly well be able to after all its the top 10% anyway and daddy will get them a job so it doesnt really matter what we teach them. Its too stressful for teachers and academics to be held accountable, to expect to behave within minimum standards, and to make sure they are doing their job - in fact its insulting. People don't decide to go into academia to be asked to be responsible! Fancy asking how money is spent? How base.
Will we be requesting the same quality control systems for the financial sector whose recent bailouts contributed to the current round of public sector cuts?
@Hero: You seem to have a very large chip on your shoulder...William is correct when he says that quality control has nothing to do with checking the actual teaching quality. It is all about box ticking and having a paper trail in place. It is a total wast of time and money.
OTOH, many of us could share anecdotes about colleagues who did get pissed at lunchtime and lose assignments; and the notion that second-marking is a waste of time really is a licence for that kind of utter slackness to come back as a norm... OK, maybe not a norm, but do you really think that the innate virtue of individual academics is enough to guarantee the quality of their assessment practices, and hence of the degrees awarded, with no external checks at all?
We are all in favour of better quality education and improving teaching quality, or so we are told, I think the problem lies in that very few academics I meet are willing to make any changes to what they do on a day to day basis. So the committee might be wrong in what its solutions are but there are very few suggestions coming from the sector itself about how to deal with the issues of poor quality higher education. I agree with Dr Rust, I think the committee are well justified in their findings because of the poor quality of argument in many of the VCs evidence, which just seemed to be saying "we are doing well, leave us alone". This is not an argument for the status quo or any suggestions for how to improve. I agree that some of the ideas coming from European Universities would be just as good or even better than some of the ideas coming from the committee. But the real problem seems to me that many VCs I talk to want to do nothing at all, because what we are currently doing is good enough, and this is the defensiveness and complacency that the committee is talking about. The old adage stands: Do nothing and the decision will be made for you.
A recent study of heart surgeons who were forced to use quality control and target setting found that more operations were done, more were successful and more risky operations were taken on and were more successful. Measuiring means understanding. Not measuring allows excuses, justification and BS to lead good management astray.
It is very important to monitor quality of work. What the anti-measurers (their obvious fear of being caught out aside) seem to assert is that because quality of learning in the students' heads is impossible to measure directly, then it is pointless to measure either the effects of teaching, or to determine if components of teaching activity we have decided are necessary are being done correctly. This is surprsing thinking for an academic - most research measures a series of effects and/or behaviours and draws a collective inference from it to either confirm, deny or come up with new theories that point towards solutions to particular problems. To say 'I can't measure the temperature of the water, so there is no point in measuring the temperature of the pipe' is ridiculous. Of course its notgoing to be exact, but its closer and representative. I think the real thing that should be measured here is 'How afraid are teachers and lecturers that they will be found out for being lazy/incompetant/biased/etc?' If acadmic confidence is such that each academic does his or her best then why worry?
This reminds me of the 'professor and god' story when a pupil questions the existence of the professor's brain - can't see it, can't touch it, know that a lot of dead people had one in their head, but not that there is definitely one in his/her particular head but can measure the effects of it being there and because these effects have been seen in many other cases to be the action of the brain, we know that it is likely to be the brain... we still don't actually know. I think what Peter is saying is that we need to do as close as we can - not just throw our hands up and say 'what's the point'.
Now "many fewere" sentences, but the message still has problems. At least it is Hero to Zero!
While I am very critical of quality control in my article, I would not want to argue that there should be no control whatsoever. Instead, one should think about quality control which is not based on a large administration and its conception of quality, often divorced from reality, and has in-built disincentives to actually work (such as moderation/double grading systems). Why does no other Western university system I am familiar with (I stand to be corrected) have double grading/moderation and external examiners? I would argue that there are at least four ways to ensure quality: 1. Take student evaluations seriously. In top US universities, student feedback has an impact on the promotion. 2. Introduce more systematically probationary hiring or other variations of the tenure system the US has (the system has some shortcomings, so a direct copy might not be advisable). 3. Encourage and enhance competition between universities. If a department get the reputation as being sloppy, lazy or otherwise, student number will go down and the pressure will built to improve. 4. As I wrote in my article, it would be better to have robust complaints mechanism which allows students to complain about grades if they find them unfair rather than going through the motions of moderation/double grading etc. In brief, I think we should not see the debate as being for or against quality control, but to discuss how quality can be controlled and whether the current system really accomplishes was it claims to do and whether more of the same is really going to help the quality of HE.
I went to a large, state US university in the midwest for my MA, fifteen years ago, then back here for DPhil. Only one person marked my MA level coursework---the person who taught the course themselves, except for two oral exams with committees of three. There were those with a reputation for being slightly more difficult than others in marking---a form of diversity which was thought to be normal, a fact of life. That's what "real life" is like, they said to the students---you can't expect everyone to teach exactly in the same way, nor to mark exactly the same way--as different lecturers may have different demands appropriate to the course and different teaching philosophies. Exposure to this diversity was seen as a good thing, one which prepared you for the workforce and for "real life"---where standardisation in fact does not exist. Speaking as a former student, the quality in the teaching there, which I, and so many others found superb, emerged not through standardisation of paedagogic methods or marking, or anything else, but through three things: 1. the professors had experience teaching their subject and teaching more generally 2. they were active researchers in the field, with passion and wide knowledge of the field which they drew from 3. class sizes were no more than 10! I don't need to emphasise the multiple benefits of small class sizes here I hope. Let us not forget that thinking about quality in terms of standardisation only makes sense when you think about education as a "product" on the assembly line: think of cans of Coke. Standardisation is a virtue in this context, each can of Coke is exactly the same, with no deviation from the formula, this is the definition of high quality...thus, the brand can be trusted and the customer knows exactly what to expect. Does anyone really believe that quality in education can be, or should be, about standardisation? It isn't possible and it isn't a worthy goal.
I hope it is clear that I meant in my previous post that only the person teaching each course I took marked the coursework...not that I only had one professor for the entire MA!
Teaching assessment is not a bad thing. I recall some abysmal lectures delivered to me as an undergrad (80-83). Overall it was not bad for me because I just went to the library and taught myself. I have now been teaching in the sector for 20 years. The standard of lecturing in my dept is very good. New members of staff who deliver a bad course are sympathetically leaned on in the first instance and if the problem is not corrected the question posed is "are you in the right job? Surely this is the right way to be. The result is that teaching quality has gone up.
This article, and most of the posted comments, perpetuate the all-too-common confusion between Quality ('Control') and (Academic) Standards. Unless there is a clear and consistent use of these terms (and their respective meanings) discussion or debate is doomed. The comments posted so far are at cross purposes. Please try again, folks...
Quality control has taken an importance of its own and in new universities a huge departments with administrators are in place to maintain quality processes and they have departmental represetatives usually academics selelcted for their love for box ticking and form filling rather than front line teaching. The latter opt for it as they are bad lecturers with shaky knowledge and are paid more than their lecturing colleagues. Yet one can see not much of tangible quality in any of the processes in these new universities. I bet you can see large central quality departments and quality monitoring staff in these failed Mets which are in the news these days.
The type of quality control which Quality Depts. refers to is a process and concept which has emerged from, and only makes sense on, the factory assembly line. It assumes that you can control the quality of the product, by checking its basic properties as it rolls on by. If every product off the assembly line has the same properties/qualities, according to the product specification, then the products have high "quality". Does anyone really believe that this model of assessing quality can be applied to teaching? My former post 92 university, through the application of this model , has publicly claimed excellence in teaching year after year, yet everyone knows the reality of the teaching situation there---it is dominated by hordes of stressed, part-timers teaching anything and everything they can to survive. And no matter what the "quality control" processes claim (excellence year after year) the students know better.
To Hero, and like minded: I am all in favour of trying to assess teaching quality. My point is that the existing system does not do it, and never could. It is a vacuous exercise in bureaucracy. But when you "measure" and "target set" you have to be careful what you're doing. Naive approaches (such as the blind enthusiasm of Hero) are likely to be worse than useless. We could ask, for example, if the heart surgeons he likes actually did more unnecessary operations, as well as more useful ones? We certainly see that kind of behaviour in the parts of the NHS that have adopted targets as the be-all and end-all of public health. I'm sure that many of us have seen blind target-chasing in our senior management groups. In university teaching, some of the targets (such as minimizing drop-out rate) might seem to be good on the surface, but the easy route to hitting the targets will be to loosen standards. The current destruction of core subjects like chemistry and maths, in favour of media studies and the like, is an unintended consequence of enrolment targets.
As an academic manager responsible in part for quality processes in a large faculty in a post-92 institution, I take excpetion to the assertion by Qualty Depts of "academics selelcted for their love for box ticking and form filling rather than front line teaching. The latter opt for it as they are bad lecturers with shaky knowledge and are paid more than their lecturing colleagues." The reason I chose to move into this role is because of my love and experience of front line teaching, and a desire to capitalise on that to improve the overall student experience. What I feel is important is that we follow a continual process of quality enhancement - responding publicly and visibly to student feedback, focussing on key issues such as assessment methodologies, support for students with disabilities and new methods of delivery. While I recognise that the process of control as currenlty practised can sometimes be seen as no more than a series of lightweight checks and balances, these can lead to important changes, such as changes to content, assessment and marking, based on external examiner comments, and flaws in marking processes and discrepancies identified through second marking. In an ideal world, we wouldn't need to carry out quality control processes, and audit everything that we do. We could focus solely on enhancement, but while there are still benefits to be gained, and while sometimes deplorable errors can be identified through control processes, then we risk losing one weapon we have to defend our position.