Analysis Ranking Challenges: Wrong 'uns and rightful contrasts

16 September 2010

The rules of cricket are common to all who play it, but the compilation of league tables is more selective and open to interpretation, writes Paul Wellings

In the English summer of 1953, the Australian cricketer W.A. Johnston completed the season with a batting average of 102.00. The feat of averaging 100 or more had been achieved once before and, since then, on a further six occasions. However, Johnston's average is special because he scored 102 runs all season over just 17 innings and was out once.

We can be certain about all of this. The rules of cricket are well established. The database for the sport is accurate and regularly updated. Comparative measures (such as averages per wicket and runs per ball) have evolved through international consensus. In contrast, the international league tables for comparing universities have none of these characteristics, for they are still works in progress.

There are more than 17,000 universities around the world with a wide variety of governance mechanisms, funding regimes and state controls. The data on inputs to higher education systems are patchy and often severely time-lagged. The data on outputs are extremely fuzzy. It is hard to be clear about attainment rates, graduate destinations and the quality of the learning environment. This is particularly true for the emerging for-profit sector. As a result, most international league tables focus on selective aspects of research and proxy measures.

The major league tables set out to measure excellence in an attempt to identify the top 1 per cent of global institutions. The development of research-publication databases in the sciences has created the opportunity to rank papers by volume and citation. Proxies for the standing of the academic community have been derived by examining the distribution of markers of esteem, such as prizes and highly cited individuals. Some tables look at measures of international diversity, such as the proportion of non-national staff and students.

Other types of ranking systems are being explored. Some are based on specific degree programmes. For example, the Financial Times produces an annual international ranking of MBAs. Interestingly, this table is often used as a proxy for the overall quality of business and management schools. Other systems avoid rankings based on university education and research quality and look at other measures: for example, the Webometrics Ranking of World Universities examines online access, presence and visibility.

Given the growing number of students around the world and the competition to gain places at the best institutions, we can be certain that comparative tools such as league tables are going to gain ground. In future, the best examples will be those that emphasise the quality of outputs, correct for variations in institutional size and examine the subject mix on offer.

The presence of an active postgraduate research community is one clear marker of a research-led environment. At the subject level, the number of PhD completions per academic is a direct way of measuring the dynamism of graduate schools. Not all universities offer the full spectrum of degrees, but, for those with advanced degree-awarding powers, we should use some measure of their PhD productivity as an index of their ability to produce the next generation of academics and researchers.

Despite all the criticism of league tables and their construction, the academic community should take them seriously for three reasons.

First, because national policymakers will use them as a basis to ration scarce resources. Second, because internationally mobile students and staff will utilise them to help identify top institutions in their disciplines. Third, because governing bodies, seeking to improve the standing of their own universities, will want simple comparative data to help guide their strategies.

In 1955, when Johnston retired from cricket, his first-class batting average was 12.68 — some way short of the dizzy heights of the 1953 season. Does this matter? Well, the answer is no; for Johnston was a bowler, not a batsman. If we really wish to understand his achievements we should consult another section of the cricket database. Here, universities and cricketers share a common feature. There is a broad range of roles that make players and institutions different.

Put simply, when it comes to reputation, mission matters as much as ranking.

Paul Wellings is vice-chancellor, Lancaster University, and chair of the 1994 Group of smaller research-intensive UK universities