Truth in numbers: study pinpoints 'critical mass' for research success

The EPL paper referred to in this article is published as R. Kenna and B. Berche, The extensive nature of group quality, EPL 90 (2010) 58002. It is also available at http://de.arxiv.org/PS_cache/arxiv/pdf/1004/1004.3155v3.pdf

I managed to find the paper before seeing Kenna's comment, but the link should really have been given in the article.

Having looked at the original paper, I find myself baffled by what is meant by "group size". The biggest (and ellegedly best) 'groups' seem to be around 40 in size, so I can assume only that 'group' as used in the paper refers to a whole department. If that is the case, the method of analysis seems oversimplified to the point where it is unlikely to be much use, I thiink you'll find that it is a bit more complicated than the paper suggests. The mathematics disguises the fact that it is really primitive observational epidemilogy, with all the well-known hazards of that approach,

In the areas I know about at least, research groups, led by and individual principal investigator, usually consist of 2 to 10 PhD students and postdocs. I have rarely come across bigger groups, but I have not always been convinced that the PI had read all the papers on which his/her name appeared.

Another important consideration that isn't mentioned is that it is surely well-known that large groups are less productive per person (and therefore per pound) than small ones, and provide an (even) more cruel environment for working than small ones. It doesn't really matter to the reputation of the PI if a few of the members of a large group fail and end up on the scrap heap,

I have known well three Nobel prizewinners, and in all three cases what distinguished them was that they did experiments themselves, All three had rather small groups, at least in the years when they did the work for which they got the prize,

The problem with this work, like so much bibliometrics, is that it doesn't concentrate on individual brilliant people. In fact, like so much bibliometric work, it could be used to fire potential Nobel prizewinners before they ever got started.

At least this suggests areas of work that we could do without when the financial crisis really bites.

Colquhoun missed the point that the definition of the group is not our choice, but that of the group itself which submits to RAE. This is not necessarily a whole department or a subgroup centred around one individual. A group in the RAE sense is often built of several strongly collaborating entities (which we call subgroups) immersed in a less-collaborating, but still communicating bigger entity, the group.

Then he missed that the quality of the PI's in subgroups is a sub-leading effect, whatever his opinion. He also missed the fact that we base our analysis on a peer-review assessment and not on bibliometrics.

Then he is further wrong in the comment that “large groups are less productive per person” since the quality score per person is higher in big groups.

Last but not least: trying to give an example with three Nobel Prizes Colquhoun knows personally cannot invalidate a theory which is based on average trends. No theory in sociology can claim to capture the precise details of every individual component. This is the same in physics with mean-field theories: in the real world there are fluctuations and no component has exactly the mean-field interactions with its neighbours. Yet mean-field theories are extremely powerful and describe e.g. critical temperatures and produce accurate phase diagrams, even though they do not capture subtleties like critical exponents. We invoke the principle of Occam’s razor: ours is a mathematical model whose simplicity is its strength.

Thanks to Ralph Kenna for clarifying what "group" means. But is he not aware that groups, in the sense that he defines, are often imaginary constructs of administrators whose job it is to maximize the RAE scores?

The lesson to be drawn is not that big groups are good, but, as predicted by Goodhart's law, the imposition of crude assessments like the RAE distorts the behaviour of those being assessed. This distortion may, on occasions, get distressingly close to lying. That is not a good basis for making inferences or deciding policy.

All this is in addition to the fact that there is no evidence about causality in the observed correlations. Without knowledge of causality, the correlations could not be used for deciding on action (even if the data were perfect).

As we mention in a related paper (http://de.arxiv.org/abs/1006.0928), we cannot account for managerial tactics in our analysis. However, as a microscopic model, ours necessarily includes a causation hypothesis (that quality is driven by quantity, primarily through interactions) which is then tested. The opposite causal direction may of course enter as a feedback mechanism, but it is not the primary driver. Indeed, the opposite causal direction would not be expected to manifest breakpoints, the existence of which we have evidenced.

I'd just like to add that Colquhoun's original observations are spot-on, notwithstanding Kenna's singularly unimpressive rebuttals.

Since he contributes nothing original to the discussion, van den Berg's remark warrants no response. It suffices to say that our second paper in this series has just been accepted to be published in the journal Scientometrics. That paper is called "Critical mass and the dependency of research quality on group size" and is downloadable here: http://de.arxiv.org/abs/1006.0928 .