Pre-registration would put science in chains

The pre-registration of study designs must be resisted, says Sophie Scott

Hand in chains (B&W)

Science is not well served by people deciding that their methodology is the only legitimate one

As concern grows about questionable practices and outright misconduct in the life sciences, the pre-registration of study designs and hypotheses is being wrongly touted as the panacea.

The campaign’s latest push came in an open letter to The Guardian last month written by Chris Chambers, research fellow at Cardiff University, and Marcus Munafo, professor of biological psychology at the University of Bristol, which was supported by more than 80 signatories.

Drawing on a paper that asked psychologists to self-report their own dubious behaviour, they argue that large numbers of life scientists cherry-pick data, hide null results, fail to employ adequate statistical power and reinvent the aims of studies after they have been completed to make it look as though unexpected findings were predicted.

They claim that pre-registration, which would involve journals accepting future papers based on the design of experiments rather than their results, would greatly reduce such questionable practices since the incentive to indulge in them to make papers more publishable would be substantially reduced.

However, there are numerous problems with the idea. Limiting more speculative aspects of data interpretation risks making papers more one-dimensional in perspective. And the commitment to publish with the journal concerned would curtail researchers’ freedom to choose the most appropriate forum for their work after they have considered the results.

With no results to go on, reviewers would be more likely than ever to rely on reputation, which would count against junior scientists. Unsympathetic ones would also be handed the chance to veto studies at the outset. In addition, the requirement to refine studies and their interpretation prior to data collection would prevent us from learning from our mistakes along the way.

Moreover, in my fields (cognitive neuroscience and psychology), a significant proportion of studies would simply be impossible to run on a pre-registration model because many are not designed simply to test hypotheses. Some, for instance, are observational, while many of the participant populations introduce significant sources of complexity and noise; as introductions to psychology often point out, humans are very dirty test tubes.

I am also very uncomfortable with the model’s implication that hypothesis testing is the only correct way of doing science. This may be true for the clinical trials from which pre-registration takes its inspiration, but we have known since Thomas Kuhn that scientists don’t just proceed, study by study, testing individual hypotheses.

In online discussions, Chambers has claimed that pre-registered studies would have “a substantially higher truth value than regular studies” because of measures such as a requirement for authors to declare whether they are discussing results they did not predict. But this sort of language is deeply misleading. Most scientific studies are “wrong” in the long term. Science is a process rather than a method of finding out all the things that are true: a process during which we must be allowed to run studies in which we get things wrong, change our minds and are led in directions we didn’t expect.

If we allow to pass uncontested the claim that the pre-registration model is a gold standard, we will permit the denigration of the vast majority of great research and allow a number of serious constraints to be placed on it.

Science is advanced by communities of researchers and is not well served by people deciding that their methodology is the only legitimate one. Instead, our responsibility is to do the best science we can; to be open-minded and interested in the findings of others; and to do our best to support the careers of those who come to work with us.

That way, there is a fighting chance that we will be remembered for the good stuff that we do. After all, even Newton sometimes employed dubious methodologies. His celebrated physical laws were supported by data, but history tends to overlook his equally enthusiastic pursuit of alchemy, which swam in a sea of null results.

Rate this article  (3.26 average user rating)

Click to rate

  • 1 star out of 5
  • 2 stars out of 5
  • 3 stars out of 5
  • 4 stars out of 5
  • 5 stars out of 5

0 out of 5 stars

Readers' comments (1)

  • Pre-registration would force reviewers to rely on reputation of the authors alone to judge a paper's merit? It would seem there would be a need to blind reviewers to author names and vice versa. Isn't this the current system? I also question the premise that if someone's design and theory is approved for a good journal, but they end up with "great" (I'm assuming you mean statistically significant) results, you'd be shackled to publish in the journal that ok'd the design. Do great results offset good designs? Seems like you'd be rewarding sampling error and/or questionable research practices. If the argument is that we should not be so reliant on the hypothetico-deductive model & NHST, then I'm behind you 100%, but I think this article is easily misinterpreted as "everything's fine, no change needed" when in fact the system is broken and post hoc data and hypothesis tinkering is the norm, not the exception.

    Unsuitable or offensive? Report this comment

Have your say

Remember you need to be a registered THE member and logged in to comment on stories. Please read our terms and conditions for posting guidance.

  • Print
  • Share
  • Comments (1)
  • Rate
  • Save
  • Print
  • Share
  • Comments (1)
  • Rate
  • Save