Science: it's reviews all the way down

The COVID-19 pandemic focused the world’s attention on scientific research like never before. Scientific research has been there the whole way - from identifying existing medicines which could treat COVID-19 and showing how the virus affected people differently to explaining how the virus was spread and, ultimately, finding a vaccine.
But who decides what is good research? Professor Leonard Schalkwyk explains the tried and tested system of peer-reviewed publications. 

Research under the spotlight

Reading customer reviews before buying goods from Amazon or booking an Airbnb has become standard over the past few years. However, scientists have been prototyping this over the past century or two in the form of peer-reviewed scientific publications.

COVID-19 has put science in the headlines daily for more than a year and with many discussions about numerous COVID-19 studies - many not yet reviewed - this system has come to the attention of many more non-scientists.

Every legitimately published piece of scientific research has been evaluated by unpaid, usually anonymous volunteers. Although obviously imperfect, the peer review system is an essential part of the scientific enterprise. It dominates how we think about our work.

You might imagine scientists as a few powerful people with all the right tools and knowledge laying bare the nature of things. This caricature is promoted by Nobel prizes and a lot of science reporting, and it’s influential among scientists too, but the reality is messier and involves many people.

Convincing our anonymous peers 

There might not be agreement that a question we’re investigating even makes sense, the methods and instruments we use are often newly invented, and results require careful analysis to become conclusions. Through the whole process we’re thinking of the anonymous reviewers we’ll have to convince. What controls will we have to use to demonstrate to their satisfaction that our findings are real? How can we photograph, plot or describe our data so they will be able to follow and hopefully endorse our arguments?

The scientific literature that peer review has shaped is magnificent, the greatest worldwide compendium of knowledge ever, linked together with citations (old school hyperlinks) and increasingly available free online.

We have this to thank for advances in technology of every kind. It’s why the knowledge was there to develop multiple, very different vaccines for COVID-19 in a matter of months. There have been discussions for years of ways to improve, reform or replace the peer review system, and scientific publishing is continually evolving.

It's made by people though and very imperfect.  The peer review system does not always prevent dull, poor, mistaken or even dishonest work from being published. It’s also extremely laborious and under heavy pressure as the number of people publishing worldwide expands. Every researcher has been frustrated by reviewers who are difficult, or wrong, or don’t get it. Given the stupendous success of worldwide science, the conclusion has to be that however terrible the system, it’s a lot better than any available alternative. It's important to remember that no individual publication is the final word.  You can find an individual paper (or professor) to support nearly any daft proposition. You have to look at all the relevant work to get a clear understanding.

Reviewed, reviewed, reviewed

Researchers and science administrators are so convinced of the benefits of peer review that they have applied it to everything. It may come as a surprise to those outside the academic world just how many times every research project is reviewed, and how much time researchers spend reviewing and being reviewed.

Research costs money, and very few scientists are given resources with no questions asked.  Whether the source is a government agency, a charity, a company or a university, the starting point is a proposal. This is very similar in form whether the cost is five thousand or five million, and they are evaluated using anonymous peer review.

There’s a “case for support”, arguing for the scientific importance, experimental approach and feasibility of the work and suitability of the proposer. These are expected to be novel, yet well supported by the literature and preliminary data, creative and ground-breaking yet good value for money and feasible. This is accompanied by a detailed costing and additional information on how data is going to be stored and shared, on economic and other benefits of the research, on ethical implications, lay summaries and other details. In the UK, about one in five of these are funded so the vast majority of proposals are therefore written and reviewed to end up in the 'back to the drawing board' folder.

So there’s quite a lot of independent scrutiny of research, beginning and end. There’s also evaluation by committees of the ethical acceptability of research involving human subjects, of experimental animal use, biological safety, radiation safety, career progression of researchers (including external peer reviewers for promotion to reader or professor).

On top of this there are a patchwork of additional review processes, to further various policy objectives. For UK universities there is the Research Excellence Framework (REF) in which universities have to painstakingly document their research 'environment',  ‘outputs’ (publications) and ‘impacts’ (narratives of how research has had effects outside academia) every five years, and there is the new Knowledge Exchange Framework (KEF).

It’s important that the public know how thoroughly research is scrutinised. UK research is extremely successful, so you can’t argue that it’s all wrong. We assess the same research multiple times and use quite a large portion of the research effort available to do so. If this is the cost of having curiosity-driven research, it’s probably a reasonable bargain, though some simplification would be very welcome!