Five vital signs of research

From ‘TrUsT tHe ScIeNcE!’ – Study Retractions Up 13,650% in 22 Years by Jack Montgomery.  

The number of scientific papers retracted annually rose from just 40 in 2000 to almost 5,500 in 2022, representing a whopping 13,650% change over the past 22 years, with researchers estimating an astonishing 100,000 would have to be withdrawn every year with more thorough vetting. 

Delivering a blow to the “trust the science” cheerleaders, Retraction Watch’s co-founders Ivan Oransky and Adam Marcus detail the alarming issues with modern science for the left-wing Guardian newspaper. 

The surge in bogus papers is driven in part by the fact that scientists are often “required… to publish papers in order to earn and keep jobs or to be promoted,” which leads to some turning to so-called “paper mills” that “sell everything from authorships to entire manuscripts to researchers who need to publish lest they perish.” 

Only around a fifth of retractions are a result of “honest error,” Oransky and Marcus note, highlighting serious misconduct cases such as that of Joachim Boldt, a German anesthesiologist whose falsified data on an ineffective blood substitute was once widely cited and led to many people being harmed. 

A related issue is the so-called replication crisis. It has become increasingly apparent that the results found in many scientific papers – possibly a majority of them – cannot be reproduced by other researchers. In 2015, for example, efforts to reproduce psychology studies published in supposedly high-quality journals failed 61 out of 100 times, with similar results in 2018.

Reading forces me to recognize that I often conflate the two issues in my own mind.  Broadly, I assume in most cases of failure to replicate, there is a retraction.  But thinking about it, I realize the Venn diagram is not that straight forward.  

There are retractions which occur for reasons other than non-replication.  There are failures to replicate which do not lead to retractions.  There are papers which are simply not read or even attempted to be tested.  

What is the percentage of papers which, in a given year, we ought to anticipate can be trusted to not involve fraud, error, or shoddy research (failure to replicate)?  I don't know.

The rising failure to replicate numbers along with the rising number of retractions indicates a systemic issue of large scale but I don't know that from those numbers we can reverse engineer our way into an estimate of trustworthiness.

The alternative approach would be to exclude all studies which:

Are not pre-registered.

Have an unclear methodology.

Have too small a sample or population size (underpowered).

Have too small an effect size.

Have identifiable independence issues.

These are analogous to the basic vital signs in a doctors visit.

Blood Pressure

Oxygenation

Respiration rate

Temperature

Pulse rate

Vital signs are no guaranty of health but they are pretty reliable macro-indicators.  Similarly, pre-registration, etc. does not guaranty usefully accurate results from studies, but in combination, the five red flags are a reasonable approximation of reliability.

And how many academic studies are pre-registered, with clear methodologies, no conflicts of interest, are adequately powered and have strong effect sizes.  Fewer than 1% is the estimation I have seen across a number of sources.  These five disciplines are simply not routinely followed which I think most people would find very surprising.

This is no repudiation of empirical rationalism or the scientific method.  They remain the North Star for understanding the complexity of our world.  This is merely an overdue call to diligence.  We expect research to have the bare minimum of scientific methodological integrity and they do not and we should know that and assess things accordingly.

More important, it is a call for improvement.  We need more empirical rationalism and better adherence to the scientific method.  And we should expect more from our academicians and institutions of learning.