The epistemological edifice is crumbling

Well this should have set the cat among the pigeons.

From How Sure Should We Be About Healthcare Interventions? by John Mandrola.  The subheading is The Study of the Week should reset our prior beliefs about the success of healthcare interventions.

We have become accustomed to the fact that only 20-30% of psychology, sociology, etc. academic findings actually replicate.  That took a while to sink in but people are beginning to accept that there is a lot of wooliness in academic research in the softer sciences.

The same thing has begun happening in a much sterner science, i.e. medicine.  Every couple of months it seems like some foundational assumption or treatment is overturned.  For example, there is the very recent kerfuffle about the effectiveness of colonoscopies which I blogged about here.  A medical treatment which was simply assumed to be effective until actually tested.

Mandrola is writing about some research published a few months ago which looked at the research on the effectiveness of some 1,567 treatments.  From Most healthcare interventions tested in Cochrane Reviews are not effective according to high quality evidence: a systematic review and meta-analysis by Jeremy Howick, et al.  From the Abstract:

Of 1,567 eligible interventions, 87 (5.6%) had high-quality evidence supporting their benefits. Harms were measured for 577 (36.8%) interventions. There was statistically significant evidence for harm in 127 (8.1%) of these. Our dependence on the reliability of Cochrane author assessments (including their GRADE assessments) was the main potential limitation of our study.

Conclusion

More than 9 in 10 healthcare interventions studied within recent Cochrane Reviews are not supported by high-quality evidence, and harms are under-reported.

There are some methodological quibbles that could be made but this looks reasonably robust.  But if so, oh my goodness.

Only 6% of medical treatments were confidently believed to have benefits.  8% of interventions were believed to have harmful outcomes.  

It is not quite so bad as that stark conclusion suggests.  Just because the intervention does not have high quality evidence for a positive impact doesn't mean that possibly most of them are beneficial.  But we certainly don't know and shouldn't be confident.  My suspicion is that many interventions are indeed beneficial but perhaps with narrower application than currently accepted. 

Be sure to ask your doctor.