Not absolutely significant but very probably so

From Science Fictions by Stuart Ritchie.  Page 159.   

A common form this spin takes is the weasel-wording that scientists use about non-significant p-values. Recall from Chapter 4 that you normally need to have p < 0.05 for your effect in order to declare it ‘statistically significant’. The statistician Matthew Hankins has amassed a collection of genuine quotes from published papers where p-values remained stubbornly above that threshold, but whose authors clearly had a strong desire for significant results:

•  ‘a trend that approached significance’ (for a result reported as ‘p < 0.06’)

•  ‘fairly significant’ (p = 0.09)

•  ‘significantly significant’ (p = 0.065)

•  ‘narrowly eluded statistical significance’ (p = 0.0789)

•  ‘hovered around significance’ (p = 0.061)

•  ‘very closely brushed the limit of statistical significance’ (p = 0.051)

•  ‘not absolutely significant but very probably so’ (p > 0.05) 

 

There’s a whole literature of studies by scientific spin-watchers, each of them highlighting spin in their own fields. 15 per cent of trials in obstetrics and gynaecology spun their non-significant results as if they showed benefits of the treatment.  35 per cent of studies of prognostic tests for cancer used spin to obfuscate the non-significant nature of their findings.  47 per cent of trials of obesity treatments published in top journals were spun in some way.  83 per cent of papers reporting trials of antidepressant and anxiety medication failed to discuss important limitations of their study design. A review of brain-imaging studies concluded that hyping up correlation into causation was ‘rampant’.  Some forms of spin shade into fraud, or at least gross incompetence: a 2009 review showed that, of a sample of studies published in Chinese medical journals that claimed to be randomised controlled trials, only 7 per cent actually used randomisation. 

 

Even meta-analyses aren’t safe, as we’ve seen before. A 2017 review of meta-analyses on diagnostic tests (for example, blood tests for Alzheimer’s disease) found that 50 per cent of them drew a positive conclusion about how well the test worked despite finding trivial, statistically non-significant effects in their analyses. The spin, the review concluded, ‘could lead to unjustified optimism about the performance of tests in clinical practice.’ This seems to be another example of how scientists’ urge to hype their results misleads the people who most rely on them.