Monday, January 25, 2021

The sizeless stare of statistical significance

From Science Fictions by Stuart Ritchie.  Page 206.   

Making it easier for scientists to publish replications and null results might reduce publication bias. But what about the other forms of bias we encountered, having to do with p-hacking? Many dozens of papers, and even entire books, have been written on the pitfalls of p-values: they’re hard to understand, they don’t tell us what we really want to know and they’re easily abused.  There’s truth to all these criticisms. In broad terms, what is needed is less focus on statistical significance – a p-value below the arbitrary threshold of 0.05 – and more on practical significance. In a study with a large enough sample size (and high enough statistical power), even very small effects – for example, a pill reducing headache symptoms by one per cent of one point on our 1–5 pain scale – can come up as statistically significant, often with p-values far below 0.05, though they could be essentially useless in absolute terms. The economists Stephen Ziliak and Deirdre McCloskey call this the ‘sizeless stare of statistical significance’, where scientists develop a laser-like focus on p-values at the expense of considering, as Ziliak and McCloskey put it, the ‘oomph’ of their effect.

 

No comments:

Post a Comment