No psychological progress

Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature by Denes Szucs and John P. A. Ioannidis.

Ouch. Part of the continuing avalanche of evidence that much of the effort invested in neuroscience and psychology has been wasted. Or, at least, non-illuminating and misleading. From the abstract:

We have empirically assessed the distribution of published effect sizes and estimated power by analyzing 26,841 statistical records from 3,801 cognitive neuroscience and psychology papers published recently. The reported median effect size was D = 0.93 (interquartile range: 0.64–1.46) for nominally statistically significant results and D = 0.24 (0.11–0.42) for nonsignificant results. Median power to detect small, medium, and large effects was 0.12, 0.44, and 0.73, reflecting no improvement through the past half-century. This is so because sample sizes have remained small. Assuming similar true effect sizes in both disciplines, power was lower in cognitive neuroscience than in psychology. Journal impact factors negatively correlated with power. Assuming a realistic range of prior probabilities for null hypotheses, false report probability is likely to exceed 50% for the whole literature. In light of our findings, the recently reported low replication success in psychology is realistic, and worse performance may be expected for cognitive neuroscience.

Ioannidis made his first major expose of how weak were many reported scientific findings in a 2005 paper Why Most Published Research Findings Are False.

I was hoping that in this new paper, when they said recently published, it might include earlier data. Regrettably no. Their sample of papers is from 2011 to 2014, at least seven years after the first clear red flag was raised. What this indicates to me is that, despite the mounting evidence of poor scientific controls, experiment rigor has not improved at all. A large portion of the neuroscience and psychology scientific community is producing non-science.

As Max Planck said in 1906:

A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.

More commonly paraphrased as "Science advances one funeral at a time." Szucs and Ioannidis's findings are consistent with Planck's adage.