Thursday, October 29, 2020

The results, published in Science in 2015, made bitter reading: in the end, only 39 per cent of the studies were judged to have replicated successfully.

From Science Fictions: How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth by Stuart Ritchie.  Page 31. 

As you might expect, the confluence of failed replications (like the priming studies) and bizarre results (like Bem’s paranormal discoveries), along with revelations of misrepresentation (like Zimbardo’s experiment) and fraud (like Stapel’s fake data) spooked psychologists. Just how many of the studies in their field, they wondered, could be trusted? To get an idea of how bad things were, they started banding together to run large-scale replications of prominent studies across multiple different labs. The highest profile of these involved a large consortium of scientists who chose 100 studies from three top psychology journals and tried to replicate them. The results, published in Science in 2015, made bitter reading: in the end, only 39 per cent of the studies were judged to have replicated successfully.  Another one of these efforts, in 2018, tried to replicate twenty-one social-science papers that had been published in the world’s top two general science journals, Nature and Science.  This time, the replication rate was 62 per cent. Further collaborations that looked at a variety of different kinds of psychological phenomena found rates of 77 per cent, 54 per cent, and 38 per cent.  Almost all of the replications, even where successful, found that the original studies had exaggerated the size of their effects. Overall, the replication crisis seems, with a snap of its fingers, to have wiped about half of all psychology research off the map.

Ritchie provides some mitigation to the failure rates but I think it is pretty safe to assume that 50% of psychology and sociology findings are outright wrong.   

No comments:

Post a Comment