Friday, October 30, 2020

The problem with cognitive pollution is that it spreads, soiling all that it touches.

From Science Fictions: How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth by Stuart Ritchie.  Page 31. 

“Maybe it’s not quite that bad, for two reasons. First, we would expect some results that really are solid to fail to replicate sometimes, merely due to bad luck. Second, some replications might have failed due to their being run with slight changes to the methodology from the original (though if a result is fragile enough that it disappears after minor modifications to the experiment, one might wonder how useful or meaningful it really is). For these reasons, it’s sometimes tricky to decide whether a finding is ‘replicable’ or not based just on one or two replication attempts. What’s more, the replication rate seems to differ across different areas of psychology: for example, in the 2015 Science paper, cognitive psychology (studies of memory, perception, language, and so on) did better than social psychology (which includes the sorts of metaphor-priming studies we saw above).

In general, though, the effect on psychology has been devastating. This wasn’t just a case of fluffy, flashy research like priming and power posing being debunked: a great deal of far more ‘serious’ psychological research (like the Stanford Prison Experiment, and much else besides) was also thrown into doubt. And neither was was it a matter of digging up some irrelevant antiques and performatively showing that they were bad – like when Pope Stephen VI, in the year 897, exhumed the corpse of one of his predecessors, Pope Formosus, and put it on trial (it was found guilty). The studies that failed to replicate continued to be routinely cited both by scientists and other writers: entire lines of research, and bestselling popular books, were being built on their foundation. ‘Crisis’ seems to be an apt description.

Oh, I think it is quite that bad.  Possibly Ritchie's coda are relevant in the academic context.  Possibly.  But out in the broader world where weak minded wannabe thinkers fancy themselves well equipped to hold an opinion based on a widely hyped but failed-to-replicate study?  The problem with cognitive pollution is that it spreads, soiling all that it touches.   


No comments:

Post a Comment