Friday, December 25, 2020

The range of the data (the difference between the highest and lowest scores) was nearly identical, although the groups were otherwise quite different.

From Science Fictions by Stuart Ritchie.  Page 63.

This kind of reasoning is what caught out social psychologists Lawrence Sanna and Dirk Smeesters in 2011. Sanna published a study in which he claimed to find that people are more prosocial when standing at higher elevations; Smeesters claimed to show that seeing the colours red and blue affects how people think about celebrities.  The results in both papers looked impressive at first glance, easily confirming their proposed theories about human behaviour. But a closer look revealed something distinctly odd. The psychologist Uri Simonsohn showed that in the various groups in Sanna’s experiment, the range of the data (the difference between the highest and lowest scores) was nearly identical, although the groups were otherwise quite different. Simonsohn calculated that the chances of this happening in real data were minuscule. It was the same for Smeesters, except it was the averages of his groups that were too similar; again, these similarities just weren’t consistent with what would happen in real data, where error would have nudged the numbers further apart.  Once these problems, among others, were exposed, the offending papers were retracted, and both researchers resigned in disgrace. 

These kinds of statistical red flags are analogous to  what makes your bank freeze your credit card after it’s suddenly used to spend large sums on a tropical cruise: unusual activity that’s out of line with normal expectations, and which might be due to fraud.  And there are a host of other features of fraudulent data that might cause readers to become suspicious when they dig into the details. The dataset might look a little too immaculate, for example, with too few missing datapoints, which come about for all sorts of reasons in real datasets: participants dropping out of the study or instruments failing, for example. Perhaps the distribution of numbers might not follow certain expected mathematical rules.  Or the effects might be vastly larger than seems plausible in the real world, and thus too good to be true.

 

No comments:

Post a Comment