Plausible does not mean probable

From A Trick For Higher SAT scores? Unfortunately no. by Terry Burnham.

A great example of the debunking of cognitive pollution.

Wouldn’t it be cool if there was a simple trick to score better on college entrance exams like the SAT and other tests?

There is a reputable claim that such a trick exists. Unfortunately, the trick does not appear to be real.


This is the story of an academic paper where I am a co-author with possible lessons for life both inside and outside the Academy.

In the spring of 2012, I was reading Nobel Laureate Daniel Kahneman’s book, Thinking, Fast and Slow.

Professor Kahneman discussed an intriguing finding that people score higher on a test if the questions are hard to read. The particular test used in the study is the CRT or cognitive reflection task invented by Shane Frederick of Yale. The CRT itself is interesting, but what Professor Kahneman wrote was amazing to me,

“90% of the students who saw the CRT in normal font made at least one mistake in the test, but the proportion dropped to 35% when the font was barely legible. You read this correctly: performance was better with the bad font.”

I thought this was so cool. The idea is simple, powerful, and easy to grasp. An oyster makes a pearl by reacting to the irritation of a grain of sand. Body builders become huge by lifting more weight. Can we kick our brains into a higher gear, by making the problem harder?

[snip]

Roughly 3 years later, Andrew Meyer, Shane Frederick, and 8 other authors (including me) have published a paper that argues the hard-to-read presentation does not lead to higher performance.

The original paper reached its conclusions based on the test scores of 40 people. In our paper, we analyze a total of over 7,000 people by looking at the original study and 16 additional studies. Our summary

Easy-to-read average score: 1.43/3 (17 studies, 3,657 people)
Hard-to-read average score: 1.42/3 (17 studies, 3,710 people)

The data suggest that Malcolm Gladwell’s statement is false. Here is the key figure from our paper with my annotations in red:

Burnham's three take-aways are:

1. Beware simple stories.
2. Ideas have considerable “Meme-mentum”
3. We can measure the rate of learning.

I would emphasize the first one (Plausible does not mean probable) and add a fourth - Acknowledge but never rely on studies with few participants.