Small sample sizes give you an idea of what might be true, but no idea as to what is true

From Lee Jussim.

Expectations DICTATE people's life's outcomes? That is so divorced from all known data on expectancy effects, stereotype threat, effects of implicit prejudice, etc. that its wrongness beggars belief. From my 2012 book. pic.twitter.com/pVEB86heiw

— Psych Rabble (@PsychRabble) February 2, 2018

The original claim to which he is responding is here.

Here is an example of an extreme claim made in a prestigious social psych source. I won't characterize it (yet) because it would be reasonable for someone to think I was caricaturizing it: pic.twitter.com/pEUbeJtHoX

— Psych Rabble (@PsychRabble) February 2, 2018

Shifting away from the specific argument, Jussim's first graph is, to me, a splendid example of the risk and reality of so many small sample size studies. They are prone to finding effects which are not real and do not manifest when subjected to normal tests with pre-registration protocols, large, randomized sample sizes, and rigorous analytical neutrality.

No matter how intuitive or interesting the finding, any research based on small sample sizes should, in my opinion, be given almost no evidentiary weight, even if they are consistent with other opinions and other research. All a small sample size project gives you is an idea of what might be true, but no idea as to what is true.