Science - it is subject to change and refinement

From A Meta-Scientific Perspective on “Thinking: Fast and Slow" by Ulrich Schimmack.

2011 was an important year in the history of psychology, especially social psychology. First, it became apparent that one social psychologist had faked results for dozens of publications (https://en.wikipedia.org/wiki/Diederik_Stapel). Second, a highly respected journal published an article with the incredible claim that humans can foresee random events in the future, if they are presented without awareness (https://replicationindex.com/2018/01/05/bem-retraction/). Third, Nobel Laureate Daniel Kahneman published a popular book that reviewed his own work, but also many findings from social psychology (https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow).

It is likely that Kahneman’s book, or at least some of his chapters, would be very different from the actual book, if it had been written just a few years later. However, in 2011 most psychologists believed that most published results in their journals can be trusted. This changed when Bem (2011) was able to provide seemingly credible scientific evidence for paranormal phenomena nobody was willing to believe. It became apparent that even articles with several significant statistical results could not be trusted.

Only ten years and a whole field overturned.  It is not that everything published in psychology was unreliable.  Perhaps only 70% was unreliable.  But like the business executive who knows half his advertising budget is ineffective, the challenge is to know which half.  Some psychology research is reliable and we are now in the process of trying to find out which are the 30%.

Kahneman's book was the kind of chattering class phenomenon which comes along every few years.  Everyone talks about it but few actually read it.  

Stephen Hawking's A Brief History of Time in 1988 was a similar phenomenon.  Everyone, for the better part of a year, talked about it and 25 million copies were sold.  But there was a lot of evidence suggesting that it was both the most widely discussed and admired book of modern times and simultaneously the least read such book as well.

Schimmack goes through chapter by chapter of Thinking: Fast and Slow, applying statistical tests to arrive at conclusions of the viability of the studies addressed in that chapter.  

Table 1 shows the number of results that were available and the R-Index for chapters that mentioned empirical results. The chapters vary dramatically in terms of the number of studies that are presented (Table 1). The number of results ranges from 2 for chapters 14 and 16 to 55 for Chapter 5. For small sets of studies, the R-Index may not be very reliable, but it is all we have unless we do a careful analysis of each effect and replication studies.

R is a measure of robustness and reliability of the finding.

An R-Index below 50 implies that there is a less than 50% chance that a result will replicate.

Seven of the thirteen results can be dismissed as their R is below 50.  However, more chapters really ought to be given a failing grade because, even though their R is high, the population of studies is small.  Chapter Nine for example, has an almost respectable R at 57 but it is based on only even studies.  Way too few.  If you remove those with high R but few studies, you take out an additional three chapters.  Out of the examined thirteen chapters, ten would unlikely have been written today; the results were simply unbelievable.  

Kahnem was not intentionally being deceptive.  He is a scientist.  

Kahneman also started to wonder whether some of the results that he used in his book were real. A major concern was that implicit priming results might not be replicable. Implicit priming assumes that stimuli that are presented outside of awareness can still influence behavior (e.g., you may have heard the fake story that a movie theater owner flashed a picture of a Coke bottle on the screen and that everybody rushed to the concession stand to buy a Coke without knowing why they suddenly wanted one). In 2012, Kahneman wrote a letter to the leading researcher of implicit priming studies, expressing his doubts about priming results, that attracted a lot of attention (Young, 2012).

For all the chatter back when, it was mostly wrong or non-existent.   Policy implementations based on priming were incorporated into education and into all sorts of other programs.  And it was all hooey.  

Ultimately it is a very good reminder that all those advocating "Follow the Science" are wrong.  Yes, incorporate science and objective knowledge into your thinking.  But all science is only ever contingent.  Whatever you think you know, there is always an asterisk on it - it is subject to change and refinement.