Very modest effects from self-affirmation intervention on educational achievement outcomes

An example of why all knowledge is contingent. From Can Short Psychological Interventions Affect Educational Performance? Revisiting the Effect of Self-Affirmation Interventions by Marta Serra-Garcia, Karsten T. Hansen, and Uri Gneezy.

From the Abstract:

Large amounts of resources are spent annually to improve educational achievement and to close the gender gap in sciences with typically very modest effects. In 2010, a 15-min self-affirmation intervention showed a dramatic reduction in this gender gap. We reanalyzed the original data and found several critical problems. First, the self-affirmation hypothesis stated that women’s performance would improve. However, the data showed no improvement for women. There was an interaction effect between self-affirmation and gender caused by a negative effect on men’s performance. Second, the findings were based on covariate-adjusted interaction effects, which imply that self-affirmation reduced the gender gap only for the small sample of men and women who did not differ in the covariates. Third, specification-curve analyses with more than 1,500 possible specifications showed that less than one quarter yielded significant interaction effects and less than 3% showed significant improvements among women.

The original researchers conducting the research did their job poorly.

The intervention did not improve women's lower performance but it did reduce the performance of high scoring men. It narrowed the gap by the group collectively performing more poorly.

Basically, this intervention failed but it was reported as effective.

In this case, the refutation of the old knowledge was contingent, not on new knowledge but on simply applying rigorous research techniques to the existing data.