Effect size and publication bias: old small studies have big effects and new large studies have small effects

From Everything I Needed to Know (About Publication Bias), I Learned In (Pre-) Kindergarten by Gabriel.

I constantly grumble about newspapers and magazines who report on studies and never reveal either 1) the sample size on which the study was based and/or 2) the effect size. You get headlines such as "Eating vegemite extends life" but never find out that it is based on a sample size of 46 undergraduate students and the average increase in life is one day. Tell me the effect size.

Gabriel has an interesting angle on this, looking for publication bias (small studies showing large effects are privileged over larger studies with smaller effects). The particular issue at hand is pre-k education though the issues are the same across the spectrum. Pre-k seems such a logical no-brainer. It can't hurt can it? And even if it is not as effective as alternative programs, surely there has to be some benefit. The simple logic got a boost back in the late 60's and early 1970s when two studies indicated large effect sizes. The challenge was that both studies were small, one of 123 students and one of 100. Regrettably, we have seen nothing similar since then. The larger the study, the smaller the effect size, with the most recent two large studies actually showing a small negative impact (almost certainly spurious, but there is a logical argument that would support such an outcome).

The standard way to detect publication bias is through a meta-analysis showing that small studies have big effects and big studies have small effects. For instance, this is what Card and Krueger showed in a meta-analysis of the minimum wage literature which demonstrated that their previous paper on PA/NJ was only an outlier when you didn’t account for publication bias. Similarly, in a 2013 JEP, Duncan and Magnuson do a meta-analysis of the pre-K literature. Their visualization in figure 2 emphasizes the declining effects sizes over time, but you can also see that the large studies (shown as large circles) generally have much smaller β than the small studies (shown as small circles). If we added the Tennessee and Quebec studies to this plot they would be large circles on the right slightly below the x-axis. That is to say, they would fall right on the regression line and might even pull it down further.

This is what publication bias looks like: old small studies have big effects and new large studies have small effects.

I suppose it’s possible that the reason Perry and Abecedarian showed big results is because the programs were better implemented than those in the newer studies, but this is not “demonstrated definitively” and given the strong evidence that it’s all publication bias, let’s tentatively assume that if something’s too good to be true (such as that a few hours a week can almost deterministically make kids stay in school, earn a solid living, and stay out of jail), then it ain’t.