So few small scale projects scale successfully

Dylan Matthews in Hey Congress: Pre-K is a better investment than the stock market refers to the rate of return generated by a high-cost pre-school program as calculated by James Heckman, an economist whose work I greatly admire.

Heckman and his coauthors started from the observation that those who received preschool in the Perry experiment ended up earning more money — and thus paying more taxes — as well as using fewer criminal justice system resources (because they committed fewer crimes) and receiving less in the way of welfare, food stamps and other transfer payments. He tried to determine the annual return on investment, using those cost reductions to society, and to the government in particular, as benefits and comparing them to the upfront $18,000 a year cost.

He found that the annualized rate of return was somewhere between 7 percent and 10 percent. For comparison, historically the stock market has grown an average of 5.8 percent each year. Bonds have grown less. By this calculus, early childhood education is a better investment than either. And those benefits compound over time. If you assume the conservative estimate of 7 percent annual returns, that’s a sixty-fold increase over 60 years. Abecedarian was similarly successful. Thirty-six percent of recipients of early childhood ed going to college, compared to 14 percent of non-participants.

What’s going on here? Heckman’s theory, which he explained both in an essay in “Boston Review” and an episode of “This American Life,” is that preschool programs help kids develop “non-cognitive skills.” Traditionally, we think of school as imparting book smarts: adding and multiplying, reading and writing, foreign languages, etc. But especially for young students, they also teach skills like patience, cooperation, planning and delaying gratification. Over time, those basic self-discipline skills become immensely valuable, enabling other learning and making students better participants in institutions like schools and workplaces.

I think Heckman is likely right about cause-and-effect.  Good non-cognitive skills lead to better life outcomes.  The question isn't necessarily about the causation per se (though there is plenty for discussion in that sand-box).  I think the real issue is in magnitudes.  It is not enough to show that something is net beneficial.   You have to show that it is net beneficial to a greater degree than alternative uses of that time and money.  I think there are very fair questions to be asked about the confidence we have in the 7-10% number as well as whether the benefits are greater than the average benefit we might expect from some other usage of that capital. 

There are plenty of reasons to be cautious about the analysis. Avoided costs is always a dodgy argument to make. The sample is small. Education and sociology are rife with small scale trials that later turn out to not be scalable, i.e. the results show up in carefully controlled experiments but as soon as you expand the program into the real world, the benefits disappear. Head Start is a good example of what turned out to be a non-scalable program. $10 billion a year is being spent on a well intended and perfectly logical program which is not yielding (and has not yielded for decades) any of the results  which were anticipated.

But what might be the root cause of near universal failure between trial and real-world expansion. The 7-10% rate of return was what caught my eye. Generally, the larger the amount of capital at stake, the more the number of people to be affected, and the greater the number of independent variables involved, the more confident you have to be in the rate of return and in particular, the larger it has to be to account for risk. 7-10% is a very low number, particularly when it includes avoided costs. Avoided costs tend to be the category of benefits least likely to occur in business transformations and government programs. People forget about them, don't measure them, and most often, they simply don't occur. So the likely benefit from the Perry Preschool Project is much lower than 7-10%. How much lower we can't say because I don't see a breakout of the useful data such as average increase in participant's income as a class of benefits versus the avoided costs from reduced crime and incarceration. If the latter are a significant portion or majority of the benefits captured in the 7-10% range, then it is likely that the real ROI is much less, perhaps 3-5%.

3-5% ROI is at least positive but is it real and is it enough to cover the risks attendant to a major capital investment?

I don't know. However, the other unaddressed element that is usually not discussed, has to do with a different aspect of the fact that the projects are so small (58-65 students). Most Randomized Controlled Trials (RCT) are prone to various confounding effects (and it should be noted that the Perry Preschool Project was not an RCT). The confounding effects have various terms and slightly differing causes and effects. The Hawthorne Effect (simply observing workers changes their productivity), the Halo effect (participants are influenced by self-perceived celebrity), the Placebo effect, and the Pygmalion effect (the expectations of those conducting the experiment affect the outcomes). My question - is there a standard size of these effects?

I am most familiar, from an industrial engineering perspective, with the Hawthorne effect but I have never seen it quantified. A quick web search does not yield an answer. Searches for the size of the Halo, Placebo and Pygmalion effects seem to indicate that on average, each of those effects drive a 30% increase in productivity independent of the nature of the trial. The Hawthorne, Pygmalion, Placebo and other effects of expectation: some notes by Stephen W. Draper is one source.

The net is that, in addition to the issues of unrepresentative sample mixes, small size, non-scalability, etc., unless the desired return or impact is very large, the likelihood of attaining material net returns when scaling from small trial to mass deployment are low to nonexistent.

The precision of a 7-10% ROI is beguiling but there is likely much less there than meets the eye, in part because of the unreliability of the avoided cost numbers and in part because of the aggregate impact of Placebo, Pygmalion, Hawthorne and Halo effects. That would explain why so few small scale projects scale successfully.

Years ago, a couple psychologists, Betty Hart and Todd Risley did a study to determine the magnitude of the impact of the spoken word on children's cognitive development and academic success (Meaningful Differences in the Everyday Experience of Young American Children)

Major findings:

• Children from all three groups of families started to speak around the same time and developed good structure and use of language.
• Children in professional families heard more words per hour, resulting in larger cumulative vocabularies.
• In professional families, children heard an average of 2,153 words per hour, while children in working class families heard an average of 1,251 words per hour and children in welfare-recipient families heard an average of 616 words per hour. Extrapolated out, this means that in a year children in professional families heard an average of 11 million words, while children in working class families heard an average of 6 million words and children in welfare families heard an average of 3 million words. By age four, a child from a welfare-recipient family could have heard 32 million words fewer than a classmate from a professional family.
• By age three, the observed cumulative vocabulary for children in the professional families was about 1,100 words. For children from working class families, the observed cumulative vocabulary was about 750 words and for children from welfare-recipient families it was just above 500 words.
• Children in professional families heard a higher ratio of encouragements to discouragements than their working class and welfare-supported counterparts.

This project also only involved some 40 or 50 families. I introduce it not to disavow the results from either Hart & Risley or the Perry Project. Only to suggest that there needs to be materiality, trade-off balancing, ease of implementation considerations and consistency.

The Perry Project would, simplistically, call for a doubling of national investment in early childhood education, some hundreds of billions of dollars. All on a single, small scale, non-replicated study.

The Hart & Risley study, simplistically, would indicate that we could achieve a 200% or greater improvement in academic achievement (with attendant beneficial life outcomes in terms of employability, income, etc.) simply by coaching parents to speak with their children more and more effectively. In other words, a dramatically higher rate of return at a dramatically lower cost.

Both projects have the same basis for support, which no matter how much you might agree with them, is weak and inadequate.