I continue to think we put way too much stock in complex correlational studies

From Long-lasting Effects of Suspensions? by jkaufman.

I recently read "The School to Prison Pipeline: Long-Run Impacts of School Suspensions on Adult Crime" (Bacher-Hicks et. al. 2019, pdf, via Rob Wiblin) which argues that a policy of suspending kids in middle school leads to more crime as an adult.

Specifically, they found that after controlling for a bunch of things, students who attended schools with 0.38 more suspensions per student per year were 20% more likely to be jailed as adults:

A one standard deviation increase in the estimated school effect increases the average annual number of days suspended per year by 0.38, a 16 percent increase. ... We find that students assigned a school with a 1 standard deviation higher suspension effect are about 3.2 percentage points more likely to have ever been arrested and 2.5 percentage points more likely to have ever been incarcerated, which correspond to an increase of 17 percent and 20 percent of their respective sample means.

This is a very surprising outcome: from a single suspension in three years they're 20% more likely to go to jail?

Much research seems determined to show the value of an intervention rather than dispassionately trying to find what is true or not and what works or not.

But does the research really show what is claimed? Kaufman thinks not.

This sort of problem, where there's some kind of effect outside what you control for, which leads you to find causation where there may not be any, is a major issue for value-added models (VAM) in general. "Do Value Added Models Add Value?" (Rothstein 2010, pdf) and "Teacher Effects on Student Achievement and Height" (Bitler et. al. 2019, pdf) are two good papers on this. The first shows that a VAM approach yields higher grades in later years causing higher grades in earlier years, while the second shows the same for teachers causing their students to be taller.

THIS

I continue to think we put way too much stock in complex correlational studies, but Bacher-Hicks is an illustration of the way the "natural experiment" label can be used even for things that aren't very experiment-like. It's not a coincidence that at my day job, with lots of money on the line, we run extensive randomized controlled trials and almost never make decisions based on correlational evidence. I would like to see a lot more actual randomization in things like which teachers or schools people are assigned to; this would be very helpful for understanding what actually has what effects.