A Covid natural experiment

From A Covid Border Battle by John Hinderaker.

Natural Experiments are a wonderful thing. From the Wikipedia article.

A natural experiment is an empirical study in which individuals (or clusters of individuals) are exposed to the experimental and control conditions that are determined by nature or by other factors outside the control of the investigators. The process governing the exposures arguably resembles random assignment. Thus, natural experiments are observational studies and are not controlled in the traditional sense of a randomized experiment. Natural experiments are most useful when there has been a clearly defined exposure involving a well defined subpopulation (and the absence of exposure in a similar subpopulation) such that changes in outcomes may be plausibly attributed to the exposure. In this sense, the difference between a natural experiment and a non-experimental observational study is that the former includes a comparison of conditions that pave the way for causal inference, but the latter does not.

Natural experiments are employed as study designs when controlled experimentation is extremely difficult to implement or unethical, such as in several research areas addressed by epidemiology (like evaluating the health impact of varying degrees of exposure to ionizing radiation in people living near Hiroshima at the time of the atomic blast) and economics (like estimating the economic return on amount of schooling in US adults).

Some of our best knowledge comes from a clever researcher who recognizes the conditions of a natural experiment and explores the outcomes. It is a special gift to recognize the conditions of a natural experiment. It is like recognizing negative space.

Hinderaker is pointing out that there are some natural experiments occurring at the moment in the US relevant to Covid-19.

In the ongoing debate over whether shutdowns have been useful, a comparison of Minnesota and Wisconsin is a valuable data point. These two adjoining states are of comparable population, demographics, history and geography. A Wisconsinite is basically a Minnesotan without the smugness.

It would be interesting to know if there are any marked differences in addition to their many similarities. Hinderaker does not mention that there are.

I have an impression of Minnesota being more urbanized and having more of a services economy than Wisconsin. A quick glance at Indexmundi supports that but the percentage differences are relatively small. Minnesota households do have about a 15% greater income than in Wisconsin but both are very close to the national average.

The only difference I can think of is that while their racial compositions are quite similar, Minnesota has 8.4% foreign born versus 5%. In addition, the Minnesotan foreign born are dominated by Somalis.

But other than that one variable, foreign born immigrants, the two states are pretty identical, so a reasonably meaningful natural experiment. With most variables being so similar, we would expect that Covid-19 would have a similar impact.

What about policies - how similar have their policies been?

On the coronavirus, the states parted company on May 13, when the Wisconsin Supreme Court struck down that state’s “Safer at Home” order. Minnesota, meanwhile, continued under a lockdown, eventually in modified form, to the present day. To an observer, the difference is obvious: Wisconsin is open for business. Minnesotans cross the St. Croix to eat out and hang out in the restaurants and bars on the Wisconsin side of the river. Wisconsin isn’t quite South Dakota, but compared with Minnesota it is a bastion of freedom.

OK. Similar populations and conditions but contrasting policy responses. This is a natural experiment we can get our teeth into.

Doing a mobility comparison between Wisconsin and Minnesota, we can see that Minnesotans did indeed constrict their mobility (proxy for reduced interaction) to a much greater degree than did Wisconsinites. And are still doing so. So what did the more interventionists policies of Minnesota and the much reduced mobility gain them? Back to the natural experiment.

Figures from the Minnesota Department of Health and Wisconsin Department of Health Services show that, from May 14th to July 27th, Wisconsin suffered 472 Covid-19 deaths and Minnesota suffered 939, as seen on Figure 1. Again, given the two state’s broadly similar populations – 5.6 million in Minnesota and 5.8 million in Wisconsin – that means a much higher rate of Covid-19 deaths in our state as well as a much higher number. Indeed, between May 14th and July 27th, Wisconsin saw 81 Covid-19 deaths per million residents. In Minnesota, we saw 167 Covid-19 deaths per million residents – a rate 2.1 times higher.

The interventionist policies created a death rate more than twice that of the noninterventionist policies.

That is similar to what has happened with two much larger states, Florida (21 million people) and New York (19 million people). New York has so far, with very interventionist policies, had a death rate of 178 deaths per million whereas as Florida has had a death rate of 28 per million.

It is too simplistic to then conclude that obviously high intervention lockdowns don't work. There are material differences between the states including latitude, average temperatures, ethnic mix, differences in countries of origin of the foreign born, economic attributes, degree of urbanization and density, etc. Two very different states, with two very different policy responses, had two very different outcomes. That could be for innumerable reasons.

But two very similar states, with two very different policy responses, and two very different outcomes. That's a natural experiment. It becomes much more likely that the different outcomes are due to the two different policy responses.

Do lockdowns work? We still don't know, but this is one more element in the increasingly populated "Probably Not" column.

And there is actually one more element of learning that could be useful. Perhaps it is more about which interventions than the degree of intervention.

The obvious contrast between Florida and New York has been that Florida has been much less interventionist. But another contrast is in specific policy. Florida in general has been low intervention but it has been high intervention around vulnerable populations. It protected its nursing homes and the elderly.

New York, in contrast, has been high intervention across the board. Except around protecting the vulnerable. In fact, this was one of the more manifest policy failures. Early in their response, New York moved contagious but low symptom Covid patients out of hospitals and into assisted living facilities in order to maintain hospital capacity for a wave that never occurred. They put the Covid contagious in with the ill and elderly.

It was a slaughter. Regrettably, as happens with more centralized and controlling governance, New York ignored the evidence of the tragedy far longer than was warranted and only very late in the game, once the worst had occurred, did they change their failed policy.

It is not just a matter of low versus high intervention but also smart and adaptive intervention versus maladroit and inflexible intervention. Florida has had low intervention policy which has been very targeted, very smart, and very adaptive. New York, not so much.

It appears that, on a smaller scale, something similar has been happening between Minnesota and Wisconsin.

Minnesota’s terrible COVID performance is due mostly to the fact that nearly 80% of its fatalities have been in nursing homes and assisted living facilities. Minnesota has discharged infected patients directly into such long term care facilities, a practice that Governor Walz, apparently unwilling to admit error no matter the human cost, continues to this day. (For an in-depth discussion of Minnesota’s COVID disaster, go here, or else read Scott’s multi-part series on Coronavirus In One State.)

The link in article is to an hour-long discussion that is very local, very data-based and very informative.

Overall though, this is to me, a very interesting deepening of the simplistic Florida - New York comparison.

It is also relevant to the broader clash of world visions between those who see wisdom in crowds and always wish to default in an uncertain decision environment towards freedom and those who wish to subject themselves to the dictates of the "experts."

The latter view is often articulated as "follow the science." A simplistic, arrogant, and duplicitous stricture if there ever was. There is scarcely any science yet around Covid-19. Lots of scientists are working on it and we are beginning to learn things but in terms of meaningful science supporting consequential decisions where we can anticipate with high confidence that there will be positive outcomes? We have barely cleared the starting line.

In that environment of uncertainty, what we are really contrasting is whether you let the community decide or you let the centralized statists decide on behalf of everyone. So far, with Wisconsin and Florida, the evidence is tilting towards low intervention policy (wisdom of crowds), very targeted intervention (do a few things well rather than many things poorly), very informed intervention (go for root causes), and very adaptive (let evidence drive policy).