We (still) really don't know what is going on here

An excellent post integrating our knowledge of cause, consequence and impact of Covid-19 eighteen months after it burst onto the world stage.  From Lockdown Effectiveness: Much More Than You Wanted To Know by Scott Alexander.

Alexander is thorough in his analysis and logical in his parsing of the data.  

And I think the answer is still - "We really don't know what is going on here."

We are better informed but the data is so ambiguous we can read just about anything into most of it.  

I would argue some things differently than Alexander but we seem to be reading similar sources of data.  He treats all causes mortality as a lesser measure than I think is warranted.  He doesn't emphasize the vast definitional distinctions of died with Covid-19 and died from Covid 19 between states and countries as I think should be done.

He does not discuss, but his piece prompts the observation that measuring the latent reservoir of high morbidity individuals or population groups might be a clarifying activity.  In other words, for a jurisdiction, the percent of population that is aged as well as the percent of the population at particular risk owing to significant co-morbidities is likely predictive of both death rates and the progression of the pandemic.

Alexander has an excellent discussion on CoronaGame, a game which allows users to try and optimize the range of choices (lockdowns, treatments, etc.) against outcomes such as cases, deaths, and economic damage.  

CoronaGame gives you the role of a Czech health official charged with choosing when to initiate vs. stop various preventative measures. You get points for preventing coronavirus cases and deaths, but also for minimizing economic costs and damage to public morale. The game’s model is based on the Brauner et al data, so if you believe the paper it should be a pretty realistic look at how different policies would have worked.

Here’s a graph of how players have done (economic damage on the vertical axis, deaths on the horizontal).

Click to enlarge.

In a blog post, one of the epidemiologists who worked on the game talks about how you can view this as a Pareto frontier:

Click to enlarge.

…ie a line of choices representing a particular tradeoff made as effectively as possible. Point A, at the top of the frontier, represents a perfectly effective planner trying to maximize lives saved, without worrying about cost (except that if two plans save the same number of lives, they will use the lowest cost as a tiebreaker). Point B, at the bottom, presents the opposite - minimizing cost perfectly effectively, using lives saved only as a tiebreaker. The rest of the yellow line represents trading off between lives and cost at some exchange rate, while maintaining perfect effectiveness; the higher the exchange rate, the further along the line you go. In a perfect world, we would be debating where on the yellow line to be.

Click to enlarge.

In the real world, where we aren’t perfectly rational and efficient planners, it’s totally possible to end up arbitrarily far away from the Pareto frontier, eg the points in red. These represent various plans that are not the result of a perfectly effective planner using some tradeoff between lives and cost - in other words, somebody screwed up. They are bad plans which could have saved more lives while not costing any more money, or else could have saved more money without costing any more lives. For example, Point I is what you get if you just institute every single restriction you can think of, and refuse to ever lift any of them.

(the real Czech Republic ended up at Point C)

One of the morals that the designers of this game were trying to drive home is that the discussion we’re having in this post is unvirtuous. It’s political point-scoring - did Team Pro-Lockdown win more glory than Team Anti-Lockdown, or vice versa? The real question we should be asking is what set of policies countries should have implemented.

On my best playthrough of CoronaGame, I instituted all possible restrictions throughout March, April, and May, kept cases controlled enough to institute test-and-trace, switched to a pretty minimal suite of restrictions in the summer, then instituted school closures, gatherings <10, and high-risk business closures during the winter, relaxing them gradually as more people got vaccinated (I still didn’t quite make it to the Pareto frontier, so there’s room for improvement on this strategy). Trying these same restrictions but at different times turns out to be a disaster, which really emphasizes the point that there’s no single “best level of lockdown”. The right thing to do is whatever combination of policies in whatever order gets you on the Pareto frontier in CoronaGame (I still haven’t figured it out!), which means instituting the exact right set of restrictions at the exact right times to minimize cases, then lifting them at the exact right times to minimize cost.

Then, once you’re on the Pareto frontier, you can argue about which direction on that frontier to move - a little further toward stricter lockdowns / more lives saved, or toward looser lockdowns / fewer lives saved. If you’re not on the frontier - if you’re arguing about whether to be at Point F vs. Point G - probably you should shut up and work on getting on the frontier first.

Excellent discussion and worth visiting the original article for the links.

Alexander concludes the voluminous discussion with what I think is the most salient point.

7: All of this is very speculative and affected by a lot of factors, and the error bars are very wide.

Which I characterize as "We really don't know what is going on here." 

The overall sense I get from reading the piece is that we still don't have the measurement specificity, reliable data collection, and the totality of measured context to reach strong conclusions on just about anything related to Covid-19.