Wicked systems - If your theory is weak and your data are noisy, your experiments won't tell you much

From I fear that many people are drawing the wrong lessons from the Wansink saga, focusing on procedural issues such as “p-hacking” rather than scientifically more important concerns about empty theory and hopelessly noisy data. If your theory is weak and your data are noisy, all the preregistration in the world won’t save you. by Andrew Gelman. That's a long post title.

What I want to focus on is the conclusion -

If your theory is weak and your data are noisy, all the preregistration in the world won’t save you.

I have said before that we are closing in on the low hanging epistemic knowledge. That which is obvious and easy to know, we have a pretty good handle on. Still more to be discovered but our progress is advanced. We know more and more about the more obvious facts of the world and the more deterministic processes.

The epistemic wild west are those issues attached to wicked systems (process predicate to wicked problems.) Wicked systems have a number of characteristics. They are:

Dynamic
Evolving
Interdependent between multiple systems
Chaotic
Non-linear
Non-obvious feedback loops
Homeostatic mechanisms

What systems have these sorts of attributes? Productivity, governance, education, affiliative groups, climate, innovation, financial markets, crime, ethics, violence, etc. Just about all systems which involve people are wicked systems. Forecasting in each of these arenas is characterized by low levels of accuracy, low sustainability of accuracy, absence of specificity and lack of precision.

In deterministic systems, you are eventually able to get to high degrees of precise and reliable forecasting.

So far, we have demonstrated no such capability with wicked systems. Wicked systems lack encompassing theories though not for want of effort.

Take national economic productivity as an illustrative example. Everyone wants to be a wealthy nation (high productivity.) How do we go about achieving that? There are dozens of explanatory theories, virtually all of them monocausal theories rather than encompassing theories. Culture, geographical location, resource endowment, culture, institutions, governmental structure, policies, power structures, property rights, historical path dependency, etc. In addition to these general theories, there are hundreds of micro-level theories. But none of those theories has accurate, reliable, enduring and precise forecasting capability. The nature of wicked systems are beyond our current crop of theories. Our theories are weak.

Correspondingly, our data is noisy. This seems counterintuitive, inundated as we appear to be in data.

But we are only a decade or two into the period of data ubiquity. We are still inexperienced at providing reliable context to data, data genealogy, chain-of-evidence purity of data, etc. While we have a lot of data, virtually all data associated with wicked systems is at least noisy (if not inaccurate, misunderstood, misinterpreted, corrupted, imprecise, irrelevant, and/or absent context.)

Wicked systems are the most interesting and pertinent to us but for virtually all of them, our theories are weak and our data is noisy. We are still, therefore, very inexperienced at running good experiments on wicked systems, experiments that will actually reliably move our knowledge frontiers forward.