Using single methodologies to do univariate analysis of multivariate systems requiring expansive specification curve analysis

From The Definitive Analysis of Observational Studies by John Mandrola.  The subheading is Buckle up for this Study of the Week. It shocked me. You may never read another observational study in the same way.

The original stucy is Grilling the data: application of specification curve analysis to red meat and all-cause mortality by Yumin Wang, et al.  

I observe with some frequency that much of our analysis in academia, in policy, and in random life is often univariate analysis of multivariate systems.  There are many causal elements and we explore only one.  What we find and conclude may be correct to an extent and under particular circumstances but it is in no way usefully true because our comprehension is markedly limited.  

This study takes a different approach.  Instead of taking one method of analysis, it wants to know what we would find if we were to use many methods of analysis.  A very relevant question in the context of the univariate analysis issue.  

Specification curve analysis is similar to a multiverse analysis, meaning it’s a way of defining and implementing all plausible and valid analytic approaches to a research question. This time in nutritional epidemiology.

Take a moment and think about the methods section of a standard association study. Say blueberries and rates of stroke. The authors of such papers will write that we analyzed the data in this way. In other words: one way.

But. But. There are, of course, many choices of ways to analyze the data.

Since most observational studies are not pre-registered, you can imagine a scenario where authors actually did a number of analyses and published the one that yielded an association with a p-value of less than 0.05.

Read the post for the whole picture about their own methodology.  The outcomes?

The results also provide a sobering view of nutritional epidemiology. Of 1200 different analytic ways (specifications) to approach the NHANES data, only 48 yielded significant findings. The vast majority found no significant association.

I would extend this paper beyond nutritional epidemiology. I mean, every time we read an observational study, in any area of bio-medicine, the authors tell us about their analytic method. It’s one method. Not 1200, or a 10 quadrillion.

Now consider the issue of publication bias wherein positive papers get published and null papers not so much.

Take the example of this paper.

There were 40 specifications that yielded a favorable red meat-mortality association and 8 that yielded a negative association. Red meat proponents could publish a positive one; vegetarian proponents could publish a negative one.

Nutritional epidemiology is one of those fields which is hugely multivariate.  Red meat effect on mortality?  What meat?  How much?  What age of the person consuming?  Male or female?  Comorbidities?  How prepared?  What medications being taken?  What time of day consumed?  On and on.  

Multivariate and many ways of studying, many specifications.  

And most of what we do with multivariate systems and many specification approaches is to fall back on a single study methodology using a univariate approach.

No wonder we get so many things wrong.  

What this study suggests, indirectly, is that for our most complex systems (sociology, diet and nutrition, public health, economy, etc.), the research we have been conducting for fifty years and more is underpowered (too small and not random), unsophisticated (univariate instead of multivariate), and unrigorous (single methodology instead of multiple methodologies.)  

Underpowered, unsophisticated, and unrigorous - pretty weak foundations for the coercion and propaganda usually attendant to the cherry picked headlines and biased conclusions reached by the Mandarin Class from the existing studies.

This paper sheds light on just how much greater needs to be the humility of decision makers in the face of complex systems.

And, I would argue, how more deference we ought to pay to the culture and class heuristics which have come down to us through the testing of time.