Over-the-top enthusiasm is a tell for weak findings

An example of the sort of over-the-top absurdity of the mainstream media creating cognitive pollution.

This is an astoundingly good summary by ⁦@dylanmatt⁩ of excellent left wing academic research. I wonder if the massive quantity of high quality research means that none of it gets absorbed by the general public. https://t.co/mRNQ4S49Ds

— David Brooks (@nytdavidbrooks) December 26, 2019

The original Vox article is The 2010s featured a lot of great social science. Here are my 12 favorite studies. What economists, political scientists, sociologists, and philosophers taught me about the world in the 2010s. by Dylan Matthews.

In the first place, sociology and psychology are fields riddled with studies which fail to replicate. Not just occasionally, but most of the time. If it is an exciting and significant discovery being reported, you can bet that it is a small sample size, small effect size, non-random population, and will fail to replicate.

Now exciting sociology and psychology are the sandpits Brooks dabbles in and are the source of most his ponderous musings and books. The fact that he is writing in an ephemeral fashion on ephemeral findings is sort of ironic but not an uncommon irony.

The striking thing about his tweet is its heavy-handedness, almost smacking of desperation. "Astoundingly good", "excellent left wing research", "massive quantity", "high quality research" - Put your thumb a bit heavier on the scale David. Brooks defies Betteridge's Law. Brooks wants you to believe the answer to his question is yes. Brooks is so deep into the left worldview that he wants to argue that the more something is true, the less likely people are to believe it.

But Betteridge has the last laugh. The answer is no. The public does not absorb the left wing research because it is weak and faulty research. Not because it is so awesomely, massively, compellingly true. It isn't.

Is this research all that good given what we have seen in terms of the replication crisis? A crisis which became far better quantified in this past decade? I don't usually read the young adult news site Vox owing to its own high error rates of reporting.

But if David Brooks thinks it so massive, astoundingly, excellent, high quality reporting, perhaps it is worth checking.

Saved you a click. No.

I am familiar with much of the research and most of it suffers from the same common issues. Not so much in these instances of small and non-random samples (though that remains true for some of them.) Rather, the weakness is that the methods are not rigorous or robust, the research effect sizes are small, the findings are contested by other, more robust research.

They may be the best of the left wing lot but that doesn't make them good. No matter how much they are puffed up by name media celebrities. Oddly, Matthews has the same tick as Brooks, compensating for small effect sizes by large claims. An example (emphasis added):

So Meager uses techniques from Bayesian statistics to measure how much the results of a specific intervention — microcredit or microfinance programs for the global poor, of the kind offered by groups like Grameen or Kiva — vary from study to study. She doesn’t have a huge number of studies to go on (only seven) but she’s able to use this method to find that the effectiveness of microcredit varies a bit, but not a huge amount, from place to place. That suggests our evidence on microcredit is reasonably externally valid: The results in a new location are likely to resemble the results in past locations pretty closely, if hardly perfectly.

Overall, this is a hugely promising new way to synthesize evidence in emerging social science literature. Meager’s research along with the work of David Roodman synthesizing evidence on issues like incarceration and immigration, gives me hope that we’re getting better at blending knowledge across studies to come to a more complete understanding of the world.

So a review of a tiny number of studies (hopefully not cherry-picked) reveals there is variance in outcomes (unspecified), such that the forecasts are hardly perfect. And this is considered HUGELY PROMISING.

Well, to quote Betteridge, No.

If this one-sided, cherry-picked smattering of a dozen left wing studies across the decade is the best the left has, then they don't have much fuel for their arguments. Not only are the studies cherry-picked, some of them are already broadly refuted. That would have been useful knowledge to add.

Instead, as is often the case with Vox, they are much better at telling you what they wish you would believe to be true but not especially good, despite Brooks's admiration, at telling you what is true. Certainly not this dozen of anemic studies.