Saved by low expectations

From Is psychology going to Cincinnati? OR: The mystery of the televised salad by Adam Mastroianni.  He is focusing on psychology but his questioning is pertinent to all epistemics.  How do we know what we know?

Here’s one way to measure our progress: take some people who are armed with the psychological literature and pit them against other people who are armed only with their own intuitions. The more we learn, the more this should be like a fight between gun-toters on one side and knife-wielders on the other. For instance, a bridge built using actual physics should hold up better than a bridge built using folk physics.

We don’t run a lot of these John Henry-style showdowns, but when we do, psychology does not win a resounding victory. Here are three examples.

1. Our anti-hate interventions are about as good as a Heineken commercial

The Strengthening Democracy Challenge tested all sorts of ideas for reducing animosity between Democrats and Republicans. Many of them didn’t work. One of the top performers, though, was this Heineken ad from 2017, which was made not by psychologists for the purposes of helping people get along, but by marketers for the purposes of selling beer.

2. Licensed therapists aren’t obviously better than a random nice person

In clinical and counseling psychology, there’s an ongoing debate about whether their training actually does anything. One uncomfortable finding: trainees can do just as well as fully licensed therapists. 

In fact, in the late 1970s, two researchers assigned2 a small group of college men to receive treatment either from trained therapists or from professors in a variety of disciplines who were selected based on their “untutored ability to form understanding, warm, and empathetic relationships.” At the end of the study, it looked like the professional therapists and the affable professors were equally effective. This study is too small and low-quality to count for much, but it’s concerning that it wasn’t a slam-dunk in favor of the pros.

3. Personality psychologists perform about as well as shamans

The Big Five theory claims that all aspects of human personality boil down to five factors: openness to experience, conscientiousness, extraversion, agreeableness, and neuroticism. There’s some disagreement about whether it’s better to describe personality using six or two factors or whatever, but most people will tell you that the Big Five is solid, a great achievement, a good theory backed up by decades of empirical studies.

He asks

So how well does the Big Five perform against, say, some personality tests that people pulled out of their asses?

And the answer is:

If you use the continuous Meyers-Briggs and Enneagram scores, the Big Five’s lead narrows or disappears altogether (higher numbers = better prediction):

Source: ClearerThinking

When I saw this result, I broke out in a sweat. Whatever it is we psychologists do, we’ve done a lot of it to the Big Five. After millions of dollars and thousands of studies, we are not obviously better at predicting life outcomes than people who… didn’t do any of that. 

Over the years I have been critical of Meyers-Briggs as anything but a catalyst for self-reflection and had always assumed that the Big Five model had some empirical grounding which included predictive utility.  Only my deep skepticism (despite my interest) in the whole psychological field saves me from being horrified that the Big Five model is no more predictive than Meyers-Briggs.