The Achile's heel of psychology and sociology

This is a very revealing piece of research and what it reveals is not entirely reassuring. From Are the Rich Really Less Generous Than the Poor? by Stephen J. Dubner of Freakonomics.

There has been a longstanding assumption that the answer is that yes, the rich are less generous.

But here’s something important to keep in mind. A lot of the scientific evidence for the rich being selfish came from lab experiments.

Nikos NIKIFORAKIS: But there are differences between our lab setting, and the field.

That’s Nikos Nikiforakis, another economist.

NIKIFORAKIS: It’s important to go and check your intuition in the field.

And that’s exactly what Nikiforakis, Stoop and Andreoni did — they ran a field experiment.

Good for the researchers. The evidence to support the hypothesis is based on abstract, theoretical experiments run in the lab. There is no evidence from the real world beyond the lab and that is what they are seeking. For a long time it has been known that lab conditions, especially in economics, psychology and sociology do not replicate in the wild of real life. The most recent spectacular flop has been the widely touted Implicit Attitude Test (AKA Implicit Association test), administered in the lab and assumed to reveal inherent biases against other groups. After the first few years it was discovered that the results of the test were not reliable over time and that results were uncorrelated with any real world outcomes. In other words, the test designed to unveil hidden biases failed to match real world behaviors.

Nikiforakis, Stoop and Andreoni were justified to be suspicious about lab results without field results. They put together a reasonably well structured field test which is discussed in detail in the article. Basically, they distributed letters with money to selected households and then measured whether and how quickly recipients (half the households being wealthy and half poor) forwarded or kept the letters (the letters had a proper address, just not the one to which it was initially delivered; all the recipient had to do was put the letter back in the mail.)

It is a well structured experiment to test the assumption that the wealthy are less generous than the poor. What were the results?

DUBNER: The big question, then, is what did you find? What did you learn about rich people versus poor people in this creative, interesting, albeit very unusual form of what we might call altruism?

ANDREONI: Given the research that we had been reading from Paul Piff and others like that, we expected the rich people to be less likely to return these envelopes. What we found was in fact the opposite.

STOOP: For us, the results were quite shocking. The rich returned way more than the poor — in fact, they returned twice as much. Return rates of the rich were roughly 80 percent and return rates of the poor were roughly 40 percent.

ANDREONI: The rich didn’t care whether there was money in the envelope or not, cash versus the check, they returned them at about the same rate.

STOOP: We find that roughly 25 percent of the cash came back from poor families, whereas roughly 75 percent of cash came back from the rich families. But, for us, the biggest shock was in observing that the non-cash envelopes were also not returned by the poor families.

ANDREONI: It’s looking to us, when we first get the results, that the rich people are actually much more altruistic than the poor.

Fair enough. A good trial and an objective result.

Here is where things begin to go off the rails and it is a feature of much research in economics, sociology, and psychology. The results are the results. From a scientific perspective, mission accomplished. But that is not how the researchers reacted.

NIKIFORAKIS: I thought, ”We failed. We found the wrong result.” In a sense, I did expect that the poor would greatly outperform — in terms of kindness — the rich.

Everyone gets unexpected results on occasion. You replicate the experiment to see if it was a fluke. If confirmed, you then redesign the experiment to further test assumptions.

That's not what they do here. They go fishing to conjure hypotheses which would explain away there result.

DUBNER: Since rich people and poor people may differ on dimensions other than income, what were some of the potential confounding factors that might influence or pollute your findings?

STOOP: When the experiment was over, we got data from statistics Netherlands that provided us with details on all our subjects houses: their education level, their age, that’s for all people in each household. We include all of these in this regression analysis. Basically, we can see, statistically, if all these factors matter. Quite to our surprise, none of them seem to have an effect.

ANDREONI: But then we noticed a couple other things: first of all, the envelopes that the rich got were being returned much faster than the envelopes of the poor folks.

The commenters are all over this departure from the scientific method. A couple of comments:

Daniel

I feel like if they found the result that they wanted (the rich being more greedy) they wouldn't have given it a second thought. I feel like they really jumped through a lot of hoops and then topped that with a bunch of unproven assumptions to FINALLY come to a conclusion that the rich and poor, once their assumptions were factored in, were equally generous. My only thoughts about this study is that the conclusion is anything but scientific.

Brian K
The study featured in this episode is a great example of both good science and non-science.

Good science: The researchers set up a clever experiment, propose a hypothesis, and collect results. Analysis of the data ends up disproving their hypothesis. That's OK, it's science---hypotheses are disproven all the time.

Non-science: The researchers engage in post-hoc data mining until they arrived at a more satisfying explanation of the data. In fact, during the episode they openly remark that they "found the wrong result". While intriguing, the explanation they arrive at during their post-hoc data mining is *not* scientific. They need to validate their new hypothesis by running a follow-up experiment with fresh data.

There are numerous other comments pointing out additional flaws and issues.

Letting your priors determine how you deal with empirical data is a critical issue and I think it is one of the most pervasive issues that stands in the way of psychology, sociology, and to some lesser extent, economics from being treated as seriously as I think they could be. If your assumptions determine your interpretation of the data, then everyone has equal standing. We can all make assumptions. It is only those who test those assumptions a priori and in a transparent fashion who actually discover real things that could be valuable.