Consequences of small mean differences and small standard deviations

Professor Campbell's comment sparks a thought.

A problem with the gender similarities hypothesis. Small mean level differences in neuroticism result in very large gender diffs at tails. https://t.co/dg35QHPqn2

— W. Keith Campbell (@wkeithcampbell) August 26, 2017

Gender just happens to be the nexus for his comment, but the underlying statistical issue applies to any normal distribution where there might be small differences in either the mean or the standard deviation (SD). The consequence of small mean differences and small standard deviations are big differences in the tails of the distribution.

The thought is - imbalances in the tails of statistical distribution are where stereotypes are generated and the tails are extremely sensitive to small differences in either mean or standard deviation.

Professor Campbell's comment "Small mean level differences in neuroticism result in very large gender diffs at tails" addresses differences in the mean but the same effect occurs when there are small differences in standard deviation as well.

For example, men and women have the same mean IQ, 100, but the standard deviation is somewhat greater for men than women. In effect, most women are pretty close to the mean of 100 whereas there is a greater spread around the mean for men. Of course that spread is on both side of the mean, more low IQ men than women as well as more high IQ men than women.

Very roughly, as you get out on the far side of the distribution, the differential impact becomes much more noticeable. At IQs above 140, there are very roughly 2 men for every woman with that score and the further out you go, the greater the imbalance.

Why might we have evolved to be highly attuned towards small differences in means and standard deviation? So sensitive that we create material stereotype differences that are only true for the exceptions in the population and not for the norms?

I am guessing that it has to do with risk and opportunity. Take, as an extreme example, psychopathy. It is rare in both male and female populations but among those with an extreme manifestation of psychopathic traits, men outnumber women by as much as 20:1, presumably through some combination of both mean and standard deviation (I would suspect the latter more than the former).

In a whole lifetime, the odds of you meeting a person with strong psychopathic traits is small. And among the flow of people with whom you interact, you likely won't be able to detect any significant difference in the mean. But from an existential and evolutionary perspective, over long time frames, the risk, while small is going to be concentrated in the tail of the distribution.

There are positive aspects of this tail-determined stereotyping as well. While the causative details are contested, there is general agreement that women have higher mean scores on empathy than do men (I don't know about the SD but I suspect there are differences there as well.) So again, while at the mid-point, men and women are substantially the same, at the extreme of the distribution tail, you are going to see many more highly empathetic women than men.

In most circumstances, the means and SD differences are so small that it doesn't make any difference in lived lives. But in extreme circumstances, either from risk or need, those small differences in mean and SD represent a material difference in existential risk. If you are marooned on an island, you don't want it to be with that rare exception who is an extreme psychopath. Likewise, if you are running a sales force in a commodity industry, you don't want a lot of team members with extreme empathy (they will give away all your margin.) On the other hand, if you are in extreme need, hunger, injury, poverty, you really want to meet someone with extreme empathy.

So over a lifetime of experiences and over generations of genetic selection, I suspect that we have become very good at recognizing small differences in mean and SD because it has large implications in the tail of the distribution.

Following this train of thought, stereotypes then become a non-biological means of transmitting useful information in an environment of uncertainty. When you meet a stranger, the odds are very high that they are pretty much like anyone else but knowing the tail-risks is important for survival.

Many of our social problems arise from four different factors.

We conflate the actual mean with the possible extreme - The average man is only as violent as the average woman but among those who are extremely violent, men outnumber women by many factors. When we meet a stranger, male or female, we should treat them the same but our risk assessment says that one represents a greater risk, no matter how remote, than the other. Given our inclination towards risk aversion, conflating the actual with the possible leads us to treating unknown strangers quite differently depending on whether they are male or female.

We are terrible at rationally assessing low probability/high consequence scenarios - This is the life-work of Nassim Nicholas Taleb (Fooled by Randomness, The Black Swan, Antifragile). We have a strong inclination to make suboptimal decisions because we overestimate the probability of low probability/high consequence events. That inclination makes us much more attuned to tail effects than mean effects.

We fail to update our stereotypes with real information - Our stereotypes, particularly when they entail negative risk, tend to be sticky. We meet a stranger and get to talking. As we learn more about them, and then as we begin to interact with them, then collaborate, we generally are acquiring more and more data that allows us to discount the tail-risk. But because all new information has its own risk factor (you don't accept everything as true just because it seems to be true), we are slower to update and move from the stereotype to the actual. In extreme circumstances, some people will never update their stereotype. For example, someone, who as a child had a traumatic experience swimming in the ocean, may never update their personal stereotype that the ocean is dangerous.

Stereotype accuracy - Lee Jussim and others have studied the phenomenon of stereotype accuracy: are stereotypes accurate? To what degree? Under what circumstances? Are there differences in stereotype accuracy between type of stereotype? Their findings are that stereotypes, as a class but with a few exceptions, provide useful information in the absence of actual knowledge. This finding is in contrast with the older view in the psychology field that broadly stereotypes were malicious in origin and lacking in correlation with empirical data. When something, such as stereotypes, prove in experience to be usefully true, they tend to persist.

My supposition from this trail of thought is that stereotypes originate from small differences in mean and small differences in standard deviation manifesting themselves strongly at the extreme tails of normal distributions. While existentially useful, those stereotypes pose a challenge to ourselves as rational beings because we tend to conflate the actual mean with the possible extreme, we are really bad at treating low probability/high consequence events, we are slow to update stereotypes and stereotypes end up, none-the-less, being usefully true (not all stereotypes but enough).

Confusion about how small mean differences and small standard deviations can lead to large differences in the distribution tail are, I suspect, behind the uproar surrounding the firing of Google scientist James Damore for publishing a reasonably accurate summary of current scientific knowledge about male/female differences. For background see The Google Memo: What Does the Research Say About Gender Differences? by Sean Stevens and Jonathan Haidt and The Most Authoritative Review Paper on Gender Differences by Sean Stevens and Jonathan Haidt. Damore's document was substantially consistent with mainstream scientific consensus. So why the uproar? I suspect it is that the media and ideologues are not attuned to the statistical implications of small mean differences and small standard deviations.

UPDATE: Not directly relevant to my point about far tail disparities driving stereotypes but a good technical explanation of how small differences in either or both mean and variation can lead to dramatic dissimilar outcomes. A politically incorrect guide to affirmative action by Philippe Lemoine.