Sunday, July 26, 2015

Better statistical design and controlling for confounding variables gives the lie to most postmodernism/critical theory assumptions

A wonderfully succinct explication of a couple of the confounding issues attached to research into discrimination, from How To Study Discrimination (or Anything) With Names; If You Must by Uri Simonsohn. So much of what I see performed and trumpeted as damning evidence has statistical flaws you can drive a truck through, starting with the fact that it is usually motivated research. The researchers are proclaiming the results that they were intending to find in the first place.

Many studies will do something along the lines of assigning names to a standard resume, randomizing either "typical" black names and "typical" white names or randomizing "typical" female names and "typical" males names. They then send these to an HR department and ask which resumes they would select for interviews, expecting to find that "typical" white names are preferred over black and "typical" male names are preferred over female.

But names, as Simonsohn points out, signal much more than race and gender. They signal class and age and appearance and religion and geographic region and parental socioeconomic status and all sorts of other things. In addition there are many behavioral attributes that, for whatever reason, are popularly attributed to particular names. This has nothing to do with any statistical correlation between the name and real attributes, just what exists in the popular mind. These are stereotypes, pure and simple.

For example, look at the dramatic perceived difference in attributes between Sarah and Sally, between Charles and Chuck. Both are "white" names, both are common, for each, one is an abbreviation of the other. So close, and yet so far apart. Compared to his sophisticated, strong, wholesome, mature, formal, manly, smart, and serious older brother Charles, Chuck is seen as common, not very smart, a bit youthful/immature, somewhat devious, and unsophisticated.

Sarah is much like her sibling Charles, seen as wholesome, refined, smart, serious, formal, and classic, whereas her cousin Sally is seen as country, informal, and young.

If you google images of Sarah, you end up with pictures of women twenty years younger than when you google images of Sally.

As Simonsohn notes about comparing two seemingly innocuous generic names, Jennifer and John, its not just gender that is being signalled:
Jennifer was the #1 baby girl name between 1970 & 1984, while John has been a top-30 boy name for the last 120 years. Comparing reactions to profiles with these names pits mental associations about women in their late 30s/early 40s with those of men of unclear age.
Likewise with racially identifiable names:
Distinctively black names (e.g., Jamal and Lakisha) signal low socioeconomic status while typical White names do not (QJE .pdf). Do people not want to hire Jamal because he is Black or because he is of low status?
So names signal a lot more than race. If you want to isolate discrimination based on race or gender, you have to control for all those other signals which becomes very difficult to do.

One of Simonsohn's solutions to this problem of confounding signals is "Stop Using Names." Anonymize the names (and pronouns) of the candidates so that the reviewer can only focus on the actual accomplishments of the candidates. When you do this though, sometimes the results are the exact opposite of the postmodernist/critical theorist assumption about a world that is patriarchal and racist. For example, National hiring experiments reveal 2:1 faculty preference for women on STEM tenure track by Wendy M. Williams and Stephen J. Ceci. As they comment,
Our experimental findings do not support omnipresent societal messages regarding the current inhospitability of the STEM
professoriate for women at the point of applying for assistant professorships (4–12, 26–29). Efforts to combat formerly widespread sexism in hiring appear to have succeeded. After decades of overt and covert discrimination against women in academic hiring, our results indicate a surprisingly welcoming atmosphere today for female job candidates in STEM disciplines, by faculty of both genders, across natural and social sciences in both math intensive and non–math-intensive fields, and across fields already well-represented by women (psychology, biology) and those still poorly represented (economics, engineering).
What about the racism presumed to be so pervasive. This study is from France (so not directly comparable to the US) but it has the benefit of using real outcomes, not simulated ones (i.e. actual employers making actual decisions, not just HR personnel making simulated decisions.) Unintended Effects of Anonymous Resumes by Luc Behaghel, Bruno Crépon, Thomas Le Barbanchon.
We evaluate an experimental program in which the French public employment service anonymized resumes for firms that were hiring. Firms were free to participate or not; participating firms were then randomly assigned to receive either anonymous resumes or name-bearing ones. We find that participating firms become less likely to interview and hire minority candidates when receiving anonymous resumes.
In other words, it appears that companies were previously exercising some form of corporate affirmative action by hiring minorities that they would not otherwise hire. When they lost the status signal of minority names and had to rely simply on the actual performance record, the recruitment of minorities fell by 10%.

All of this is to say three things - 1) social processes are incredibly complex, 2) sociologists tend to seek confirmation of their priors (patriarchy, racism, etc.) and 3) sociologists tend to be very poor at managing the statistical controls of their studies.

When good statistical controls are in place with proper protocols, most of the thin evidence for patriarchy and racism and the other hobgoblins of postmodernism/critical theory disappear.

No comments:

Post a Comment