Proving yet again what we wish to believe.

Proliferating instances of confirmation bias. I mentioned a while ago (Does the experiment actually show what it is being stated to show? ) a study that purported to document racial bias in infants but which through flaws in the study design, only showed that infants were utilitarian rationalists.

Now we have Written in Black & White from Nextions purporting to show that law firm partners are more critical of memos written by black associates than the same memo when written by white associates. But as with the infant study, it appears that the results are due to the predicate assumptions of the study rather than an indication of the real world. In other words, they set out to prove what they thought to be true and ended up doing so.

The set up is straightforward.

Nextions, along with the assistance of 5 partners from 5 different law firms, drafted a research memo from a hypothetical third year litigation associate that focused on the issue of trade secrets in internet start-ups. We followed a simple Question Presented, Brief Answer, Facts, Discussion and Conclusion format for the memo, and we deliberately inserted 22 different errors, 7 of which were minor spelling/grammar errors, 6 of which were substantive technical writing errors, 5 of which were errors in fact, and 4 of which were errors in the analysis of the facts in the Discussion and Conclusion sections.

This memo was then distributed to 60 different partners (who had previously agreed to participate in a “writing analysis study” from 22 different law firms of whom 23 were women, 37 were men, 21 were racial/ethnic minorities, and
39 were Caucasian. While all of the partners received the same memo, half the partners received a memo that stated the associate was African American while the other half received a memo that stated the associate was Caucasian.

The 60 partners in the study received the memo electronically (an attached pdf) along with the research materials used in the preparation of the memo. The cover email thanked each of them for participating in a study on “writing competencies of young attorneys,” and asked them to edit the memo for all factual, technical and substantive errors. The partners were also asked to rate the overall quality of the memo from a 1 to 5, with “1” indicating the memo was extremely poorly written and “5” extremely well written.

The partners were originally given 4 weeks to complete the editing and rating, but we had to extend deadline to 7 weeks in order to obtain more responses. 53 partners completed the editing and rating of the memo. Of the 53 completed responses, 24 had received the memo by the “African American” Thomas Meyer, and 29 had received the memo by the “Caucasian” Thomas.

The results are also reasonably clear (though unelaborated).

The exact same memo, averaged a 3.2/5.0 rating under our hypothetical “African American” Thomas Meyer and a 4.1/5.0 rating under hypothetical “Caucasian” Thomas Meyer. The qualitative comments on memos, consistently, were also more positive for the “Caucasian” Thomas Meyer than our “African American” Thomas Meyer.

In regards to the specific errors in the memo:

 An average of 2.9/7.0 spelling grammar errors were found in “Caucasian” Thomas Meyer’s memo in comparison to 5.8/7.0 spelling/grammar errors found in “African American” Thomas Meyer’s memo.

 An average of 4.1/6.0 technical writing errors were found in “Caucasian” Thomas Meyer’s memo in comparison to 4.9/6.0 technical writing errors found in “African American” Thomas Meyer’s memo.

 An average of 3.2/5.0 errors in facts were found in “Caucasian” Thomas Meyer’s memo in comparison to 3.9/5.0 errors in facts were found in “African American” Thomas Meyer’s memo.

I look at this first from a quality perspective and focus on the quality control process. In an ideal system, 100% of errors are caught and corrected. How did the partners do?

Grammar Errors - Only 41% of Caucasian errors were caught (versus 83% for African-Americans)
Technical Errors - Only 68% of Caucasian errors were caught (versus 82% for African-Americans)
Factual Errors - Only 64% of Caucasian errors were caught (versus 78% for African-Americans)

Yikes! Overall, even when looking for errors, partners only caught problems ~70% of the time. In most processes, this would be a significant problem.

So there is a quality control problem. Is there a race problem? We don't know owing to the design of the study. What we do know is that

There was no significant correlation between a partner’s race/ethnicity and the differentiated patterns of errors found between the two memos. There was also no significant correlation between a partner’s gender and the differentiated patterns of errors found between the two memos. We did find that female partners generally found more errors and wrote longer narratives than the male partners.

So African-American partners and Caucasian partners are both exhibiting the same pattern of finding more errors in African-American associates than in Caucasians.

What this demonstrates is that there is a preconception on the part of both African-American and Caucasian partners that there will be a higher rate of errors among African-American associates than among Caucasians and therefore warranting, whether consciously or unconsciously, extra scrutiny. Now that is a real problem if, IF, there is no basis for that belief.

We need to know whether African-American associates generate more errors than Caucasian associates. This is one of the tragic and unintended consequences of affirmative action. We already know that African-Americans admitted under preferences, end up with worse grades, are more likely to not complete their course of study, and, in the case of law, have higher bar fail rates and take the bar more times before passing than Caucasians. What started out as a well intended strategy to right an historical wrong has ended up wreaking havoc on individual African Americans. It doesn't have to be this way. See California experience for an example where abolishment of affirmative action has led to higher enrollment, higher grades, higher completion rates and higher bar pass rates.

Based on those documented experiences in higher education and law, is it reasonable to expect that African-American associates might have a higher than average error rate? Sure. Would have to be documented but there is a reasonable probability that that is the case. And if it is the case, then the inclination of partners to administer extra scrutiny begins to make sense.

Nextions, if they had wanted to produce useful information rather than try and prove a point they already believed in, would have to make at least three adjustments to their study. First, they would need to document the base line scrutiny. How many errors do partners discover when unprompted by race? When they cannot call upon prior assumptions about anticipated quality, they might actually exert an even higher degree or review.

Second, they would need to document whether African-American associates do in real life have a higher or lower error rate than Caucasians. If higher, then their extra scrutiny is warranted. If lower, then it is not.

Third, it would be interesting to know how scoring is correlated with partners' prior experience with African-American associate error rates. If partners come from firms that recruit solely on a color blind basis, one would anticipate that Caucasians and African-American associates in that environment might have about the same error rate and correspondingly that partners exerted an equivalent level of review.

Nextions produced a study with an outcome that seems to be consistent with what they want to believe but is too poorly designed to actually support their prior belief. More cognitive pollution.