In 2022? Really?

Via the TaxProf Blog.  From Examining the Bar Exam: An Empirical Analysis of Racial Bias in the Uniform Bar Examination by Scott Devito, Kelsey Hample, and Erin Lain.  From the Abstract.

The legal profession is among the least diverse in the United States. Given continuing issues of systemic racism, the central position that the justice system occupies in society, and the vital role that lawyers play in that system, it is incumbent upon legal professionals to identify and remedy the causes of this lack of diversity. This Article seeks to understand how the bar examination—the final hurdle to entering the profession—contributes to this dearth of diversity. Using publicly available data, we analyze whether the ethnic makeup of a law school’s entering class correlates to the school’s first time bar passage rates on the Uniform Bar Examination (UBE). We find that higher proportions of Black and Hispanic students in a law school’s entering class are associated with lower first-time bar passage rates for that school in its reported UBE jurisdictions three years later. This effect persists after controlling for other potentially causal factors like undergraduate grade-point average (UGPA), law school admission test (LSAT) score, geographic region, or law school tier. Moreover, the results are statistically robust at a p-value of 0.01 (indicating just a 1% chance that the results are due to random variation in the data). Because these are school-level results, they may not fully account for relevant factors identifiable only in student-level data. As a result, we argue that follow-up study using data relating to individual students is necessary to fully understand why the UBE produces racially and ethnically disparate results.

This research is rather remarkable at this late stage in the game.  It might be comprehendible were it written in 1972 or 1982 instead of 2022.  But we have fifty years of research refuting their basic assumptions and providing an answer to that which they regard as mysterious.  

Devito, et al start with the declared belief that 

This Article is written with two purposes. First, we seek to sound the alarm that the bar exam is racially and ethnically biased.  For many, such an alarm will seem absurd. They will argue that the bar examination is difficult, but it is also fair; passing is a function of ability, work ethic, writing skill, and knowledge of the law—not race or ethnicity. Unfortunately, such a belief is unsupported by the evidence.

Second, we write this Article as a call to action. Philosophers have, for millennia, warned us of the dangers of accepting beliefs without adequately testing them. Yet that is precisely what the legal community has done with regard to the “objectivity” of the bar examination. The profession simply proceeds as if race and ethnicity are irrelevant to the probability that a bar-taker will pass the bar examination, or whitewashes the evidence of disparate outcomes and deems those differences inconsequential.  Given empirical evidence to the contrary, we contend that the legal community must demand in-depth analysis of the bar examination’s questions, administration, and grading to determine why race and ethnicity appear to impact bar passage rates. 

The persistent social justice unwillingness to engage concretely with reality is on display, particularly the misuse of language to establish opening positions that are otherwise untenable.

"The bar exam is racially and ethnically biased" is obviously and inherently untrue.  The bar exam is an inanimate process and holds no emotions or biases.  What is true is that "The bar exam results are racially and ethnically disparate."  Different ethnic groups score differently on uniform tests.  This has been well established and consistently replicated for all of modern sociology and psychometrics.

Why the results differ is what is in dispute and that they might be because the questions on such exams are in some way deliberately skewed to achieve predetermined outcomes is at least a defensible hypothesis.  

Such skewed questioning, often unintentional, was not uncommon in the early days of psychometrics.  I have a vague recollection of reading a report decades ago on an admissions test to a prestigious high school or college,  I think in Britain from around the turn of the last century.  It was something along the lines of "Where is the soup spoon placed at a sitting?"  A question of little value in assessing cognitive ability depending as it does on crystalized knowledge but almost guaranteed to favor the upper class test takers over the middle and lower class candidates.  

When I was in high school in the late 1970s we were well on our way towards stripping out such obvious questions which were not predictive of capability but which could be used for class (or religious or race or ethnicity) selection.  I recall a long running court case in, I think, New York.  

The test was for some civil service job with the government, police or fire department I believe.  On such tests there is an obvious component given the physicality of the job.  How much can you lift, how durable are you, etc.  

But there was a part of the exam which effectively served as an IQ test and which was initially loaded with questions that were unrelated to cognitive ability and/or were crystalized knowledge questions which could be effective in screening against some ethnic group.  Which team were the Ice Hockey league champions in 1971 is a question probably more often correctly answered by white New Yorkers than black.  I am guessing.

The court required the department to revise the test so that it was more neutral.  They had already, over four or five years, revised the test multiple times in order to get rid of such questions.  The first couple of cycles of revision were reasonably straightforward.  By the time I read of the case, all the obvious issues had been resolved.  The test was demonstrably far more a straightforward cognitive assessment and no longer contaminated by crystalized knowledge questions which might be more likely to answered correctly by one group more than another.  

But the test results were still racially disparate.  The court was trying to find a path forwards.  They wanted tests that were clear of any design flaws and were predictive of desired outcomes.  They had achieved that.  And yet they still had disparate results.

We now know much more clearly than was demonstrable then that the disparate results were due to G or IQ.  Individuals from different races, classes, religions who had the same IQ, did equally well on the uniform exams but IQ was not identically distributed by class, race, religion, etc..

That unpleasant reality is still with us.  But contra Devito et al, we now know why the disparate results occur.  And it is not because the "exam is racially and ethnically biased."  

Devito, et al are, however, documenting the Achille's Heel of affirmative action.  Again a well documented and well-known issue.  

If you admit candidates to a program with different capabilities in order to achieve a predetermined demographic compositional outcome (by race, class, religion, etc.), then your rates of achievement will fall.  On the other hand, if you hold fast on the capabilities, then you will have equal results but different demographics.

Devito et al are pretending that this is not well known and among the most replicated outcomes in the psychometric field for the past sixty years.  Why they want to make this assertion is unknown but it is not due to familiarity with the known facts.

This Article engages in statistical analysis of first-time bar passage rates, at the school-jurisdiction level, for schools in Uniform Bar Examination (UBE) jurisdictions.  The analysis reveals a highly significant, negative correlation between a school’s proportion of Black or Hispanic students and the first-time pass rate for that school jurisdiction. In essence, as a school’s proportion of Black or Hispanic students increases, the school’s first-time bar passage rates decline three years later (when the enrollees are expected to graduate). This result is statistically significant to a p-value of 0.01. Such a result should greatly concern the legal profession as it provides clear evidence of disproportionate bar examination outcomes based on race and ethnicity and suggests that such disproportionality may result from the exam itself. It should also prompt the legal community to further study the bar exam, using student-specific data, to better understand why bar passage rates decline as a school’s proportion of Black or Hispanic students increases.

You admit more students with lesser capabilities and your first time bar passage rate will almost necessarily decline.  Certainly if you hold all other things equal.  Why Devito et al find this suspicious is a mystery.  

By their blind rejection of consistently replicated data, Devito et al are making a bad situation worse.  There are perfectly well qualified black and Hispanic students for any profession, just not in identical portions.

You can hold fast to standards which will result in everyone who qualifies, regardless of race, to extract equal respect.  

Or you can fudge the results so that the demographic profile imperative is achieved but the successful students no longer represent equal ability.  When this happens, you end up with reinforced negative stereotypes.

This is well known.  And still we have researchers in 2022 trying to insist that a well known reality is somehow a mystery.  Just because we don't like reality because it differs from our ideological expectations does not in anyway make reality less real.