Running the numbers without taking into account statistics

From The surprising thing Google learned about its employees — and what it means for today’s students by Valerie Strauss.

I have mentioned some number of times in the past, my speculation that at least some portion of cognitive pollution entering the public discussion is perhaps not due to ideological bias but due to innumeracy on the part or journalists and editors. It is, of course not either/or - bias and innumeracy might be operating in conjunction.

In the Strauss article, drawing on a recent book by Cathy N. Davidson, the proposition is advanced that:

All across America, students are anxiously finishing their “What I Want To Be …” college application essays, advised to focus on STEM (Science, Technology, Engineering, and Mathematics) by pundits and parents who insist that’s the only way to become workforce ready. But two recent studies of workplace success contradict the conventional wisdom about “hard skills.” Surprisingly, this research comes from the company most identified with the STEM-only approach: Google.

Sergey Brin and Larry Page, both brilliant computer scientists, founded their company on the conviction that only technologists can understand technology. Google originally set its hiring algorithms to sort for computer science students with top grades from elite science universities.

In 2013, Google decided to test its hiring hypothesis by crunching every bit and byte of hiring, firing, and promotion data accumulated since the company’s incorporation in 1998. Project Oxygen shocked everyone by concluding that, among the eight most important qualities of Google’s top employees, STEM expertise comes in dead last. The seven top characteristics of success at Google are all soft skills: being a good coach; communicating and listening well; possessing insights into others (including others different values and points of view); having empathy toward and being supportive of one’s colleagues; being a good critical thinker and problem solver; and being able to make connections across complex ideas.

Those traits sound more like what one gains as an English or theater major than as a programmer. Could it be that top Google employees were succeeding despite their technical training, not because of it? After bringing in anthropologists and ethnographers to dive even deeper into the data, the company enlarged its previous hiring practices to include humanities majors, artists, and even the MBAs that, initially, Brin and Page viewed with disdain.

There is no doubt that under the right circumstances, diversity of knowledge domains within a team of equally accomplished individuals can add much insight, innovation, and cross-fertilization between domains.

But Davidson appears intent on buttressing an otherwise anodyne assertion by twisting the actual implications of the data. The conclusion she wants to arrive at is that the humanities are important (I concur) and they are equally valuable (which is not what the data indicates.)

STEM skills are vital to the world we live in today, but technology alone, as Steve Jobs famously insisted, is not enough. We desperately need the expertise of those who are educated to the human, cultural, and social as well as the computational.

No student should be prevented from majoring in an area they love based on a false idea of what they need to succeed. Broad learning skills are the key to long-term, satisfying, productive careers. What helps you thrive in a changing world isn’t rocket science. It may just well be social science, and, yes, even the humanities and the arts that contribute to making you not just workforce ready but world ready.

Without reading the book, there may be much nuanced insight that does not come across in this summary article.

What comes across, though, is a set of fairly fundamental cognitive errors. It centers on this finding:

Project Oxygen shocked everyone by concluding that, among the eight most important qualities of Google’s top employees, STEM expertise comes in dead last. The seven top characteristics of success at Google are all soft skills: being a good coach; communicating and listening well; possessing insights into others (including others different values and points of view); having empathy toward and being supportive of one’s colleagues; being a good critical thinker and problem solver; and being able to make connections across complex ideas.

I suspect there are actually four errors going on simultaneously: Category Error, Range Restriction error, Survivorship Bias error, and the Simpson's Paradox.

It seems that the Washington Post commenters are alert to the errors. It makes you wonder, stuffed to the gills with STEM talent as Google is, how they allowed such a set of mistakes to color their own internal research.

Category Error - The categories of "coders/technologists" is not the same as the category of "people with STEM degrees." This isn't the biggest error, but it is an important one because it almost de facto reduces the validity of the conclusions. In our vernacular, we are playing fast and loose with our terminology. There are plenty of coders/technologists who entered the field via a route different than a STEM degree (or even a degree at all) and there are plenty of STEM degree people who redirect into non-STEM fields of endeavor. Using a degree as an indication of interest and accomplishment in technology is only a very loose proxy.

Range Restriction - Freddie deBoer has a good description of range restriction, here. When you select on one attribute among many which might be contributive to outcome, the other attributes become more explanatory of outcomes than does the range restricted attribute selected. As someone on twitter put it, "Conditional on having top-1% technical skill, technical skill is not very predictive of job success."

Survivorship Bias - The conditionality referenced above is a form of survivorship bias. "The logical error of concentrating on the people or things that made it past some selection process and overlooking those that did not, typically because of their lack of visibility. This can lead to false conclusions in several different ways. It is a form of selection bias."

Simpson's Paradox - "A phenomenon in probability and statistics, in which a trend appears in several different groups of data but disappears or reverses when these groups are combined." The patterns and correlations you might see in a sub-population (STEM degrees from prestigious schools population) can vary or be the opposite of those that you see in an amalgamated population (all degrees from prestigious schools.)

Some of the Washington Post comments which tackle these issues

If the study did not include control groups that weren't subject to the initial hiring criteria, it is likely the results are strongly skewed by cofactors related to the study population. Among teams that all start out with very strong STEM skills (based on Google hiring practices) OF COURSE the teams that excel are going to be the ones that differentiate themselves through some other skill set(s).

The difference in STEM abilities among a population that was selected specifically for their STEM abilities is going to be much smaller than the variation among the other skills they weren't selected for. It's a bit like saying professional football players who did well in their college classes tend to stand out. It's because every professional football player has tremendous physical skills, but only some of them also did very well in school.

[next]

"being a good coach; communicating and listening well; possessing insights into others (including others different values and points of view); having empathy toward and being supportive of one’s colleagues; being a good critical thinker and problem solver; and being able to make connections across complex ideas.

Those traits sound more like what one gains as an English or theater major than as a programmer."

as anyone who has spent (serious) time with either english and/or theater majors would tell you, this inferential leap is complete, total and utter nonsense...there is little-to-nothing about the way english and/or theater has been or is being taught that relates to 'good coaching' or 'empathy' in the slightest....i am the first person to argue that each and every one of these so-called 'soft skills' are desirable and important....but the assertion/argument that you are more likely to acquire/learn these skills in a humanities curriculum vs a stem curriculum is nonsense on stilts....let's be clear: any curriculum that explicitly cultivates critical thinking skills, collaborative projects and measurable/meaningful deliverables is good for developing these skills....but, as someone who both teaches and does interdisciplinary research in a world-class university, i beg parents and students alike to recognize these sorts of curricula are the exception, not the rule....ms davidson's 'analysis' is sadly and fatally flawed....her heart is surely in the right place; her facts are not

[next]

Graduate education in the humanities rarely teaches critical thinking skills beyond textual analysis. More than anything else, graduate education in the humanities teaches students to compete with each other, while navigating the politics of their chosen field. Nor do critical thinking, collaboration, empathy, etc., flourish in the highly competitive world of tenure-track Ph.D.'s. The idea that these same Ph.D.'s will somehow transmit soft skills to their students is, more often than not, fatuous -- how can they teach what they've never learned?

[next]

You start by saying they only hired people with good grades in stem from top schools, and then finished by saying that STEM was ranked last on their list of important traits. As a statistician, I can tell you what your problem is. STEM was just a common factor all hires had, so of course it is last on the list. Saying people had STEM and insight into people means that they have mastered everything basically. It doesn't mean that an English major can succeed at Google.

[next]

After working in the tech field for almost 50 years (starting as a programmer, then manager of programmers, managing system programmers running data centers, more software development and management experience, ending in my own consulting business) it was obvious to me that my success was because of my people skills and "soft" skills. BUT, I could never have done the job if I had not had the innate technical abilities and earned respect of my subordinates. And while some of the soft skills came from education (not much team oriented projects in the 60's), most of it came from experience. And innate personality. The important thing is to recognize the innate intellect and potential and nurture it thru education as well as on-the-job mentoring and training.

[next]

Indeed. If Google hired a bunch of people with non-STEM degrees, and then evaluated them, what qualities do you think will come out to be most important?

[next]

The key to this article is that having the computer science degree is what got people in the door, having softer skills made them a better employee.

Like it or not our economy rewards those with skills that are numbers, systems, analytical and scientifically focused. Softer skills are indeed important but provide depth to a technical skillset, not a substitute for one. I tell my college-bound kids that their primary degree should be technically oriented but blending this with a minor or coursework in design, history, philosophy or so forth will round them out and also open their mind to approaching problems and issues in new and unique ways. Some chafe at this notion but we need citizens with a moral voice too, something math and computer science won't provide on it's own. We need people who can discuss whether some new science or technology should be done, not just can it be done.

There are all sorts of interesting issues in here of causal origins of innovation, technical competence, team effectiveness, group productivity, etc.

They warrant investigation and research. Progress is not facilitated by erroneous interpretation (whether owing to innumeracy or ignorance or bias or ideology). In a STEM environment dominated by people with technical skills, it is unsurprising that those with corresponding skills in team management or communication might shine above and beyond their counterparts with just the equally superior technical skills.

But if, from that, you conclude that everyone can be equally successful in tech regardless of their degree and that is what you communicate to young adults, then you are misleading them.

Yes, there are complex issues which need high-achieving individuals with a broad array of capabilities to tackle them. But if you want to achieve tech industry levels of compensation, find a way to be motivated to not only pass, but excel in STEM fields. And anticipate that if you have a residual competency or avocation in people management or communication, that will bode well for your career success.

But don't tell kids who want to major in history that they can anticipate achieving tech industry levels of compensation with just a history degree. That is not where the probabilities lie.