Empirical measurement trumps gut instinct in hiring.

Well, Ouch!

From Mechanical Versus Clinical Data Combination in Selection and Admissions Decisions: A Meta-Analysis by Nathan R. Kuncel, David M. Klieger, Brian S. Connelly, and Deniz S. Ones.

In employee selection and academic admission decisions, holistic (clinical) data combination methods continue to be relied upon and preferred by practitioners in our field. This meta-analysis examined and compared the relative predictive power of mechanical methods versus holistic methods in predicting multiple work (advancement, supervisory ratings of performance, and training performance) and academic (grade point average) criteria. There was consistent and substantial loss of validity when data were combined holistically— even by experts who are knowledgeable about the jobs and organizations in question—across multiple criteria in work and academic settings. In predicting job performance, the difference between the validity of mechanical and holistic data combination methods translated into an improvement in prediction of more than 50%. Implications for evidence-based practice are discussed.

I need to read this later when I am less tired. My initial read is that IQ tests and similar mechanistic means are substantially better at forecasting future performance than more holistic approaches that seek to incorporate human interpretation.

An elaboration on the abstract:

The results of this meta-analysis demonstrate a sizable predictive validity difference between mechanical and clinical data combination methods in employee selection and admission decision making. For predicting job performance, mechanical approaches substantially outperform clinical combination methods. In Lens Model language, the Achievement Index (clinical validity) is substantially lower than the Ecological Validity.

This finding is particularly striking because in the studies included, experts were familiar with the job and organizations in question and had access to extensive information about applicants. Further, in many cases, the expert had access to more information about the applicant than was included in the mechanical combination. Yet, the lower predictive validity of clinical combination can result in a 25% reduction of correct hiring decisions across base rates for a moderately selective hiring scenario (SR .30; Taylor & Russell, 1939).

A 25% reduction in forecasting accuracy is very substantial. In high turnover industries this could be the difference between a profitable year and a loss.

This goes against much of the received wisdom of the academy. Received wisdom is often wrong. Worth waiting and seeing but this is an interesting finding.