Relative frequency of rational arguments and empirical arguments

From Debunked and Well-Refuted by Scott Alexander. As to be expected, a thoughtful essay on the challenges of defining accurate data. Well, to be precise, he is actually focusing on what constitutes being debunked and what constitutes being well-refuted but in his discursive style, he covers a lot of territory.

Alexander is addressing, from a different perspective, an issue which I have sought to clarify in decision-making. Prior posts include How to assess a piece of writing, especially outside one's expertise, Identifying Cognitive Pollution, and The intensity of the conviction that a hypothesis is true has no bearing on whether it is true or not.

Alexander has the following passage.

A naive empiricist who swears off critical thinking because they can just “follow the evidence” has no contingency plan for when the evidence gets confusing. Their only recourse is to deny that the evidence is confusing, to assert that one side or the other has been “debunked”. Since they’ve already made a principled decision not to study confirmation bias, chances are it’s going to be whichever side they don’t like that’s “already been debunked”. And by “debunked” they mean “a scientist on my side said it was wrong, so now I am relieved from the burden of thinking about it.”

This sparked a line of thought. This is part of the long running dialectic between the stereotypical French concept of Descartian Rationalism, thinking through things at a conceptual level based solely on reason and the British empirical tradition of muddling through and letting theory follow from practice.

Can we set up a chart which maps these two traditions? With Powerpoint, of course we can.

On the left, you have the vertical axis with some measure of the volume and consistency of the rational argument behind some topic of interest. A 1 would indicate that the principals and causes are poorly understood or defined and that therefore the rational argument is poorly based and speculative. A 10 indicates that there is precision in definition of terms, principals and the flow of causation is well understood. A 10 indicates that the conceptual understanding is very robust and widely shared and that from that theory there ought to be concrete, measurable predictions that can then be verified.

On the horizontal scale is Empirical Evidence. A 1 indicates that there is little empirical evidence at all, or that the empirical evidence, regardless of volume is inconsistent. A 10 indicates that there is a voluminous population of robustly gathered evidence with no inconsistencies and with plenty of replication from multiple sources.

With those definitions, let’s then look at the quadrants. I have populated the chart with dots reflecting types of conversations with what I suspect are the relative frequency with which they happen.

D represents propositions or arguments or phenomenon which have both voluminous, robust, consistent, replicated empirical evidence AS WELL AS logical rational bases for explaining the cause of the proposition. These are, in the scheme of things, relatively scarce. Evolution, laws of physics, things like that. They represent the pinnacle of cognitive civilization. It is important to acknowledge that such instances, robust as they appear, like all knowledge, remain contingent. What we know is always subject to revision. It just happens that these items are least likely to be revised.

C is the quadrant where the empirical data is robust but we don’t have a theoretical or rational basis to explain the data. The flight of the bumblebee and eyes in animals are historical examples where there was robust empirical evidence but the logical/rational explanation was weak.

B is the quadrant where there are elegant, logical/rational explanations for an argument or proposal or phenomenon but for which there is little robust empirical data. String theory might be an example here. It fits the data but makes no testable predictions that can be tested.

A is the quadrant where there is little consistent robust empirical evidence and a multiplicity of possible but contradictory logical explanations. Who will make a great teacher, which new book/movie will do well, which athlete will do well on the team, which business will prosper are all issues for which there are many rational arguments and for which the data is either missing or conflicting.

A couple of observations.

I have populated the above the line space with many more dots than below the line. This follows from economic principles. There is, for most arguments, little cost to generating theoretical explanations. Being cheap, there likely is more. Below the dividing line, there are fewer dots because empirical evidence is time consuming and expensive to generate and replicate. Being expensive, there will be less of it.

I have heavily populated the A quadrant because I think that the bulk of conversations/arguments are characterized by low logical/rational coherence and low empirical data. Or at least that has been my experience.

There is a missing dimension from this graph and that is the third dimension of Intensity of Belief. This would also cluster down in quadrant A. There are many things which many people believe passionately but for which they are unable to muster a rational argument or coherent empirical evidence.