The discipline of asking hard questions of hard problems

In dealing with complex systems, our capacity to establish a meaningfully robust cause-and-effect relationship between inputs and outputs becomes negligible. We might believe that voluminous habitual reading at an early age is a critical activity to establish the habit and discipline of reading and that the habit of reading is causative of good life outcomes (status, income, productivity, health, education attainment, etc.). That argument on its own is very difficult to establish because of the multitude of confounding variables (neighborhood, school, household income, parental education attainment, health circumstances, familial structure, sibling circumstances, etc.). After some fifty years, we have the rudiments of evidence to support that argument but it is a fragile foundation.

Now try and extend the argument to an equally sensible proposition: the types of books that you read will have a influence and possibly determinative effect on life outcomes. If you read books that reinforce the positive/productive bourgeois values, will you more likely demonstrate those values later in life to your benefit? If you read morally ambiguous books with themes of abuse and failure and calamity, will you be more predisposed to believing those to be inevitable and accede to life's reverses? There are even more barriers to establishing that linkage including sequence of reading, age of reading, reading capabilities, etc.

There is a term for the challenges in identifying the causal variables (and the size effects of those causes) - complex adaptive systems. In a complex adaptive system, not only are there numerous causative variables, but the system is constantly evolving in a fashion that both changes the population of causative variables as well as changing the size effect for each of those causal variables. For example, if you run a meticulously planned and executed longitudinal reading study starting with a large number of participants starting in 1950 and running for 65 years, and establish some material correlations or even causative relationships between early reading and life outcomes, can you confidently believe those lessons learned to be still valid in 2015 for new-borns today entering a world of vastly changed economic circumstances, familial structures, technological capacities, etc.? That's the problem with complex adaptive systems: after spending time and money, the closer you get to being confident that you understand a causal relationship, the more likely that the context has changed rendering the finding moot. It is a social sciences variant of the Heisenberg Uncertainty Principle, for major social issues: You can't act quickly and with confidence. You can act slowly and with confidence or quickly without confidence but not both simultaneously.

Megan McArdle touches on this uncertainty in Everything We Don't Know About Minimum-Wage Hikes. Talking about the quality of research regarding the impact of minimum wage hikes and referencing one of the most robust studies (which was not meaningfully robust).

Pretty neat, huh? Here's the problem: That study only covered what happened to 410 stores over a period of less than a year. I'm not quarreling with the design of their study, mind you; there are very good reasons to stick with a limited sample over a short period of time. Over longer time periods, more and more extraneous factors will start to swamp your results: changes in state labor law or tax policy, local recessions, a municipal ordinance banning cheap restaurants with lurid signs.

McArdle is pointing out a very real-life situation where our knowledge frontier does not support our making fact-based decisions. The economics of minimum wage are terrifically complex with an inordinate number of contextual and confounding variables. We do not know the full range of pertinent variables or how they interact and cannot reliably predict the outcome of a policy change in minimum wage. How will it affect productivity, employment, income, consumption, etc.? We simply do not know, and at this point after several decades of research are not likely to know in any reasonable timeframe.

What we are left with is a desire to make things better but without the knowledge to do so with any predictability. All we are left with are faith-based initiatives, i.e. a speculation that if we do X we will achieve Y but with no convincing evidence or rationale for believing that.

The proper counsel is to try to break the problem down into ever smaller increments in order to get to some foundation of concrete actions that can be executed with high confidence and keep experimenting and adding elements till the larger goal is achieved. What usually happens instead is that we remain focused on the big picture and the bold action and, because it is faith-based, it is a dichotomous decision. We either have the votes to implement the bold action or we don't. If we don't, the issue goes away for a while. If we do have the votes, we then are locked in to a long experiment (think of the multidecadal Head Start program which has never achieved its intended goals despite close to $10 billion a year).

Usually these exercises in faith-based altruism end ruinously. But we keep doing it.

An example where confident action is advocated in the face of uncertain information comes in Stop Trying to Save the World by Michael Hobbes.

Armed with his rigorously gathered results, Kremer founded an NGO, Deworm the World. He launched it at the 2007 World Economic Forum and committed to deworming ten million children. He was feted by the Clinton Global Initiative; GlaxoSmithKline, and Johnson & Johnson pledged $600 million worth of deworming treatments a year, enough for every infected primary school student in Africa. The World Health Organization issued a statement of support. Kenya asked him to help create a national program to deworm 3.6 million children. Two states in India initiated similar programs, aiming to treat millions more. The organization now claims to have helped 40 million children in 27 countries.

But wait a minute. Just because something works for 30,000 students in Kenya doesn’t mean it will work for millions of them across Africa or India. Deworm the World’s website talks a lot about its “evidence-based” approach. (It has now been folded into an NGO called Evidence Action.) Yet the primary evidence that deworming improves education outcomes is from Kremer’s single Kenya case and a post-hoc analysis of deworming initiatives in the American South in 1910. In 2012, the organization said that it had treated 17 million children in India, but didn’t report whether their attendance, school performance, or graduation rates improved.

I keep thinking I’m missing something really obvious, that I’m looking at the wrong part of their website. So I call up Evidence Action and ask: Are you guys really not testing how deworming affects education anymore?

“We don’t measure the effects on school attendance and school performance,” says Alix Zwane, Evidence Action’s executive director. At the scale they’re going for in India, entire states at a time, splitting into control and treatment groups simply wouldn’t be feasible.

Kremer tells me that enough trials have been done to warrant the upscaling. “There’s more evidence for this than the vast majority of things that governments spend money on.” Every time you want to build a new road, you can’t stop to ask, Will this one really help people get from place to place?

“Meanwhile,” he says, “there’s a cohort of children that, if you don’t implement the policy now, will go through years of schooling without treatment.”

It’s an interesting question—when do you have enough evidence to stop testing each new application of a development idea?—and I get that you can’t run a four-year trial every time you roll out, say, the measles vaccine to a new country. But like many other aid projects under pressure to scale up too fast and too far, deworming kids to improve their education outcomes isn’t the slam-dunk its supporters make it out to be.

And here's the kicker.

In 2000, the British Medical Journal (BMJ) published a literature review of 30 randomized control trials of deworming projects in 17 countries. While some of them showed modest gains in weight and height, none of them showed any effect on school attendance or cognitive performance. After criticism of the review by the World Bank and others, the BMJ ran it again in 2009 with stricter inclusion criteria. But the results didn’t change. Another review, in 2012, found the same thing: “We do not know if these programmes have an effect on weight, height, school attendance, or school performance.”

In this instance, a well intentioned advocate ran a successful program in a single country and did a rigorous review of the very positive benefits. Based solely on that single experience, they then scale to multiple locations in multiple countries but with no further evidence of efficacy.

These are good people seeking to solve real problems. What can you do?

I would argue that there is a conceptually simple and behaviorally difficult approach that everyone could adopt. Acknowledge that reality is hiding from you and ask whether you have worked hard enough to find that reality which might be true and useful. Is the effect real? Is it material? Do we understand the causes? What is the context? What are the critical assumptions? Can we change it? Can we measure success? What are the trade-offs? It is most commonly the case that these uncomfortable questions have been glossed over and assumptions made without an empirical basis for those assumptions.

The issues are often real. We usually want to help. The argument is usually logically persuasive. But how do you distinguish persuasive ideas from effective ideas? Try the idea in controlled conditions and see if it produces the results expected. Then incrementally expand the test to a larger and larger set of circumstances. Slower and less glamorous and more demanding but also more likely to be contributive and less likely to have significant unexpected consequences.