The crisis fleeting; experience perilous

Excellent explication. From Is it fair to say that most social programmes don’t work? from 80,000 Hours.

And the answer is yes, most social programs don't work. What's valuable is not so much the conclusion (though that is useful information) but rather the detailed analysis supporting the conclusion.

The original finding is from 2008 by researcher David Anderson.

The vast majority of social programs and services have not yet been rigorously evaluated, and…of those that have been rigorously evaluated, most (perhaps 75% or more), including those backed by expert opinion and less-rigorous studies, turn out to produce small or no effects, and, in some cases negative effects.

Nine years later, 80,000 hours is seeking to validate whether that conclusion is still sustainable.

But in exploring various sources to test that earlier work, they get into a lot of necessary detail.

What counts as the numerator? Is it all suggested projects?, just those with a published methodology?, only those with an RCT?, only those which have been replicated?, only those which have been scaled?

The lower down the pyramid is the numerator, the higher will be the failure rate.

Similarly, what is the specification for successful?

Sample size?
p>5%?
Effect size?
Standard deviation of effect?
Duration of effect?
Value of realized benefits for the specified target > Anticipated costs + unanticipated costs?
All of the above?

If you include only those projects which are funded, have a well articulated goal, have a p>5%, have a positive effect size, with some durability of outcomes, and have few unintended costs, you have a smaller sample size and yet the success rate is still pretty abysmal. Very roughly, a failure rate of ~80% seems the norm.

This about what I see with Fortune 500 corporations and major initiatives. Less than 10% come in on-time, on budget, AND achieve the stated measures of success. 70% are cancelled or otherwise fail on at least two of the criteria.

That is a useful heuristic to have in mind when considering any proposition - 80% of policies will fail. As Hippocrates said, "Life is short, and Art long; the crisis fleeting; experience perilous, and decision difficult."

A good discussion of failure rates in government policy is that in Can Government Replicate Success? by Stuart M. Butler & David B. Muhlhausen.

Without a lot of ordering, here are some of the most important systemic issues I see in policy/project decision-making exercises, both in government, non-profits, and in business. They all make many of these errors with some degree of frequency, though different sectors are more inclined to particular errors than others.

Time constrained decision-making

Unavailability of pertinent and reliable data

Dearth of implementation expertise

Tunnel vision (instead of 360 degree perspective)

Lack of goal clarity

Absence of goal measurement

Ignorance of incentive structures

Poor or no goal prioritization

No awareness or consensus on goal trade-off issues

No skin in the game (Beneficiaries need to either drive the process or have input)

Quantification aversion

Avoidance of negative feedback mechanisms

Disposition to plan-invariance (don't change the plan as new information becomes available)

Incomprehension of causal paths, context components, and parameter clarity.

Insensitivity to unintended negative consequences

Beyond these particular issues, I think there are some dynamics to government decision-making which create distinct problems.

1) It is hard to get strapped government agencies to expend resources for a strong assessment and knowledge tracking process so that policy proposals can be systematically evaluated. (Lack of transparency of what actually works)

2) Causal density and complexity make it difficult for many participants in a process to comprehend the dramatic difference between linear, deterministic decision-making and complex, contingent decision-making. Most human issues are characterized as complex and contingent and yet most decision-making processes applied to them are linear and deterministic. (Misunderstanding of what outcomes can actually be anticipated)

3) Advocacy groups unable to persuade the general public of the legitimacy of their issue or whose solution requires the power of coercion only available from government have to seek their funding and/or authority from the government. To break through the cacophony of those seeking governmental endorsement, these advocates have to create a message that elevates their concern to that of an existential threat. The problem may be real but the claim outstrips the evidence. Income inequality, early childhood education needs, global anthropogenic climate change, rape culture, systemic racism, gender wage gaps, guaranteed basic income, etc. are all policy examples where there is some legitimate kernel of a real issue which is blown all out of proportion in order to warrant the coercion or funding that advocates cannot otherwise obtain from the general public. (Systemic overpromising and underdelivering)

4) Overambition/Silver bullet syndrome. Partly a function of the 2 and 3 above. A complex process is simplified beyond recognition, a policy is proposed for the simplified model, the policy addresses only one element of the complex process, and an ideal outcome is promised. "If you do X, you will get Y." Decision-makers are frequently alarmed by the risks and nuance attendant to complex systems and therefore allow themselves to be persuaded that it is not a complex system but a simple one. One for which there is a silver bullet solution.

UPDATE: An example of some of the issues discussed above: The Afterlife of Big Ideas In Education Reform by Michael Hobbes. A silver-bullet solution to a complex problem, implemented without understanding the critical steps that made the trial successful.

Its almost as if the only successes occur locally under gifted leadership. As if there are no scalable solutions to complex systems.