In social sciences, the null hypothesis is that nothing makes a long-term, scalable, replicable difference

From The Null Hypothesis in Education is Hard to Disprove by Arnold Kling.

In education, the null hypothesis is that nothing makes a long-term, scalable, replicable difference. That is:

1. Take any pedagogical innovation or educational intervention.

2. Subject it to a controlled experiment.

3. Evaluate the experiment’s outcome several years later.

4. If the experiment works, attempt to replicate the experiment in more situations.

By the time you reach step 4, if not sooner, you will be unable to show that the innovation makes any difference in outcomes. What this suggests to me is that in the long run it is the characteristics of the students that determine outcomes, at least on average. Think of an individual student as “predestined” to reach a certain outcome. An educational intervention can disturb their path to the predestined outcome but will not change the outcome. I do not literally believe this model, but it is a null hypothesis that is difficult to disprove.

I would expand this null hypothesis to the social sciences in general, not just education.

I would also argue that this is an example of the argument I have made elsewhere on Thingfinder, that one of the challenges in the social sciences is that the pace of social change (along with technology, regulation, demographic, etc.) effectively precludes the application of the scientific method. You can never step in the same river twice. You run a longitudinal study showing pedagogical practices X in grade school in the 1970's are associated with desirable life outcome changes Y in the 2000s. According to the scientific method, you now have to replicate these findings. But technological, regulatory, societal, and other changes mean that there are material differences between your grade school population in 2015 and your grade school population in 1970. You are now, on an empirical basis, comparing apples and oranges. In other words, for reasons beyond your control, you cannot actually replicate the conditions of your experiment conducted in 1970 in 2015.

That is the tragedy and paradox of social sciences. They want to be respected like the hard sciences and yet the nature of the field precludes them from using the methodologies and techniques which allow the hard sciences to be respected. Social Sciences can give us indications or informed speculation but it struggles to produce anything that is robust and replicated.