"Science says" is a fallacious appeal to authority with no evidentiary validity

The other day I posted Contact hypothesis bites the dust. Or at least it takes a big stumble.

In that post, I noted that the researcher found 515 papers on their topic of interest, reporting the results from 713 studies. When they screened those 515 papers and 713 studies to find those that had the required, and bare, minimum of necessary attributes such as articulated methodology, etc. they were left with only 8 papers and 9 studies.

In other words, only 1.5% of papers over some sixty years of research in the field of Contact Hypothesis demonstrated the basics of what you would expect for a solid scientific program of research.

Mother Jones has a different report which is similar in suggesting that much of what passes for research out of the academy is pretty much bunkum. From Chart of the Decade: Why You Shouldn’t Trust Every Scientific Study You See by Kevin Drum.

In this instance, the field is somewhat more scientific than psychology and sociology. This has to do with medical research where lives (and billions of dollars) are at stake. One might expect that there would be better and more robust scientific protocols than one might anticipate in the wannabe fields of sociology and psychology. But one would be wrong.

The authors collected every significant clinical study of drugs and dietary supplements for the treatment or prevention of cardiovascular disease between 1974 and 2012. Then they displayed them on a scatterplot.

Prior to 2000, researchers could do just about anything they wanted. All they had to do was run the study, collect the data, and then look to see if they could pull something positive out of it. And they did! Out of 22 studies, 13 showed significant benefits. That’s 59 percent of all studies. Pretty good!

Then, in 2000, the rules changed. Researchers were required before the study started to say what they were looking for. They couldn’t just mine the data afterward looking for anything that happened to be positive. They had to report the results they said they were going to report.

And guess what? Out of 21 studies, only two showed significant benefits. That’s 10 percent of all studies. Ugh. And one of the studies even demonstrated harm, something that had never happened before 2000

Reports for all-cause mortality were similar. Before 2000, 5 out of 24 trials showed reductions in mortality. After 2000, not a single study showed a reduction in mortality.

And the results?

Click to enlarge.
Drum translates:

Before 2000, researchers cheated outrageously. They tortured their data relentlessly until they found something—anything—that could be spun as a positive result, even if it had nothing to do with what they were looking for in the first place. After that behavior was banned, they stopped finding positive results. Once they had to explain beforehand what primary outcome they were looking for, practically every study came up null. The drugs turned out to be useless.

Not everyone in the academy is a fraud or a cheat. There is good science being done. But we can never forget that we are all people working in human systems with frequently skewed incentives. Every time someone tries to seal an argument by claiming that "Science says . . . " or "Studies indicate . . . ", or "The science is settled . . .", that these are all simply appeals to authority. More than that, they are a direct appeal to sublimate individual skepticism to the authority of "science." These are deceptive and malicious arguments. Science says nothing. You have to always go to the data yourself rather than rely on unknown "experts" whose incentive structures are biased and to which they consciously or unconsciously respond. As do we all.