The current methods need substantial improvement to produce trustworthy scientific evidence

I am always on the lookout for well articulated means of assessing arguments and creating proof of an argument. Alvan R. Feinstein tackled this in a paper back in 1983, Scientific Standards in Epidemiologic Studies of the Menace of Daily Life. He was specifically attempting to address the issue that many epidemiologic studies were not compliant with the scientific process and therefore, the results were at the very best indicative and most often useless.

While his paper is specifically directed at the technical requirements of epidemiology, they have general utility.

He identifies five necessary scientific standards. His requirement, my discussion in italics.

1) A stipulated research hypothesis. To plan an experimental trial, the investigator identifies the cause-effect comparison that will be tested as the research hypothesis. It is astonishing to me how far arguments go before people get around to specifying exactly what their hypothesis is.

2) A well specified cohort. In randomized trials, the cohort under study is well specified by examinations done before the exposure (or nonexposure) begins. A lot of arguments turn on the fact that apples are not being compared to apples.

3) High-quality data. While admitting and following the individual people studied in an experiment, the investigators can get relatively high-quality data because each person is directly examined with methods that can be carefully calibrated for their reproducibility and validity. It seems that in many arguments data integrity and validity are mostly an afterthought, if considered at all. Way too many arguments are advanced with no data at all. The next largest population are arguments where the data is alluded to but not presented. The third largest are arguments where the data is said to exist but are not presented for review. Finally, in minimal numbers, there are arguments where the argument is advanced with relevant and robust data which is provided for review and assessment.

4) Analysis of attributable actions. An ideal experimental design should allow an observed agent to be held responsible for the outcomes that follow it. Human issues are almost invariably dense and multicausal. It is hard to isolate one factor from others in order to isolate causation. In addition, most arguments advance a particular position but do nothing to disprove the alternate hypotheses.

5) Avoidance of detection bias. The double-blinding process that keeps both investigators and recipients unaware of the assigned maneuvers has several important roles in a randomized trials. It is pretty well documented that people, including scientists, see what they want to see and will reinterpret, discard or ignore that which is inconsistent with their expectations.