Cargo Cult Science

Female-named hurricanes kill more than male hurricanes because people don’t respect them, study finds by Jason Samenow

But of course it doesn't find that at all owing to glaring errors in the structure of the study. That the Washington Post should have let this one slip through is a bit embarrassing. This is a quintessential example of researchers finding what they were seeking, of the logical fallacy of confirmation bias.

And its not like they are hiding the flaws.

Researchers at the University of Illinois and Arizona State University examined six decades of hurricane death rates according to gender, spanning 1950 and 2012. Of the 47 most damaging hurricanes, the female-named hurricanes produced an average of 45 deaths compared to 23 deaths in male-named storms, or almost double the number of fatalities. (The study excluded Katrina and Audrey, outlier storms that would skew the model).

The first red flag is right there - "The study excluded Katrina and Audrey, outlier storms that would skew the model." In other words they can arrive at the conclusion that "Female-named storms have historically killed more" people by excluding two of the most lethal storms which happen to have female names. By including all storms, it would likely have high-lighted the fact that the results are essentially random. You are dealing with a relative small population (the number of hurricanes that reached shore in the designated time frame) with hugely varying attributes (Force measurement, speed, location/storm surge, duration). The researchers appear to be trying to hide the obvious randomness of the data by omitting data that call attention to that randomness.

Any study that manipulates data, particularly by subjectively excluding data points inconsistent with the conclusion should be tossed out on its ear.

The second issue in the quoted paragraph is a little less obvious, it does require at least some contextual knowledge. Their study range is 1950-2012. However, male names were not introduced till 1979. If they wished to compare apples-to-apples, they would have looked only at the period 1979-2012. Why? Changes in satellite technology, building codes, emergency preparedness, national wealth, and population density. Earlier storms occurred under different contexts. The population was smaller, less dense and more spread out making emergency responses more difficult. Satellite technology came available in the 1970s helping make hurricane tracking and forecasting much more reliable. The nation is richer now than in the 1950s allowing for better emergency response infrastructure (think of helicopter evacuations). Associated with that increasing wealth have been more and stricter enforcement of building codes which reduce human fatalities.

Any study that fails to compare apples-to-apples should be tossed out.

The third issue, as if any more were needed, is the choice of measurement to determine the degree to which people are responding to the name of hurricanes. The researchers used a measure of hurricane lethality - Deaths per hurricane. But this is an absolute number which needs to be put into a ratio. And it is a single measurement. You always want several measures of a phenomenon to ensure you are not getting a spurious result and preferably the measure is a proportionality or rate measure. For example: two hurricanes strike Long Island, one in 1950 and one in 2000. They are the same force. The hurricane in 1950 kills 100 people and the hurricane in 2000 kills 150 people. Regardless of names, does that prove that people took the first hurricane more seriously than the second. That conclusion does not follow. We need more information. If there are three times as many people living on Long Island in 2000 as there were in 1950, then the death rate is actually much lower than it was. In order to demonstrate that there was some sort of causal relationship between name attitude and outcomes, the researchers need multiple measures, preferably ones that are rate based. Examples: Deaths per 100,000 of affected population, Insurance claims per insured amounts for the affected areas, Federal and Local Emergence Response Expenditures, Emergency Room Visits per 100,000 of affected population. If all show a common response depending on storm name, you have a better probability that there is an actual phenomenon. A single absolute measure tells you almost nothing.

Any study with a single measure in absolute terms should be tossed out.

The original question is not an unreasonable one. Do societal naming attitudes affect emergency response? It seems unlikely, but new answers are discovered in the unlikely places.

The problem is that the study is constructed in a fashion that would earn a high school science project a C or D. How it got past the team, then the department, then the grant process, then the peer review and finally through to the Washington Post is a mystery. Exposed to the public, people are taking it apart.

As it is, you have, according to the Post account, people actually wasting time and breath talking about a study that has no merit.

Richard Feynman referred to this type of study as Cargo Cult Science. It has the appearance of science, it uses big words and mathematics and is presented as a scientific study. But all it is is foolishness dressed up as science or, as is likely in this case, ideology masquerading as science. In addition to the above considerations, the real tell that this is Cargo Cult Science is that there appears to be no effort to find disconfirming evidence. That's what scientists do when they have a hypothesis that seems workable; they look for what would prove it wrong. If you have a hypothesis driven by faith, ideology or world view, you skip the step where you look for disconfirming evidence. Too uncomfortable.

If you really believe that society is misogynist and that that misogyny spills over into real world consequences, then what's a little apples-to-oranges comparison, data manipulation, data exclusion, and picking single measures that might support the desired conclusion. All in a day's work for the clerissy.