Poor methods get results

From The Lancet’s editor-in-chief Dr. Richard Horton in the April 11th edition. commenting on a symposium examining peer review and the quality of science reporting. I found this quote in Science publication is hopelessly compromised, say journal editors from American Council on Science and Health.

“A lot of what is published is incorrect.” I’m not allowed to say who made this remark because we were asked to observe Chatham House rules. We were also asked not to take photographs of slides. Why the paranoid concern for secrecy and non-attribution? Because this symposium — on the reproducibility and reliability of biomedical research — touched on one of the most sensitive issues in science today: the idea that something has gone fundamentally wrong with one of our greatest human creations.

The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness. As one participant put it, “poor methods get results”.

I like that list. Indicators of cognitive pollution:

Small sample sizes
Tiny effect sizes
Invalid exploratory analyses
Flagrant conflicts of interest
An obsession for pursuing fashionable trends of dubious importance

I was originally led to this through the comments related to this unfortunate post, The North-South Divide on Two-Parent Families by David Leonhardt. Leonhardt often does a good job of reporting on interesting work, but he is certainly hostage to the prejudices and biases of the New York Times.

In this case, there is an effort to arrive at a conclusion by ignoring critical information. An omission called out by the commenters. The argument attempting to be made is:

When it comes to family arrangements, the United States has a North-South divide. Children growing up across much of the northern part of the country are much more likely to grow up with two parents than children across the South.

[snip]

Mr. Wilcox and Mr. Zill argue that there are actually two models for having a large share of stable families: the blue-state model and the red-state one.

In the blue-state model, Americans get more education and earn higher income — and more educated, higher-earning people tend to marry and stay married. In Minnesota, New Jersey, Massachusetts and Connecticut, at least 51 percent of teenagers are being raised by both biological parents, among the highest rates in the nation. (That figure excludes families in which the two parents are together without being married; such arrangements are still rare — and less likely to last than marriages.)

In the red-state model, educational attainment is closer to average, but “residents are more likely to have deep normative and religious commitments to marriage and to raising children within marriage,” write Mr. Wilcox and Mr. Zill, in a paper for the Institute for Family Studies. This model applies across much of the Great Plains and Mountain West, including Nebraska and Utah.

The article is trying to argue that family stability is primarily a function of higher education attainment (in the north) or religiosity (in the south).

There are a lot of other issues with the research, called out both in the article itself or in the comments. But the biggest issue is that this is not a map of family stability through education and religiosity and geography, this is a map of minority representation.
As commenters point out, the researchers try to ignore the correlation by dismissing it.

Mr. Wilcox and Mr. Zill also point out that two-parent families tend to be more common in states with predominantly white populations. But race is hardly the only explanation for the patterns. White single-parent families have become much more common in recent years. And in the Deep South, single parenthood is common among both whites and blacks.

And technically the researchers are correct. This has nothing to do with race per se. There is no biological or genetic necessity driving these outcomes. Race, at least in the US, is a general proxy for culture, i.e. individuals of different ethnic groups, on average, also demonstrate different cultural attributes, behaviors and expectations.

You want more familial stability? Focusing on education and religiosity won't get you very far. Focusing on behavioral attributes will get you a lot further. That's the danger of these shy articles which attempt to avoid hard conversations by diverting attention to non-root cause issues. They postpone actually addressing the real problems. Not education. Not religiosity. Not race. Not geography. Behaviors and values and expectations.

UPDATE: A statistical refutation of Leonhardt's position at Upshot sells Wilcox, but I’m not buying by Philip N. Cohen.