Narrative Fallacy

The Black Swan by Nassim Nicholas Taleb

The narrative fallacy addresses our limited ability to look at sequences of facts without weaving an explanation into them, or, equivalently, forcing a logical link, an arrow of relationship upon them. Explanations bind facts together. They make them all the more easily remembered; they help them make more sense. Where this propensity can go wrong is when it increases our impression of understanding.

It simply fails to remove the plausibility of that hypothesis

A great example of the distinction between telling a story (the reporter's interpretation of events) and telling the story (the reporter's report of the facts). Pam Belluck of the New York Times reports on the results of a social psychology study in Better Social Skills, Scientists Recommend a Little Chekhov. Regrettably it is almost 100% gullible interpretation and 0% critical reporting.

The story is that the scientists conducted a study that purports to support the popular notion that reading literary fiction (as opposed to other forms of fiction such as romance, mysteries, etc.) improves one's social skills.

But that is not what was done at all. The reporter has done hardly any reporting and instead focused on telling a story rather than the story. The story that the reporter wants to tell, and that many readers enthusiastically endorse, is that enthusiastic readers of literary fiction have a greater reservoir of empathy developed through that reading of literary fiction and therefore are more socially adept than non-readers of literary fiction.

The original study can be viewed here, Reading Literary Fiction Improves Theory of Mind by David Comer Kidd and Emanuele Castano.

The claims in reports from the NYT and elsewhere regarding what the study yields include:

"The results suggest that reading fiction is a valuable socializing influence."

"But psychologists and other experts said the new study was powerful because it suggested a direct effect — quantifiable by measuring how many right and wrong answers people got on the tests — from reading literature for only a few minutes."

"Experts said the results implied that people could be primed for social skills like empathy, just as watching a clip from a sad movie can make one feel more emotional."

"“This really nails down the causal direction,” said Keith Oatley, an emeritus professor of cognitive psychology at the University of Toronto."

Reading the commenters to the NYT article, there is a clear preponderance, probably 90%, who are interpreting the study to support that which they already knew. But there are some lone voices calling for actual attention to the scientific method.

When reviewing new science findings, you usually look for several key elements in order to determine how much weight to attach to those findings. These usually include such things as

How rigorous was the design of the study?
Are there indications that the study was intended to produce predetermined outcomes?
Who is paying for the study?
Professional standing of the authors
How many people participated?
What was the time duration of the study?
How diverse (age, class, profession, culture, race, religion, etc.) were the participants?
How consistent are the findings with other comparable studies?
How rigorously are counter factuals addressed?
How rigorously are terms defined?
Are alternative views or opinions discussed?
How careful are the researchers to ensure that apples are compared to apples?
How specific are the measures of performance?
To what degree were the studies lab bound versus real world observational?
To what degree does the report address direct results as opposed to proxy results? (Ex. demonstrated empathy versus measured empathy)
To what extent are the participants neutral or have a stake in the study outcome?
Is there a baseline of before and after?
Do they report the absolute and relative degrees of performance improvement?

Are these present in this study or in the NYT summary of the study? Are they present to the degree that would support the enthusiastic reception and overintepretation of the results? Hard to say since the study is gated. But from what is reported in the NYT and Scientific American articles we can answer some of the questions.

How rigorous was the design of the study? - Not at all, see below and see Annals of overgeneralization by Mark Liberman for a good run down on the specific weaknesses.

Are there indications that the study was intended to produce predetermined outcomes? - From the researchers comments, it appears that they found what they were expecting to find.

Who is paying for the study? - Unknown

Professional standing of the authors - Affiliated with The New School for Social Research which is reasonably credible. However, the field of sociology and psychology are plagued with extraordinarily high rates of withdrawn or unreplicated research.

How many people participated? - Unknown. However, the sample of texts being compared was markedly limited, usually three texts in each population of the five experiments. Meaninglessly small populations.

What was the time duration of the study? - Reading segments were 3-5 minutes and then they were immediately tested for empathy. That leaves open the question whether there is any lasting impact, i.e. read for five minutes and then test three hours later or three days later.

How diverse (age, class, profession, culture, race, religion, education attainment, etc.) were the participants? - Unknown but a real likelihood that the participants were unrepresentative across the measures of diversity. They seem to have ensured at least some diversity of age (18-75) but since they were recruited from Amazon, that might imply certain preconditions around pre-existing reading habits, class, income, technology access, education, etc.

How consistent are the findings with other comparable studies? - Evidence not discussed in the NYT article. Another research report published the same month has the exact opposite to this study's finding. For example, What You Read Matters: The Role of Fiction Genre in Predicting Interpersonal Sensitivity by Katrina Fong et al, examining four genres (but not including literary fiction) indicates that "Romance and Suspense/Thriller genres remained significant predictors of interpersonal sensitivity."

How rigorously are counterfactuals addressed? - Not at all.

Are alternative views or opinions discussed? - No. Only those wishing to endorse the findings are reported in the article.

How rigorously are terms defined? - Not at all. Literary fiction is not defined in terms that identifies specific attributes different from popular or genre fiction.

How careful are the researchers to ensure that apples are compared to apples? - Intentionally designed to skew the results towards a favorable outcome for literary fiction. For example, literary non-fiction was intentionally excluded. By failing to make apples-to-apples comparisons, the researchers leave open the glaring possibility that it is the literariness of the writing that might make a difference rather than the fictionality. This obvious act of exclusion reinforces the perception that the researchers were seeking a particular conclusion.

How specific are the measures of performance? - The measures are not reported at all. From elsewhere it appears that a 30 point scale was used for measuring empathy but that is not referenced in the NYT article.

Do they report the absolute and relative degrees of performance improvement? - Not in the NYT article. From elsewhere it appears that the degree of difference in the various experiments was 1-2 points on the 30 point scale. It appears that while the results might have been statistically significant, they were not, in fact, material.

To what degree were the studies lab-bound versus real world observational? - Completely lab-bound.

To what degree does the report address direct results as opposed to proxy results? (Ex. demonstrated empathy versus measured empathy) - Completely reliant on proxy results. The subjects' level of empathy was determined through tests rather than demonstrated behaviors.

To what extent are the participants neutral or have a stake in the study outcome? - Unclear though it appears that there may have been a degree of self-selection among participants. In addition, they were given a nominal compensation for participation.

Is there a baseline of before and after? - No. This is especially critical when dealing with small populations.

So The Story is that a couple of researches in a field notorious for flawed research, conduct a badly designed experiment involving too few samples, with inconsistent comparisons, with selected participants (not random), with poorly defined categories, in an apparent attempt to yield an expected outcome. The reporter revealed hardly any of that, instead telling A Story, the story she would like to believe, and apparently which most of her readers also would like to believe - reading literary fiction makes you a better person.

As it happens, I would like to believe, and I suspect that it is true, that enthusiastic reading is contributive to positive life outcomes. I can demonstrate a correlation for that proposition but am weak on establishing the causative direction and the causative process. I am somewhat skeptical, but open to the proposition, that enthusiastic reading over time might have some contributive effect towards greater degrees of empathy. However, this study does not provide any support to that proposition. Instead, it is just more cognitive pollution; popularly received and endorsed pollution, but pollution none-the-less.

So all the enthusiastic conclusions mentioned above; valuable socializing influence, a direct effect, causal direction, etc.? Tosh! All tosh. People are stating what they want to believe. This study provides no evidence for any of those beliefs.

Commenter zstansfi at Scientific American has a great observation:

This common fallacy needs to be rooted out. An empirical study which is "consistent with" (that is, does not oppose), but yet which remains entirely unsupportive of a hypothesis does not, in fact, strengthen that hypothesis. It simply fails to remove the plausibility of that hypothesis.

It is a city’s people who ultimately ruin or save themselves.

An interesting insight from The Unrise of the Creative Working Class by Richey Piiparinen. Piiparinen is criticizing the fad-de-jour which is that urban centers should renew themselves by attracting the "creative classes."

He casts the discussion in a new light. It is usually all bubbly, feel-goodish. But if you look at it from an economic or class perspective, you could just as easily see this as the self-described "creative clases" attempting to create demand for themselves. Give us free stuff (tax exemptions, better security, subsidized housing, whatnot) and we will grace your city. No nobility here, just simple self-interest at the expense of others.

If this strategy sounds like an overly simplified way to change what ails Detroit and Cleveland, it’s because it is. In fact Florida himself acknowledged this, stating in Atlantic Cities that, “On close inspection, talent clustering provides little in the way of trickle-down benefits [to the poor].” In fact, because housing costs rise, it makes the lives of lower- and middle-income people worse.

But cities keep revitalizing this way because it is a feel-good prescription that is politically palatable. Who hates art, carnivals, drinking, and eating? Displays of abundance provide the incentive to look the other way. Writes Thomas Sowell, “The first lesson of economics is scarcity: There is never enough of anything to satisfy all those who want it. The first lesson of politics is to disregard the first lesson of economics”.

Where does that leave the millions operating on the wrong side of scarcity? Florida’s answer is for cities to somehow convince corporate America to pay their service workers more. While admirable, I doubt Daniel Schwartz, CEO of Burger King, is listening.

Another option would be refocusing the lens through which modern urban revitalization is viewed. The default setting is to compete for scarcity of the educated elite. Instead, we should alleviate the scarcity from the struggling. But flipping this script requires cities to give up on the idea that there is some audience that will save them. It is a city’s people who ultimately ruin or save themselves.

His comments harken to my argument that we have too long focused on poverty and not enough on productivity. Our goal should not only be about alleviating people's material wants. That is a perfectly fine tactical goal in exigent circumstances. Our real goal, our strategic goal, has to be about preparing people to create their own social capital (knowledge, skills, experience, values and behaviors), to achieve the productivity which frees them from the tyranny of the bureaucrat and allows them to be paid-up members of the community, free to make their own decisions.

Actions often end up being dictated not by what is known to be true but by what people believe to be true.

From Ideas that spread fast and slow by Andrew Gelman. Read his post for his discussion, but I liked this example.

What I focus on with this story is the dual issues of Knowledge and Incentives and how the real world rarely works as intended. Specifically, in this case, Columbia knew what the right solution should be but for reasons unclear, took a different approach (Knowledge issue). And then compounded the Knowledge issue by creating incentives for its tenants to choose the less effective approach. All within the world and control of Columbia University.

The example I have in mind is roach extermination. When I worked with Ginger Chew and her colleagues in the school of public health at Columbia several years ago, I learned that the way to get rid of roaches in your apartment is to clean up your apartment, throw away all the open food, put boric acid in the cracks in your floor and walls, and seal up the cracks. It’s not easy but it does the job. But that’s not what they do in the Columbia-owned-and-operated building where I live. What they do is, every month they put a signup sheet up by the elevator and then an exterminator comes into the building and bombs the apartments of everyone on the list. The same people sign up every month, of course. Instead of thinking, “Hey, bombing doesn’t work,” they seem to think that it’s something they need to do monthly. Good business for the exterminators but not so effective at getting rid of roaches.

So why do they do it that way? One thing I’m definitely not going to do is talk with my neighbors and suggest they try a different approach. My impression is that people get very defensive about things like this. Also, I’m no roach expert; really I’d want to bring someone in from the school of public health to have this conversation.

Anyway, my impression is that people like any treatment that feels like “pushing a button” and they don’t like anything that feels like work. And if you tell people that pushing the button doesn’t really work, they get all bristly on you. Even though, in this case I think the effective treatment is ultimately less work than the bomb. Unfortunately, Columbia has it set up so they bomb for free, but they don’t provide a free cleanup and sealing service.

And Gelman's right about the bristly part as well. Actions often end up being dictated not by what is known to be true but by what people believe to be true.

We find no effects on any educational outcomes, including grades, test scores, credits earned, attendance, and disciplinary actions

From Will a Computer at Home Help My Children in School? by Timothy Taylor. Too small a study (N=1,100) and too small a duration (one or two years) to be anything other than indicative. The conclusions?

"Although computer ownership and use increased substantially, we find no effects on any educational outcomes, including grades, test scores, credits earned, attendance, and disciplinary actions. Our estimates are precise enough to rule out even modestly-sized positive or negative impacts. The estimated null effect is consistent with survey evidence showing no change in homework time or other “intermediate” inputs in education."

Taylor comments:

Having a computer at home increased computer use. Students without a computer at home (the "control group") reported using a computer (at school, the library, or a friend's house) about 4.2 hours per week, while students who now had a computer at home (the "treatment group") used a computer 6.7 hours per week. Of that extra computer time , "Children spend an additional 0.8 hours on schoolwork, 0.8 hours per week on games, and 0.6 hours on social networking."

Of course, any individual study is never the final say. Perhaps having access to a home computer for several years, rather than just one year, would improve outcomes. Perhaps in the future, computer-linked pedagogy will improve in a way where having a computer at home makes a demonstrable difference to education outcomes. Perhaps there is some overall benefit from familiarity with computers that pays off in the long run, even if not captured in any of outcomes measured here. It's important to remember that this study is not about use of computers in the classroom or in education overall, just about access to computers at home.

My suspicion is that, as is normal, it is not access to computers per se but the behaviors and values you bring to the task of studying that are the determinant of outcomes.

And the revelry of the loungers shall pass away

Lesson Amos 6:1a, 4-7. New Revised Standard

I love the language of the Bible, mostly the King James Version but others as well.

Alas for those who are at ease in Zion, and for those who feel secure on Mount Samaria. Alas for those who lie on beds of ivory, and lounge on their couches, and eat lambs from the flock, and calves from the stall; who sing idle songs to the sound of the harp, and like David improvise on instruments of music; who drink wine from bowls, and anoint themselves with the finest oils, but are not grieved over the ruin of Joseph! Therefore they shall now be the first to go into exile, and the revelry of the loungers shall pass away.

The laws of probability, so true in general, so fallacious in particular

From 'Memoirs of my Life and Writings' by Edward Gibbon.

The present is a fleeting moment, the past is no more; and our prospect of futurity is dark and doubtful. This day may possibly be my last: but the laws of probability, so true in general, so fallacious in particular, still allow about fifteen years.

I wonder sometimes whether our popular discourse is not gravely handicapped by an incapacity towards statistical thinking, the kernel of which Gibbon so cleverly catches, "so true in general, so fallacious in particular."

There is little about any human related activity about which we can speak with precise confidence but we can certainly speak with a high degree of statistical confidence. In a given matriculating college class (depending on the university), I can speak with great statistical confidence that 40% will not graduate (or whatever number is applicable to that particular institution) but I cannot with equal confidence identify precisely which of the population will constitute that 40%.

But many people will disavow the utility of the general statistical truth because it cannot be applied with precision to the particular. Whether this is done because of ignorance or cognitive dissonance or has some other root cause, I do not know, but it is a common problem that muddies the waters and holds us back from making the sort of progress that might be beneficial to individuals and communities.

Mere magazine submissions trying to make their way in the world

George Orwell is, to me, a constantly fresh author, always full of surprises. From Looking Through Orwell by Andrew Ferguson. Ferguson offers an explanation for Orwell's freshness - he had multiple unreconciled dimensions to his personality.

It’s odd that a writer who respected transparency and clarity of language above all things should himself be so misunderstood by at least half his admirers. Among the scribbling classes, Orwell fans seem to me to be equally divided between right and left. To cite an easy illustration: Norman Podhoretz’s famous essay from 1983 claimed Orwell as an early incarnation of neoconservatism (a proto-neo!), owing to his staunch anti-communism and pro-Western sympathies. Podhoretz’s essay was furiously rebutted by the late Christopher Hitchens, who later went on to produce a book called Why Orwell Matters, citing his hero’s emphatic atheism, anti-imperialism, and socialism as evidence of his undying identity as a man of the left. Somebody here has got Orwell all wrong.

Maybe both sides do, or, just as likely, it may simply be a fool’s errand to try to cut a writer as expansive and free-ranging as Orwell into categories easily understood today, more than 60 years after his death. His friends didn’t quite get him either. “It was difficult to know what he felt about anything personal,” Malcolm Muggeridge wrote. In his helpful introduction to this volume, editor Peter Davison writes that “Orwell had within his deepest self an unresolved conflict that made him so contradictory a character.” He was frail and sickly for most of his life but possessed enormous physical courage and took pleasure in arduous outdoor labor. At his death, he was an internationally successful author but had never abandoned his belief that he was a miserable failure. He took every opportunity to slag Christianity but consecrated both of his marriages in a church, made sure to baptize his adopted son, and provided for his burial in an Anglican graveyard accompanied by the Book of Common Prayer. He was a professional controversialist and attention-getter who hated to have a fuss made over him by friends.

I liked this observation from the essay.

The letters remind us that works we have come to understand as masterpieces of literary journalism were, once upon a time, mere magazine submissions trying to make their way in the world, blocked at most every turn by the stupidity, tastelessness, and present-mindedness that all editors, except mine, habitually succumb to. “It doesn’t matter about the Tolstoy article,” Orwell wrote to an American editor who temporized about publishing the towering essay “Tolstoy, Lear, and the Fool.” He asks the editor to pass the manuscript on to his American agents. “It’s possible they might be able to do something with it, though as they failed with another…article (one on Swift), perhaps this one is no good for the American market either.” That “(one on Swift)” was “Politics vs. Literature,” a staple of essay collections for two generations and an undoubted masterpiece.

Our study raises the possibility that summer inequities in nonacademic learning may be even more egregious than the academic disparities that past research has emphasized.

From Social Reproduction and Child-Rearing Practices: Social Class, Children's Agency, and the Summer Activity Gap by Tiffani Chin and Meredith Phillips.

Methodologically, our qualitative data clarify why surveys that simply ask about children's participation in various activities, such as whether children went on vacation, attended camp, or practiced academic skills over the summer, will likely miss much of the social class-related variance in children's experiences. These nominally similar activities are so heterogeneous that surveys must ask much more detailed questions about the content of activities and how well they were supervised for sociologists to model the causes of differential summer learning.

Substantively, our study raises the possibility that summer inequities in nonacademic learning may be even more egregious than the academic disparities that past research has emphasized. Because norms about summer "vacation" dictate that it should provide a "break from school," few children-from any social-class background-do rigorous, sustained academic work or practice, other than reading, over the summer (see also, Heyns 1978). Instead, parents use summer as a time to augment the education that children receive during the school year. Over the summer, many children learn more about their religion; develop their talents in music, art, and sports; and gain exposure to new environments that not only provide entertainment, but may stimulate their future interest in music, art, science, history, and culture.

But stark class differences exist in the quantity of these opportunities. Whereas middle-class children who express an interest or talent to their parents typically receive an opportunity to develop it, working-class and poor children rely more heavily on challenging themselves, being lucky in finding free programs, and having friends who can help them develop their talents. Moreover, middle-class children, especially those in families in which their parents or nannies can spend time shuttling them from one activity to the next, have the opportunity to develop numerous talents and gain exposure to a wide array of new environments over the summer. These social-class differences probably produce both a "talent development gap" and a "cultural exposure gap," which, if exacerbated each summer, contribute to disparities in children's future life chances.

The family as school according to John Locke

From Some Thoughts Concerning Education by John Locke.

The well educating of their children is so much the duty and concern of parents, and the welfare and prosperity of the nation so much depends on it, that I would have every one lay it seriously to heart; and after having well examin'd and distinguish'd what fancy, custom, or reason advises in the case, set his helping hand to promote every where that way of training up youth, with regard to their several conditions, which is the easiest, shortest, and likeliest to produce virtuous, useful, and able men in their distinct callings.