Randomness, path dependency, popularity and complex dynamic systems

This article, Is the Tipping Point Toast? by Clive Thompson, is a dated review of a good book by Duncan Watts. Some very good passages.

On the Gatekeeper/Influencer model.

But Watts, for one, didn't think the gatekeeper model was true. It certainly didn't match what he'd found studying networks. So he decided to test it in the real world by remounting the Milgram experiment on a massive scale. In 2001, Watts used a Web site to recruit about 61,000 people, then asked them to ferry messages to 18 targets worldwide. Sure enough, he found that Milgram was right: The average length of the chain was roughly six links. But when he examined these pathways, he found that "hubs"—highly connected people—weren't crucial. Sure, they existed. But only 5% of the email messages passed through one of these superconnectors. The rest of the messages moved through society in much more democratic paths, zipping from one weakly connected individual to another, until they arrived at the target.

Why did Milgram get it wrong? Watts thinks it's simply because his sample was so small—only a few dozen letters reached their mark. The dominance of the three friends could have been a statistical accident. "And since Milgram's finding sort of made sense, nobody even bothered to redo the experiment," Watts shrugs. But when you perform the experiment with hundreds of successfully completed letters, a different picture emerges: Influentials don't govern person-to-person communication. We all do.

The more Watts examined the theory of Influentials, the less sense it made to him. The problem, he explains over lunch in a Midtown restaurant, is that it's incredibly vague. None of its proponents ever clearly explain how an Influential actually influences.

"It sort of sounds cool," Watts says, tucking into his salad. "But it's wonderfully persuasive only for as long as you don't think about it."

Further testing on the ideas behind the influencer model.

That may be oversimplifying it a bit, but last year, Watts decided to put the whole idea to the test by building another Sims-like computer simulation. He programmed a group of 10,000 people, all governed by a few simple interpersonal rules. Each was able to communicate with anyone nearby. With every contact, each had a small probability of "infecting" another. And each person also paid attention to what was happening around him: If lots of other people were adopting a trend, he would be more likely to join, and vice versa. The "people" in the virtual society had varying amounts of sociability—some were more connected than others. Watts designated the top 10% most-connected as Influentials; they could affect four times as many people as the average Joe. In essence, it was a virtual society run—in a very crude fashion—according to the rules laid out by thinkers like Gladwell and Keller.

Watts set the test in motion by randomly picking one person as a trendsetter, then sat back to see if the trend would spread. He did so thousands of times in a row.

The results were deeply counterintuitive. The experiment did produce several hundred societywide infections. But in the large majority of cases, the cascade began with an average Joe (although in cases where an Influential touched off the trend, it spread much further). To stack the deck in favor of Influentials, Watts changed the simulation, making them 10 times more connected. Now they could infect 40 times more people than the average citizen (and again, when they kicked off a cascade, it was substantially larger). But the rank-and-file citizen was still far more likely to start a contagion.

Influencing versus receptivity.

Why didn't the Influentials wield more power? With 40 times the reach of a normal person, why couldn't they kick-start a trend every time? Watts believes this is because a trend's success depends not on the person who starts it, but on how susceptible the society is overall to the trend—not how persuasive the early adopter is, but whether everyone else is easily persuaded. And in fact, when Watts tweaked his model to increase everyone's odds of being infected, the number of trends skyrocketed.

"If society is ready to embrace a trend, almost anyone can start one—and if it isn't, then almost no one can," Watts concludes. To succeed with a new product, it's less a matter of finding the perfect hipster to infect and more a matter of gauging the public's mood. Sure, there'll always be a first mover in a trend. But since she generally stumbles into that role by chance, she is, in Watts's terminology, an "accidental Influential."

Critical thinking.

No researcher, he points out—including Keller—ever analyzes interactions between specific Influentials and the friends they're supposedly influencing; no one observes influence in action. In essence, Keller appeals to common sense—our intuitive sense of how the world works. Watts thinks common sense is misleading.

Mind you, Watts does agree that some people are more instrumental than others. He simply doesn't think it's possible to will a trend into existence by recruiting highly social people. The network effects in society, he argues, are too complex—too weird and unpredictable—to work that way. If it were just a matter of tipping the crucial first adopters, why can't most companies do it reliably?

As Watts points out, viral thinkers analyze trends after they've broken out. "They start with an existing trend, like Hush Puppies, and they go backward until they've identified the people who did it first, and then they go, 'Okay, these are the Influentials!'" But who's to say those aren't just Watts's accidental Influentials, random smokers who walked, unwittingly, into a dry forest? East Village hipsters were wearing lots of cool things in the fall of 1994. But, as Watts wondered, why did only Hush Puppies take off? Why didn't their other clothing choices reach a tipping point too?

Randomness and path dependency.

Actually, if you believe Watts, the world isn't just complex—it's practically anarchic. In 2006, he performed another experiment that chilled the blood of trendologists. Trends, it suggested, aren't merely hard to predict and engineer—they occur essentially at random.

Watts wanted to find out whether the success of a hot trend was reproducible. For example, we know that Madonna became a breakout star in 1983. But if you rewound the world back to 1982, would Madonna break out again? To find out, Watts built a world populated with real live music fans picking real music, then hit rewind, over and over again. Working with two colleagues, Watts designed an online music-downloading service. They filled it with 48 songs by new, unknown, and unsigned bands. Then they recruited roughly 14,000 people to log in. Some were asked to rank the songs based on their own personal preference, without regard to what other people thought. They were picking songs purely on each song's merit. But the other participants were put into eight groups that had "social influence": Each could see how other members of the group were ranking the songs.

Watts predicted that word of mouth would take over. And sure enough, that's what happened. In the merit group, the songs were ranked mostly equitably, with a small handful of songs drifting slightly lower or higher in popularity. But in the social worlds, as participants reacted to one another's opinions, huge waves took shape. A small, elite bunch of songs became enormously popular, rising above the pack, while another cluster fell into relative obscurity.

But here's the thing: In each of the eight social worlds, the top songs—and the bottom ones—were completely different. For example, the song "Lockdown," by 52metro, was the No. 1 song in one world, yet finished 40 out of 48 in another. Nor did there seem to be any compelling correlation between merit and success. In fact, Watts explains, only about half of a song's success seemed to be due to merit. "In general, the 'best' songs never do very badly, and the 'worst' songs never do extremely well, but almost any other result is possible," he says. Why? Because the first band to snag a few thumbs-ups in the social world tended overwhelmingly to get many more. Yet who received those crucial first votes seemed to be mostly a matter of luck.

Word of mouth and social contagion made big hits bigger. But they also made success more unpredictable. (And it's worth noting, no one in the social worlds had any more influence than anyone else.) So yes, Watts figures, if you rewound the world to 1982, Madonna would likely remain a total unknown—and someone else would have slipped into her steel-tipped corset. "You cannot predict in advance whether a band gets this huge cascade of popularity, because the social network is liable to throw up almost any result," he marvels.

My take-aways from this research:

Cascades occur randomly
There is a threshold of viability but beyond that, there is little predictability
Network effects dominate
Network effects magnify small variances

These lessons would explain some notable patterns in the book business. Publishing is awash with examples of books that were passed over multiple times and then became enormous bestsellers (Harry Potter, Fifty Shades of Grey, etc.). I suspect that what shows up in the bestseller lists today are substantially driven by the processes identified by Duncan Watts.

However, and this is interesting, there clearly is a separate process in play regarding long term viability of books. In other words, if you pick any random year in the past, say 1955, the books that were bestsellers in that year are rarely still in print. But likewise, if you look at the books still in print that were first published in 1955, they were rarely bestsellers at that time. There is a similar dynamic with regards to literary awards. The books that receive the most critical acclaim and awards in one particular year are rarely the ones that survive over the longer term. The two exceptions of which I am aware are the Newbery Medal and the Caldecott in children's literature, both of which are virtual guarantors of long term resilience.

The distributed, uncoordinated but networked process which leads to a long term emergent order is, to me, the most interesting dynamic here and the most mysterious. My best guess is that it has something to do with intra-cohort network effects.

In other words, commercial success at a given point in time can be achieved through a cascade effect within a single cohort such as five year old boys (Thomas the Tank Engine) or forty year old women (Twilight) or everyone in a calendrical period (Jonathan Livingston Seagull), etc. The books that will last are those that cross into at least a few other cohorts within a given year as well as establishing themselves within multiple (evolving) cohorts over time. Sixty seconds over Tokyo might have appealed to an entire nation newly at war in 1942 but has established itself firmly with the cohorts of historians in general, military historians in particular, and young boys.

Intense appeal within a networked cohort explains short term commercial success but long term viability depends upon appeal to multiple longstanding cohorts.