The design and practice of better architectures of informational relevance

From Death by PowerPoint: the slide that killed seven people by James Thomas. An interesting and useful argument which I think is both useful and wrong.

No, PowerPoint did not kill seven people. It was one element in a long chain of improbable events or failures which contributed to a bad decision which led to seven deaths. Almost all bad outcomes in complex dynamic systems are multi-causal in nature. To pick out a single contributing element is almost always incorrect.

Here is the case Thomas is making.

It was impossible to tell from Earth how much damage this foam, falling nine times faster than a fired bullet, would have caused when it collided with the wing. Foam falling during launch was nothing new. It had happened on four previous missions and was one of the reasons why the camera was there in the first place. But the tile the foam had struck was on the edge of the wing designed to protect the shuttle from the heat of Earth’s atmosphere during launch and re-entry. In space the shuttle was safe but NASA didn’t know how it would respond to re-entry. There were a number of options. The astronauts could perform a spacewalk and visually inspect the hull. NASA could launch another Space Shuttle to pick the crew up. Or they could risk re-entry.

NASA officials sat down with Boeing Corporation engineers who took them through three reports; a total of 28 slides. The salient point was whilst there was data showing that the tiles on the shuttle wing could tolerate being hit by the foam this was based on test conditions using foam more than 600 times smaller than that that had struck Columbia. This is the slide the engineers chose to illustrate this point:

Click to enlarge.

NASA managers listened to the engineers and their PowerPoint. The engineers felt they had communicated the potential risks. NASA felt the engineers didn’t know what would happen but that all data pointed to there not being enough damage to put the lives of the crew in danger. They rejected the other options and pushed ahead with Columbia re-entering Earth’s atmosphere as normal. Columbia was scheduled to land at 0916 (EST) on February 1st 2003. Just before 0900, 61,170 metres above Dallas at 18 times the speed of sound, temperature readings on the shuttle’s left wing were abnormally high and then were lost. Tyre pressures on the left side were soon lost as was communication with the crew. At 0912, as Columbia should have been approaching the runway, ground control heard reports from residents near Dallas that the shuttle had been seen disintegrating. Columbia was lost and with it her crew of seven. The oldest crew member was 48.

[snip]

Edward Tufte, a Professor at Yale University and expert in communication reviewed the slideshow the Boeing engineers had given NASA, in particular the above slide. His findings were tragically profound.

His findings:

Firstly, the slide had a misleadingly reassuring title claiming that test data pointed to the tile being able to withstand the foam strike. . . .

Secondly, the slide contains four different bullet points with no explanation of what they mean. This means that interpretation is left up to the reader. Is number 1 the main bullet point? Do the bullet points become less important or more? It’s not helped that there’s a change in font sizes as well. In all with bullet points and indents six levels of hierarchy were created. This allowed NASA managers to imply a hierarchy of importance in their head: . . .

Thirdly, there is a huge amount of text, more than 100 words or figures on one screen. Two words, ‘SOFI’ and ‘ramp’ both mean the same thing: the foam. Vague terms are used. . . .

Finally the single most important fact, that the foam strike had occurred at forces massively out of test conditions, is hidden at the very bottom. . . .

All Thomas is saying is true and ineffective communication was a contributor to bad decision-making.

My nitpick is that "contributing to" a bad decision does not mean the same as "caused". We seek silver bullets for complex problems. We seek the magical one thing which fixes everything. But in complex dynamic systems, that is not the operational paradigm and the sooner we quit treating complex dynamic systems as the more familiar deterministic static systems, the better.

It is like a general vet practitioner doing heart surgery on a human. Sure, the fundamentals are similar but they are not the same. Decision-making in deterministic static systems invokes similar approaches and tools but it is definitely not the same as decision-making in complex dynamic systems.

Thomas's underlying point is a good one and also true but not explicitly made. Effective communication in a constrained environment with heterogenous participants about a complex dynamic system is exceedingly difficult. PowerPoint is a tool and a useful one at that but it is only a tool. The underlying problem is with the practitioners using it.

Omitting PowerPoint from the Columbia decision-making process would unlikely have prevented the accident.

We can see some of the issues in the very slide Thomas is using to make his argument. It is a mess. But why?

This is where Tufte's principles come in - We have to understand what it is that is being communicated and then figure how it needs to be communicated to the audience that needs to use the information being communicated. There is both an aesthetic art and a hard science involved in this and it is true regardless of whether one is using excel or powerpoint or word or performance art - they are all simply means to an end.

Ideally, to make a good decision in a complex dynamic system, you need current information, accurate information, contextual information, variance information (how frequently is it true), significance information (how consequentially true is it) in terms that everyone understands in as logical a sequence as possible with no more information than is necessary and no less information than is needed. That is a tall order, a self-contradicting order and an impossible order.

More than all, you need trust and respect among the decision-makers. The more heterogeneous the decision-making group, the fewer the bonds among them, the greater the possible antagonisms, the more consequential the decision, the more likely it is that communication will be presented defensively. In other words, the greater the inclination to forestall objections, anticipate questions, and provide context. In other words, the greater the noise to signal ratio.

You can see the problem in the slide above. It is essentially a scattered set of statements, more or less relevant, written under the pressure of time, with no clear architecture of relevance.

The fact that it is in PowerPoint is not the issue. It could as easily be in word or simply spoken. The problem is the absence of an architecture of relevance. As John Locke noted in An Essay Concerning Human Understanding:

I will not deny, but possibly it might be reduced to a narrower Compass than it is; and that some Parts of it might be contracted: The way it has been writ in, by Catches, and many long Intervals of Interruption, being apt to cause some Repetitions. But to confess the Truth, I am now too lazy, or too busy to make it shorter.

Or Blaise Pascal in 1657, excusing his long letter to a correspondent:

I have made this longer than usual because I have not had time to make it shorter.

If we focus our attention on PowerPoint, it diverts us from the real challenge - how do we design better architectures of informational relevance when dealing with complex dynamic systems and heterogenous audiences and how do we cultivate the practice of those architectures among those needing to communicate.