Contextual trade-off sensitivity drives the failure of issue polling.

A very interesting piece from Nate Cohn, Affirmative Action Is an Example of How Polls Can Mislead. I quit paying much attention to issue polling perhaps a couple of decades ago because such a high percentage of polls were push polls, designed in such a way as to elicit a predetermined answer. Even where there was some independence and neutrality, polls were incredibly sensitive to language, framing, and context.

From Cohn:

Based on its limited success at the ballot box and my own read of which poll questions make the most sense, my best guess is that affirmative action is fairly unpopular. But I’m troubled by a wider problem: It’s not clear that even a well-worded question would give us much insight into the politics of the issue.

Over the last decade or so, polls have shown that the public backs the liberal or Democratic position on just about every major issue. By these measures, comprehensive immigration reform, environmental protection, gun background checks and many other issues ought to have been political winners for the Democrats. And yet Republicans now hold full control of government in Washington. There’s a loosely held but widespread assumption that many of these same issues have been a part of the Republican resurgence.

It’s not just the issue questions that failed analysts. The poll questions on character also failed to tell the story of the 2016 presidential election.

In pre-election surveys, Donald J. Trump polled worse than Hillary Clinton on just about every question. Voters viewed him more unfavorably, thought he was unqualified for the presidency and, yes, even thought he was more dishonest than Mrs. Clinton. The most straightforward interpretation of the polling — that her clear lead was fairly solid, since it was underpinned by an advantage on whether she was fit to be president — simply did not pan out.

This isn’t a small problem. Journalists have traditionally relied on issue and character questions to frame the story of American elections. It’s how most public pollsters rationalize the cost of horse-race polling. Political consultants often take a similar approach to try to shape their messages, which winds up influencing the promises and programs of elected officials. But the seemingly clear story told by the polls has led us somewhat astray, and it’s probably part of why elected officials, journalists and pollsters were caught off guard in 2016.

So what’s going on?

There aren't any easy and obvious answers.

Over the years, I think we have begun to pay less attention to the specifics of proposed policies and more to the perceived underlying character of the individual proposing the policies. We know that circumstances are always changing and therefore whatever might be promised in a campaign can not be anticipated to be delivered, sometimes at all, sometimes in the original proposed fashion, eighteen or thirty-six months later. We have, I think, gotten a little more sophisticated than we used to be in that regard.

So if we can't take the specifics at face value because we know the details will necessarily have to change, how do we adjudge the proposer of the policy? It has to be based on their perceived character. Is the proposer sincere in their intent? This is related to Salena Zito's observation in the recent election about Trump that "When he makes claims like this, the press takes him literally, but not seriously; his supporters take him seriously, but not literally."

I think this reflects an underlying dynamic, the public were willing to believe in his good intentions, never mind the details. Trump's opponent believed that by being more specific she would be seen to be more credible. Her loss was due to many things in tandem but the deeply rooted perception of her as a self-promoting opportunist with no integrity probably was a difference maker at the margin. I don't think the public trusted either candidate, they just trusted her less because she had a longer track record of reasons to distrust her.

Cohn discusses multiple possible causes for the polling errors and I don't particularly disagree with his analysis. I do, however, think there is something else going on which he doesn't address directly. The issue is how people, individually and collectively, deal with the complexity of goal definition, goal prioritization and goal trade-offs.

I first encountered this issue decades ago working with corporate executive teams as they sought to effect major corporate change; mergers, demergers, acquisitions, turn-arounds, new product deployments, changes in strategic direction, major system implementations, etc.

What I noticed early in my career was that there are three major points of contention that had to be resolved early in order for projects to succeed. Goal definition, goal prioritization and goal trade-offs.

Goal definition sounds easy but is surprisingly challenging. How many goals do you have? How many goals should you have? How do you define them? How will you measure them? It can take a long time to get a team to agree to a singular, workable list of goals.

Even when you get everyone on the same page with the same goals, you still are not out of the woods. Say there are five primary goals and five subsidiary goals. Once everyone is agreed on the list of ten, you have to get them in synch with one another on the ordinal ranking.

If one person ranks Goal A as number one priority and a second person ranks it as number ten in priority, you still have a team out of agreement with one another. Major projects are so inherently risky that you can afford no dissension among the sponsoring executive team. You still have to get them aligned behind an ordinal ranking of the goals.

But there is still an issue. You have to get agreement not just on the ordinal ranking but also on the relative trade-offs between the goals. You are guaranteed to get to a stage in the project where if you give a little on Goal F, you can gain a multiple of Goal C. Is such a trade-off worthwhile? You have to figure out in advance, and get agreement among the executives, on the relative trade-off value among the different goals. It is technically challenging to do this and it is sociologically even more challenging to get a team of stakeholders into alignment.

To be successful, you have to have team endorsement of the defined goals, their ordinal ranking, and their relative trade-offs.

Having done this many times, one thing that is observable is that no matter how long the list of goals, there is a sharp decline in trade-off value among the first two or three goals and the rest. Yes, all the others are very desirable, but their relative trade-off value is small compared to the importance of the first couple or three goals.

And I think this is one of the key challenges with issue polling. There is no way to establish and work through the defined list of issues (goals) or their ordinal ranking or their relative importance.

My sense is that broadly, the top three issues of public concern, year in and year out, are prosperity (aspects of the economy), security (crime and international relations) and health/risk/uncertainty/confidence.

Take a look at Gallup's current list of most important problems identified by Americans. Over time, manifestations of prosperity, security and health/risk remain at the top with heavy ratings and then there is an ever changing constellation of issues which might be perceived as having merit but which show up so far down in the ordinal rankings as to be irrelevant. And some of them are volatile. Over a multi-decade time frame, race relations tends to be steadily down there around 5-10% but in any particular month it can jump to 15-20%.

Look at a couple of those that are far down the list - Education is a most important problem for 2-4% as is Lack of respect for one another. I'll guarantee though, that the trade-off importance for education is far greater than for mutual respect. If there is a proposal to choose between spending $100 billion on education and $100 billion on mutual respect, 99% will make the trade-off of spending the full amount solely on education.

The point of all this is that you lose nuance and context when you do issue polling. Is climate change important? Sure. Is criminal sentencing reform important? Sure. Is welfare reform important? Sure. While all of them are legitimate issues of significant moral importance and sometimes economic importance, they pale against the big three and they all have very weak trade-off values. I have $50 million in the city budget - will I trade-off possible future reduced criminal offense commission by investing in sentencing reform which might reduce recidivism or will I plunk it down on hiring more police.

I think the electorate is relatively sophisticated and, within parameters, understands instinctually some of the trade-offs. In that particular scenario, I suspect that there is a general inclination towards sentencing reform which is swamped by tactical security concerns and the money goes towards hiring more officers. Sentencing reform is well regarded but has a weak trade-off ratio.

Finally, issues are almost always solely described in terms of their upside potential and rarely in terms of their associated probabilities of success, cost, and risks of unintended consequences. I may regard criminal sentencing reform as morally compelling and communicate that to a polling firm so that it appears relatively high in priority. At the same time, the history of such efforts has been rife with consequential failures. I may think it is important and morally compelling but have little confidence that it will be done well, or maybe even can be done well.

The polling firm sees my regard for its importance and will therefore conclude that the candidate that has a well crafted policy on criminal sentencing reform would appeal to me. In reality, because of my skepticism, even though I regard it as important, it might be far at the margin as an issue in terms of influencing my selection of a candidate.

The upshot is that, out of context and not considering trade-offs, costs, risks, etc. any issue that is polled is likely to be erratic and disablingly sensitive to context and wording. Essentially, the polls are meaningless.

For a politician, you can have commanding support among the electorate on well supported issues that are low on the ordinal ranking and which have low trade-off propositions. You can be firmly for gay marriage, trans bathrooms, Dreamer legislation, common sense gun control, environmental protection, etc. But when those are put up against the economy, security, and health/risk, not only are they low in prioritization but they have weak trade-offs. No one is going to invest in trans bathrooms if given the choice of more police (as an example).

This then becomes a particularized political example of Simpson's Paradox. Your polling can show you as dominant among a large number of low ordinal issues, but if you are not also strong on the big three (prosperity, security, and health/risk) then you are going to lose because even though large in number, the lower ranking ordinal issues have fatally weak trade-off propositions.