But in election after election, ballot after ballot, and referendum after referendum, here and in Europe and elsewhere, voters are providing decisions that were not foreshadowed in the polls. It is not just Trump or Brexit - two major upsets within this past year. The polls dramatically underestimated the Republican wave of 2014. In Britain despite close polls, in 2014 the electorate decisively rejected Scottish independence. They also missed decisive victories for Tories in Britain and Likud in Israel in 2015.
With regard to the 2016 US election, Kornacki points out:
What complicates them, though, is how Trump became president in the first place.It is not all gloom and doom. Nate Silver and his Fivethirtyeight team have an excellent track record for forecasts, even though they missed Trump's election.
Recall some of the dire polling he faced as a candidate. More than 60 percent of voters didn’t think he was qualified to be president; not even 20 percent thought he had the temperament and personality to serve; more than half of Republicans said they weren’t satisfied with him as their nominee. On Election Day, 60 percent of the electorate said it didn’t like him.
By any historical standard, these were also politically catastrophic numbers, and yet, well, Trump is in the White House. In 2016, the numbers didn’t mean quite what we thought they did.
Looking at the wins and losses, it would be easy to assume that there is some sort of media (sponsors of many of the polls) bias occurring. The wins are almost right of center politicians and issues against the left of center assumptions of most of the media. But I doubt that media bias is a first order explanation.
And partly, I am not sure that polling error is all that dramatically greater than in the past. Up, yes, but by factors worse, not orders of magnitude worse.
Dewey vs. Truman in 1948 remains a archetypal example of polling error.
There was the surprise loss of the Labour Party in Britain in 1970 and then the surprise loss of the Tories in 1974. There is a long history of polling missing the electoral patterns.
Technology and regulation has something to do with this. Pollsters try and get a representative sample of the population by income, party affiliation, region, race, sex, likelihood of voting, etc. That makes perfect sense. They try and reach some pre-specified number of people, usually in the neighborhood of 1-3,000 in order to get their opinions.
The challenge is that communication technology is changing. In the past, the most standard means of contacting people was personal or robocalling home landlines. Now, as people become more and more exclusively accustomed to only using cell phones, the challenge has increased. By regulation, companies are not allowed to robocall cell phone numbers and it is very expensive to do personal dialing.
The upshot is that it is harder to reach people and given that, it is also harder to establish representative sample sizes, i.e. pollers have to mathematically extrapolate. The Week had a pretty good description of these issues a year or two ago.
I suspect that there is yet more going on but haven't seen much good research on it so my speculation is absent any data.
Deliberate misrepresentation by those being polled. This is discussed by Scott Alexander in Noisy Poll Results and Reptilian Muslim Climatologists from Mars.
I have only done a little bit of social science research, but it was enough to make me hate people. One study I helped with analyzed whether people from different countries had different answers on a certain psychological test. So we put up a website where people answered some questions about themselves (like “what country are you from?”) and then took the psychological test.Non-representativeness of self-selected poll takers. This has always been an issue but I wonder if it is not becoming an even larger issue. Are people who choose to take polls representative of all Americans, and more specifically, are they equally representative of those who do not take polls. There are all sorts of reasons to speculate that poll-takers are unrepresentative of non-poll takers. They might systemically be less likely to work, might overrepresent the personality trait of extroversion, might be older, might be more home bound, etc., etc. Pollsters are aware of this and try and compensate by making adjustments to samples but I suspect that this unrepresentativeness is more of an issue than is being acknowledged.
And so of course people screwed it up in every conceivable way. There were the merely dumb, like the guy who put “male” as his nationality and “American” as his gender. But there were also the actively malicious or at least annoying, like the people (yes, more than one) who wrote in “Martian”.
I think we all probably know someone like this, maybe a couple people like this.
I also think most of us don’t know someone who believes reptilian aliens in human form control all the major nations of Earth.
Public Policy Polling’s recent poll on conspiracy theories mostly showed up on my Facebook feed as “Four percent of Americans believe lizardmen are running the Earth”.
(of note, an additional 7% of Americans are “not sure” whether lizardmen are running the Earth or not.)
Imagine the situation. You’re at home, eating dinner. You get a call from someone who says “Hello, this is Public Policy Polling. Would you mind answering some questions for us?” You say “Sure”. An extremely dignified sounding voice says – and this is the exact wording of the question – “Do you believe that shape-shifting reptilian people control our world by taking on human form and gaining political power to manipulate our society, or not?” Then it urges you to press 1 if yes, press 2 if no, press 3 if not sure.
So first we get the people who think “Wait, was 1 the one for if I did believe in lizardmen, or if I didn’t? I’ll just press 1 and move on to the next question.”
Then we get the people who are like “I never heard it before, but if this nice pollster thinks it’s true, I might as well go along with them.”
Then we get the people who are all “F#&k you, polling company, I don’t want people calling me when I’m at dinner. You screw with me, I tell you what I’m going to do. I’m going to tell you I believe lizard people are running the planet.”
And then we get the people who put “Martian” as their nationality in psychology experiments. Because some men just want to watch the world burn.
Do these three groups total 4% of the US population? Seems plausible.
Statistical subsidiarity. I have not ever seen this addressed so it may be a non-issue but I have a strong suspicion that it might be material. Politicians and Media in particular are not especially interested in the electorate at large. They are interested in slicing and dicing the voters and the polls. We saw a lot of this in the aftermath of the 2016 election when the losing side was very eager to identify which segment of their coalition let them down. Was it males or females, professionals or blue collar, unionized or service, high-school educated or college educated? Pretty much yes, all of them.
But the real issue is to what level of statistical representativeness do you want to go? If you are interested in Sex (1), Education attainment (2), Geographical region (3), Race (4), Income (5), Union status (6), Ethnicity (7), and Marital Status (8), then you have a cumulative error potential. And those categories do not exhaust all categories people are interest in.
Let's use an example. As an extremely crude approximation, if you have a very large population and your sampling is truly random, you need a sample six of roughly 300 people to get an accurate estimation for the entire population. But that is only 1 level (Did voters vote for Clinton?).
If you are interested in geography and want to be able to answer "Did voters in the Midwest vote for Clinton? Then you need a sample size of 1,200 (for 4 geographical regions). What if you want to know "Did male voters in the Midwest vote for Clinton? Now you need 2,400 (for two sexes) in order to get 300 midwestern males for an accurate sampling. If you want to know how male Hispanic voters in the Midwest voted and you have 6 racial categories (White, Black, Asian, Native American, Hispanic, Mixed), then you need a sample size of 14,400. We are only 4 levels in and we are already at nearly 15,000 people we need to poll for statistical subsidiarity accuracy. From The Week article we know that pollsters have to contact 100 people in order to get 8 responses. So, at four levels of subsidiarity, they will have to contact 180,000 people to take their poll.
Of course all that is impossible. They contact 300 people. They only have 21 African-American respondents and they need 42 so that the sample matches the actual representation in the population (14%). They double the weight of those African-American respondents in order to achieve representation. But you can see the problem. You need 300 if you want to know African-American opinion in particular, not 21. Your sample size is way too small to be representative. When you are dealing with very small numbers, the problem is compounded. Simple random variation might mean 15 of your 21 African-Americans are in New York, further exacerbating the unrepresentativeness.
I suspect that statistical subsidiarity might be a much larger source of polling error than is being discussed.
Malicious poll takers. These are the Lizardmen Martians Alexander discusses above.
Absence of poll trade-off sensitivity. Another area where pollsters are quite sophisticated because they know it is so determinative of outcomes is the wording and framing of questions. But there's more to it than just framing; there is conceptualizing trade-offs as well. And I don't think pollsters focus on that very much or very well.
Trade-off sensitivity is, I believe, at the hear of the Trump and Brexit issues. A crude summary might be an example of what I think might possibly have happened in the Clinton-Trump election. I suspect that Clinton had articulated policies of benefit to most people but that most people also did not trust her. In contrast, Trump had no real policies, more a portfolio of sentiments. You knew what goals Clinton would go after and how she said she would accomplish them but you did not trust that she would or could actually achieve the outcome. You might have guessed that she would be effective at serving her own interests but not committed to the public interest. With Trump, on the other hand, you only had a general sense of what he was after (Make America Great Again) and no real sense of how he would bring that about. However, you might have trusted that he would actually try and achieve something for the public good, regardless of the probabilities of success.
Under this scenario, you have a trade-off between specificity but untrustworthiness and generalities but confidence. You rarely ever get polling questions that get to the trade-off particularities. With the stated scenario, Trump's popularity (which has always been low and interpreted by politicians and the press as political death) is irrelevant as to whether someone votes for him. I suspect that voters are very often in the position of 1) choosing the lesser of two evils, and 2) not being asked what is actually important to them.
Establishment Democrats and Republicans tend to run candidates who are pursuing many of the same goals and only differ mildly in terms of the rank order of the goals and sometimes in terms of the means of achieving those goals. Everyone wants a stronger economy, better policing, better education, stronger defense, cleaner air, better justice, fewer prisoners, etc. But they prioritize those goals differently and they pursue them in different ways, and usually in ways that don't work. Establishment politicians just try and shout louder about there priorities.
That's not always the choice of the voters. Yes, they want those goals but they don't want continuing failure to achieve those goals. Their primary objective becomes to shake things up and clear the deck of the establishment. Voters are making a trade-off between the strategic objective of change and the tactical objective of continuity. Pollsters focus on the common tactical goals and not the changed goals.
This model of trade-off sensitivity is what lies behind Trump's victory, the surprise showing of Sanders, and earlier, the rise of the Tea Party. The electorate was signaling a desire for strategic change and the establishment parties and their pollsters were focusing on the assumed traditional tactical goals.
You put all these together; Non-representativeness of self-selected poll takers, Statistical subsidiarity, Malicious poll takers, and Absence of poll trade-off sensitivity, and it seems very plausible that pollsters would get a lot of forecasts wrong.
If any or all that is true, then there is a different question on the table. Why do politicians and media rely so heavily on polls if they are so unreliable?
For politicians, it is probably useful because polls likely are sufficiently informative to adequately inform Go:No Go decisions. Beyond that high level, polls are likely useless. For media, it is almost certainly simply a commercial necessity. Whether or not the poll is accurate, it will be reflexively read. It helps generate readership and reader engagement.
That's my guess.
Posts
No comments:
Post a Comment