Teen lizardmen

From 'Mischievous Responders' Confound Research On Teens by Anya Kamenetz.

But a new research paper points out one huge potential flaw in all this research: kids who skew the results by making stuff up for a giggle. "Mischievous Responders," they're called.

They may say they're 7 feet tall, or weigh 400 pounds, or have three children. They may exaggerate their sexual experiences, or lie about their supposed criminal activities. In other words, kids will be kids, especially when you ask them about sensitive issues.

[snip]

Joseph P. Robinson-Cimpian, the author of the new paper, is an assistant professor of educational psychology at the University of Illinois, Urbana-Champaign. He says he first noticed this phenomenon, and coined the term "mischievous responder," in a paper he co-authored in 2011 with Dorothy L. Espelage.

[snip]

And they did find a correlation. For example, 41 percent of the students who claimed they were transgender also claimed to be extremely tall or short, and the same percentage also claimed they were in a gang.

This is important because researchers are often the most interested in minority groups, and so the undetected presence of a small number of jokesters can seriously mess up results.

In a 2003 study, 19 percent of teens who claimed to be adopted actually weren't, according to follow-up interviews with their parents. When you excluded these kids (who also gave extreme responses on other items), the study no longer found a significant difference between adopted children and those who weren't on behaviors like drug use, drinking and skipping school. The paper had to be retracted. In yet another survey, fully 99 percent of 253 students who claimed to use an artificial limb were just kidding.

It seems, at least with teens, that the false response rate is on the order of 20-40%. That's not too surprising but it completely negates, or at least calls into question, any research depending on survey answers.

What is the comparable false response rate for adults? No idea. Scott Alexander addressed the topic in his post, Noisy Poll Results and Reptilian Muslim Climatologists From Mars in which he warned about Phantom Lizardmen.

I also think most of us don’t know someone who believes reptilian aliens in human form control all the major nations of Earth.

Public Policy Polling’s recent poll on conspiracy theories mostly showed up on my Facebook feed as “Four percent of Americans believe lizardmen are running the Earth”.

(of note, an additional 7% of Americans are “not sure” whether lizardmen are running the Earth or not.)

Imagine the situation. You’re at home, eating dinner. You get a call from someone who says “Hello, this is Public Policy Polling. Would you mind answering some questions for us?” You say “Sure”. An extremely dignified sounding voice says – and this is the exact wording of the question – “Do you believe that shape-shifting reptilian people control our world by taking on human form and gaining political power to manipulate our society, or not?” Then it urges you to press 1 if yes, press 2 if no, press 3 if not sure.

So first we get the people who think “Wait, was 1 the one for if I did believe in lizardmen, or if I didn’t? I’ll just press 1 and move on to the next question.”

Then we get the people who are like “I never heard it before, but if this nice pollster thinks it’s true, I might as well go along with them.”

Then we get the people who are all “F#&k you, polling company, I don’t want people calling me when I’m at dinner. You screw with me, I tell you what I’m going to do. I’m going to tell you I believe lizard people are running the planet.”

And then we get the people who put “Martian” as their nationality in psychology experiments. Because some men just want to watch the world burn.

Do these three groups total 4% of the US population? Seems plausible.

Based on that one poll, one might speculate that 4-11% (including the not sure's) are mischievous responders, but the truth is we don't know.

Alexander's conclusion is pertinent.

The lesson from all three of the cases in this post seems clear. When we’re talking about very unpopular beliefs, polls can only give a weak signal. Any possible source of noise – jokesters, cognitive biases, or deliberate misbehavior – can easily overwhelm the signal. Therefore, polls that rely on detecting very weak signals should be taken with a grain of salt.

Especially if you are dealing with small sample sizes. The Lizardmen phenomenon is why it is so important to know effect sizes. If the effect size is small, and especially if the sample size is small, then disregard the results unless there are incredibly rigorous tests on the survey.