“Lizardman’s Constant Is 4%”, 2013-04-12 (; backlinks; similar):
I have only done a little bit of social science research, but it was enough to make me hate people. One study I helped with analyzed whether people from different countries had different answers on a certain psychological test. So we put up a website where people answered some questions about themselves (like “what country are you from?”) and then took the psychological test. And so of course people screwed it up in every conceivable way. There were the merely dumb, like the guy who put “male” as his nationality and “American” as his gender. But there were also the actively malicious or at least annoying, like the people (yes, more than one) who wrote in “Martian”.
I think we all probably know someone like this, maybe a couple people like this. I also think most of us don’t know someone who believes reptilian aliens in human form control all the major nations of Earth. Public Policy Polling’s recent poll on conspiracy theories mostly showed up on my Facebook feed as “4% of Americans believe lizardmen are running the Earth” (of note, an additional 7% of Americans are “not sure” whether lizardmen are running the Earth or not.)
Imagine the situation. You’re at home, eating dinner. You get a call from someone who says “Hello, this is Public Policy Polling. Would you mind answering some questions for us?” You say “Sure”. An extremely dignified sounding voice says—and this is the exact wording of the question—“Do you believe that shape-shifting reptilian people control our world by taking on human form and gaining political power to manipulate our society, or not?” Then it urges you to press 1 if yes, press 2 if no, press 3 if not sure. So first we get the people who think “Wait, was 1 the one for if I did believe in lizardmen, or if I didn’t? I’ll just press 1 and move on to the next question.” Then we get the people who are like “I never heard it before, but if this nice pollster thinks it’s true, I might as well go along with them.” Then we get the people who are all “F#&k you, polling company, I don’t want people calling me when I’m at dinner. You screw with me, I tell you what I’m going to do. I’m going to tell you I believe lizard people are running the planet.” And then we get the people who put “Martian” as their nationality in psychology experiments. Because some men just want to watch the world burn.
Do these three groups total 4% of the US population? Seems plausible.
…But sometimes it’s not some abstruse subtle bias. Sometimes it’s not a good-natured joke. Sometimes people might just be actively working to corrupt your data.
Another link I’ve seen on my Facebook wall a few times is this one: “Are Climate Change Sceptics More Likely To Be Conspiracy Theorists?” It’s based on a paper by Stephen Lewandowsky et al called “NASA Faked The Moon Landing, Therefore Climate Science Is A Hoax—An Analysis Of The Motivated Rejection Of Science”. The paper’s thesis was that climate change skeptics are motivated by conspiracy ideation—a belief that there are large groups of sinister people out to deceive them. This seems sort of reasonable on the face of it—being a climate change skeptic requires going against the belief of the entire scientific establishment. My guess is that there probably is an important link here waiting to be discovered…But a bunch of global warming skeptics started re-analyzing the data and coming up with their own interpretations…More interestingly, they found that pretty much all of the link between global warming skepticism and stupidity was a couple of people (there were so few skeptics, and so few conspiracy believers, that these couple of people made up a pretty big proportion of them, and way more than enough to get a “significant” difference with the global warming believers). Further, most of these couple of people had given the maximally skeptical answer to every single question about global warming, and the maximally credulous answer to every single question about conspiracies.
The danger here now seems obvious. Global warming believer blogs publish a link to this study, saying gleefully that it’s going to prove that global warming skeptics are idiots who also think NASA faked the moon landing and the world is run by lizardmen or whatever. Some global warming believers decide to help this process along by pretending to be super-strong global warming skeptics and filling in the stupidest answers they can to every question. The few real global warming skeptics who take the survey aren’t enough signal to completely drown out this noise. Therefore, they do the statistics and triumphantly announce that global warming skepticism is linked to stupid beliefs.
…The lesson from all three of the cases in this post seems clear. When we’re talking about very unpopular beliefs, polls can only give a weak signal. Any possible source of noise—jokesters, cognitive biases, or deliberate misbehavior—can easily overwhelm the signal. Therefore, polls that rely on detecting very weak signals should be taken with a grain of salt.
View External Link: