Archive for February, 2013

If I coated it in honey, would you be more likely to eat a live cockroach?

Posted in Bad polling on February 6th, 2013 by Leo – Comments Off on If I coated it in honey, would you be more likely to eat a live cockroach?

Think of something you would never do. For the sake of an example, let’s call it eating a live cockroach. Now suppose I tell you I’ve done something to make it slightly less unappealing, perhaps coated it in honey. Would you be more or less likely to eat that cockroach?

This is a question type repeatedly used by pollsters. I’m going to show why they should stop using it, and why its results should generally be ignored.

Here’s an example. Last week, Survation did a poll for the Mirror on illegal drugs.* In that poll was a question on whether or not respondents had ever taken drugs, and another on what they would do if drugs were sold guaranteed to be free of contaminants.

The results demonstrate why questions of the format “what would you do if x happened” shouldn’t be taken at face value:

So 32% of people who’ve never taken drugs say they would be less likely to take drugs that were guaranteed not to be contaminated.

Read that again. It suggests that a third of people who’ve never taken drugs are currently a bit tempted to give them a try by the thought that the drugs they aren’t buying might contain a bit of rat poison.

Obviously this is complete rubbish. Almost all of that 32% are making a different point: they would never take drugs, and nothing the pollster can say would make them change their mind.

Logic might dictate that they should be more likely to take drugs if they were a bit safer. But they so strongly don’t want to take drugs, they will give the most negative answer they can regardless of what inducements they’re offered (this is similar to the Twitter response to Tom Chivers’ suggestion that liberals should be more likely to vote Tory because of the gay marriage legislation).

That was an easy one to spot, but sometimes the silliness of the result isn’t so obvious.

read more »