If I coated it in honey, would you be more likely to eat a live cockroach?

Think of something you would never do. For the sake of an example, let’s call it eating a live cockroach. Now suppose I tell you I’ve done something to make it slightly less unappealing, perhaps coated it in honey. Would you be more or less likely to eat that cockroach?

This is a question type repeatedly used by pollsters. I’m going to show why they should stop using it, and why its results should generally be ignored.

Here’s an example. Last week, Survation did a poll for the Mirror on illegal drugs.* In that poll was a question on whether or not respondents had ever taken drugs, and another on what they would do if drugs were sold guaranteed to be free of contaminants.

The results demonstrate why questions of the format “what would you do if x happened” shouldn’t be taken at face value:

So 32% of people who’ve never taken drugs say they would be less likely to take drugs that were guaranteed not to be contaminated.

Read that again. It suggests that a third of people who’ve never taken drugs are currently a bit tempted to give them a try by the thought that the drugs they aren’t buying might contain a bit of rat poison.

Obviously this is complete rubbish. Almost all of that 32% are making a different point: they would never take drugs, and nothing the pollster can say would make them change their mind.

Logic might dictate that they should be more likely to take drugs if they were a bit safer. But they so strongly don’t want to take drugs, they will give the most negative answer they can regardless of what inducements they’re offered (this is similar to the Twitter response to Tom Chivers’ suggestion that liberals should be more likely to vote Tory because of the gay marriage legislation).

That was an easy one to spot, but sometimes the silliness of the result isn’t so obvious.

Take this Populus poll for the Times from last month, which asked how Cameron’s promise of an EU referendum would change likelihood to vote Tory. Apparently, 17% of UKIP voters are now less likely to vote Tory because of the referendum, compared with only 8% who are more likely.

This may be the same problem as the drugs poll: people who wouldn’t vote Tory are using the question to restate how extremely anti-Tory they are. Or perhaps there really was something about Cameron’s EU pledge that put off UKIP voters.

That’s the problem with this question type. Knowing that the results can be so misleading, it’s impossible to take them at face value, even if they could in principle make sense.**

So next time you see a question based on “what would you do if x happened”, think about that live cockroach. How much would it take for you to say anything except, “urrgghh! NO WAY”? A bit of honey probably isn’t going to stifle your disgust.


* Until recently, I worked for a drug policy organisation. Anything I say here is entirely my own opinion etc.

** By the way, this is about when we’re trying to understand how many people would change their behaviour if something happened, eg “how many people would vote Tory if they held an EU referendum?”. The question type still works for ranking different options, eg “out of 10 possible policy pledges, which is the one that could win the Tories most support from Ukip?”. You wouldn’t get reliable numbers on how many people they’d win, but you’d be able to see which pledges are the most and least effective.


Comments are closed.