Does the Evening Standard understand its own Boris vs Ken poll?
I try not to write much about polling methodology. I doubt it’s of interest to many people, and besides, Anthony Wells does it much better than I do.
But there’s been some truly awful reporting today of the latest London mayoral poll, and it’s time to look at weighting and so on.
They go on to say that those “interviewed before “liftgate” last Tuesday morning were split 50/50 between the two candidates. But those surveyed afterwards divided 60/40 in favour of Mr Johnson.” ITV also reported it with the same angle.
This all sounds very plausible and interesting, but it’s in fact a bad misrepresentation of the poll.
The issue is, the poll was never designed to show how opinion changed after shoutyBorisgate. Of course it wasn’t: the poll was set up without anyone knowing there would be any event to compare ‘before’ and ‘after’.
If you do know that an event is coming, say a leaders’ debate, you can run two separate polls, with comparable samples (or even, with the same people), and see how the results compare.
But this ComRes poll doesn’t do that. Instead, a little over three quarters of the poll was conducted before the interview, and the remainder after. Nothing looks to have been done to make sure the samples before and after were comparable.
So we’ve got two groups of people. In terms of how they voted in the last general election (nothing to do with Ken and Boris), the first group has 29% Labour voters and 27% Tory voters. The second group has 26% Labour voters and 32% Tory voters. A Labour 2pt lead vs a Tory 5pt lead.
We then ask them how they’d vote in the London election, and are supposed to be surprised when the group with more Tories say they’re more likely to vote for the Tory candidate!
You could in principle do something about this. You could try to weight separately the two groups to make them representative of the country. In practice this might not work very well because one of the groups may not have enough people from some sub-group to be able to weight it well – particularly if the latter part of the sample is made up of hard-to-sample groups, like young people without phone lines, or older people without the internet. There would also sometimes be a problem with margin of error if one group is quite a bit smaller than the other.
But the ComRes poll doesn’t even look to have tried this. Instead they’ve taken a representative sample, and pulled it apart into two groups, regardless of whether those groups are representative of, well, anything.
So, there is pretty much no basis for saying that today’s London poll shows anything about the impact of two men having an argument in a lift, or even how opinion has changed since then.
Even without all this methodological stuff, it should be obvious that a row that almost no-one saw was never going to have swayed opinion as much as the poll apparently shows.
Incidentally, another example of this came up just a couple of weeks ago. Another ComRes poll apparently showed a 17-pt lead for Labour in interviews conducted after the Tories’ cash-for-access scandal.
The normally excellent John Rentoul blogged on it, making the same mistake as today’s Standard. The fact that no subsequent polls picked up the same giant lead suggests that this was indeed purely a consequence of how the sample was divided into ‘before’ and ‘after’.
I do wonder why ComRes persist in putting these columns into their databooks.