Pro-Brexit survey is a long list of loaded questions

Posted in Bad polling, Europe on May 23rd, 2018 by Leo – 2 Comments

A poll on the House of Lords and Brexit, doing the rounds today, apparently shows the upper house is seen as out of tune with the public, would be wrong to try to stop Brexit and so on.

A glance at ComRes’s data tables is enough to throw up doubts about the results (the tables were published promptly after the Mail ran the story, so credit on that).

The fundamental problem is that the questions were nearly all one-sided agree/disagree questions, with each one loaded against the Lords and Remainers. A couple of examples:

  • It would be wrong for the House of Lords to try and thwart Brexit [“thwart”!]
  • It is wrong that the House of Lords has already voted against the government on Brexit 14 times
  • There are currently 780 members of the Lords compared to 650 MPs in the Commons. This is too many

If you really want to measure public opinion you ask a question that presents both sides of an argument equally, then allow respondents to choose which they are closer to. Or if you really have to ask agree/disagree questions, the collection of the questions should be balanced so you’re not pushing a particular argument and you can compare the skewed questions against each other.

A good guide of a fair poll is that you shouldn’t be able to guess the view of the organisation commissioning the poll from the questions. This clearly fails that.

The poll was done for a new pro-Brexit campaign called “We, the People”. Their website gives few clues about who they are, other than that the Fitzrovia-based outfit is a “grassroots campaigning group” that wants to “remind the liberal metropolitan elite of the ‘other Britain'”.

After the barrage of anti-Lords and pro-Brexit messages, respondents are given the opportunity to describe the Lords in terms like “out of tune with the will of the British people”  and “an outdated throwback”. They do unsurprisingly well.

The poll hasn’t broken any rules, but surveys with such skewed questions hardly help rebuild trust in the industry.

Want to stop Brexit? This is the question to watch.

Posted in Europe, Politics, Polling Matters on November 26th, 2017 by Leo – 1 Comment

Was Brexit the right decision? (image: getty)

On Polling Matters last week I mentioned something about Brexit polling that’s been on my mind for a while, but which I haven’t written anywhere. It’s this:

Polling questions on a second referendum get quite a bit of attention. They find there’s not much desire for one – typically 30-35% support the idea. The same applies for blunter questions on stopping Brexit, which find even less support.

This is often used as evidence that Brexit is unstoppable. I think that’s the wrong conclusion.

Relatively few prominent commentators currently say Brexit can be stopped. This is surely a major reason roughly 50% of 2016 Remainers have given up on the idea.

But opinion on this kind of thing can change quickly. Not long before Theresa May called the snap election 55% of Tory voters said there shouldn’t be an early vote. Just after she announced it, 64% of them said it was the right decision. This is a subject where politicians and commentators lead public opinion.

That’s not to say majority support for a second referendum is just a few taps of the keyboard away. It does need to tap into a genuine shift in the public mood – but the question we should be looking at is whether Brexit is seen as the right or wrong decision.

Opinion on that has apparently shifted towards “wrong decision”, but only very slightly. The most recent YouGov poll gives it a 4-point lead – 52% vs 48% when you exclude don’t knows – which isn’t enough to say the public mood has shifted decisively.

If that “right/wrong decision” question shifts further – perhaps to 60% saying it was the wrong decision – there will be much more justification for commentators to argue the public want another say. At that point I’d expect opinion on a second referendum to shift quickly.

That’s why, if you’re interested in knowing whether the public could ever support overturning Brexit, I suggest focusing much more on the “right/wrong decision” question and much less on the ones that actually ask about stopping it.

I talked about this, and the state of the polls since the election, with Keiran and Matt Singh, on Polling Matters:

Polling Matters: should Leave focus on immigration?

Posted in Europe, Politics, Polling Matters on June 5th, 2016 by Leo – 1 Comment

I was on Polling Matters again this week, talking – as usual at the moment – about the referendum. Other than a discussion of whether the polls really have narrowed the main question was whether the Leave campaign should focus on immigration.


Polling Matters: EU referendum & Wales

Posted in Europe, Politics, Polling Matters on May 13th, 2016 by Leo – Comments Off on Polling Matters: EU referendum & Wales

I was on Polling Matters again this week – podcast only – talking about the EU referendum. The online polls are neck-and-neck while the phone polls show Remain comfortably ahead. Which are right? Also a bit of discussion of the TV debates and the Welsh First Minister vote.



Why EU referendum turnout might actually favour Remain

Posted in Europe, Politics on April 22nd, 2016 by Leo – Comments Off on Why EU referendum turnout might actually favour Remain

When pollsters get together and talk about the EU referendum, it doesn’t take long before the conversation gets onto turnout.

The debate’s well explored – 1 in 4 sentences in this BuzzFeed article mentioned turnout – and has mostly concluded that the issue helps Leave. But I think the debate has underestimated something that helps Remain.

YouGov’s Freddie Sayers sets out well here the argument that turnout hurts Leave. As he says, Leave supporters are demographically more like people who vote, while Remain supporters – on average, younger people – look more like people who stay at home.

Polls already take this into account as far as possible. If someone says they’re not certain to vote, pollsters either discount them entirely or weight down their response.

The trouble is, polls before an election usually find that more people say they’re certain to vote than actually turn out. The ones who don’t live up to their word tend to be younger.

So, if this is replicated in the EU referendum, Remain may find that many of their younger supporters don’t actually vote, despite saying they would definitely do so, and so Remain might underperform their polls. Given that online (but not phone) polls currently find the race to be neck-and-neck, that could be crucial.

I don’t dispute this. But there’s another aspect that could be at least as important.

Currently, with two months until the election, around 6370% say they’re certain to vote.

Two months before the Scottish referendum, 7881% said they were certain to vote. In the last polls before the vote, 9495% said they were certain to do so.

Turnout in Scotland was 85%, so slightly more than 10% of those who said they would definitely vote in fact didn’t do so. But more people voted on 18 September 2014 than, two months before, had said they were certain to do so.

If something similar happens with the EU referendum polls, in mid-June we would see something like 75-82% saying they’re going to vote (and turnout would be around 70%).

One reason that could change the balance of the race is that stated turnout of Remain supporters has more scope to increase from where it is now than turnout of Leave supporters does. In ICM and Mori’s latest polls, 67-70% of Remain backers said they were certain to vote, while 74-80% of Leave supporters said the same.

So there are more people who support Remain but don’t currently think they’ll vote than there are who support Leave and don’t plan to vote. If turnout expectation increases, Remain’s support has more room to grow, without having to win over any undecideds or Leave supporters.

What I think this means in real terms is that Leave supporters tend to be more enthusiastic and already say they’re going to vote. Remain supporters are more grudging and haven’t yet decided to vote – but over the next two months a growing proportion of them might think it’s worth the effort.

When people start paying attention

But perhaps the Scottish referendum was so different from this one that we can’t learn much. Is it really likely that turnout expectation will increase for the EU referendum like it did in Scotland? There’s no way to be sure, but I think it’s a reasonable assumption (though I’m not claiming turnout will be quite as high).

read more »

Polling Matters: EU referendum turnout, Trump & spread betting

Posted in Europe, Polling Matters, U.S. on April 13th, 2016 by Leo – Comments Off on Polling Matters: EU referendum turnout, Trump & spread betting

I was on Polling Matters again this week, talking about the significance of turnout in the EU referendum, the state of the US races, and political spread betting. Mike Smithson was in the chair while Keiran’s on holiday.



Why neither side in the EU referendum will ever have momentum

Posted in Europe, Politics on March 3rd, 2016 by Leo – Comments Off on Why neither side in the EU referendum will ever have momentum

One of the most popular concepts in politics is a figment of the imagination.

Momentum is mentioned so often that saying it doesn’t exist seems like claiming gravity doesn’t make things fall. Everyone wants their campaign to have the big mo – and to be seen to have it.

I’m not the first to make this point about momentum. Nate Silver and Mark Pack have shown that it doesn’t exist in US Senate polls and UK Westminster polls, respectively.

The former sums up his findings as: “the direction in which polls have moved is not predictive of the direction in which they will move”. Pretty definitive.

Yet, otherwise-sensible political commentators keep referring to momentum as if it’s a thing.

It certainly feels intuitively right that momentum exists. When a side is on a roll and picking up in the polls, it seems obvious that they should be more likely than not to carry on gaining.

And yet those studies found it’s not real.

But neither looked at referendums. Since they’re a binary choice, referendums seem like the most hospitable environment for momentum – and there have been plenty of references to momentum in the EU campaign so far.

With data from the AV and Scottish votes, I’ve tested whether there’s any evidence that momentum does exist in referendums (using only YouGov data, to avoid any noise from different methodologies).

How would we know momentum if we saw it? Roughly it seems like the idea that, once polls have moved in a particular direction, they should continue to do so (ie they won’t just stay at the same level and certainly they won’t reverse).

But what exactly that means in practice isn’t obvious and I’ll try a few options to see if I can find any evidence for it.

Test 1

Does a movement in a single poll predict what will happen in the next poll?

This is the question Mark Pack answered in the negative. In fact, he found it’s twice as likely that the subsequent poll will be in the opposite direction of the first poll.

My results show the same. Of the 46 pairs of polls in which there was swing in both polls, only 15 swung in the same direction; 31 went in opposite directions.

So: when a referendum poll moves in one direction, there’s a two-to-one chance that the next poll will move in the opposite direction.

Test 2

To be fair, few people would claim to spot momentum on the basis of one poll. So what about those 15 pairs of polls that were in the same direction?

If we look at what happens next, we get the same negative result.  Of the 15 pairs that swung in the same direction, 9 were followed by a poll that swung in the opposite direction compared with 6 in the same direction.

So: after two consecutive referendum polls swing in the same direction, there’s a three-to-two chance that the next poll will move in the opposite direction.

Test 3

But perhaps that was too strict. If there was a general trend in one direction, one measly poll in the opposite direction might not disprove momentum.

Let’s take the pairs of polls that have moved in the same direction – the soonest that someone might reasonably claim to see momentum – and see what happens over the subsequent 3 polls.

This seems a pretty good test of whether momentum exists: it’s what I think people have in mind when they refer to it. Once a side has seen an increase over two polls (and if it had momentum) that side’s position should continue to improve over the next few polls.

We have 14 groups of polls we can test this with (not exactly a great sample size, but the best I have from this set).

Applying the new test to the average of three polls that come after two have moved in the same direction:

7 times, the polls reversed direction (5 within margin of error, 2 outside)

6 times, the polls continued to move in the same direction (all within margin of error)

1 time, there was no change.

Now we’re about evenly split between polls reversing direction and those continuing in the same direction, albeit with a slight lean towards reversing direction.

With this generous, but intuitive, definition of momentum we can say: after two consecutive referendum polls move in the same direction, it’s as likely that the following three polls will reverse that direction as it is that they will continue in the same direction.

This is the best evidence I’ve seen that momentum might exist – and it’s pretty tenuous.

Essentially, if two consecutive referendum polls swing in the same direction, and you use that as evidence that the beneficiary has momentum, you’re as likely to be wrong as you are to be right.

You might be wondering whether three consecutive polls in the same direction are a better predictor. The answer is no. Of the 51 times that could have happened in the YouGov polls, it only showed up three times; after each, the next poll showed a swing in the opposite direction.

What does this mean?

read more »

Polling Matters: South Carolina, the EU vote & Labour

Posted in Europe, Polling Matters, U.S. on February 18th, 2016 by Leo – Comments Off on Polling Matters: South Carolina, the EU vote & Labour

I was back on Polling Matters this week, talking about the US primaries, the EU referendum and Labour’s terrible poll ratings.

Polling Matters: Iowa and the EU referendum

Posted in Europe, Politics, Polling Matters, U.S. on February 5th, 2016 by Leo – Comments Off on Polling Matters: Iowa and the EU referendum

My first time back on Polling Matters for a while, in time for the pilot of a TV format. As Keiran says, faces for radio.


Apologies for the brain fade in my first answer.

Why I’m no longer so confident the UK will vote to stay in the EU

Posted in Europe on September 1st, 2015 by Leo – Comments Off on Why I’m no longer so confident the UK will vote to stay in the EU

While the Labour leadership contest – and its gruesome fallout – will still dominate politics for weeks to come, the EU referendum is on its way back into the news. Until recently, I was confident that, barring a shock political event ahead of the vote, In would win. Now I’m much less sure.

There are two reasons for my doubts about the likelihood of an In victory.

The first is the instability of the In/Out polls.

That may seem a strange description of them. YouGov’s tracking poll shows what looks like a secular trend towards In, rather than instability (see below, up to May; recent polls show similar results). Over four years, Out’s lead over In has collapsed, despite the Eurozone and refugee crises. My hunch has been that this is down to people being confronted with the reality of a possible UK exit, which has forced more to engage seriously with the question for the first time.

Remain In looks a little like a global temperatures chart of last few decades.  Hiatus, [look away for a bit], Hiatus, [look away for a bit]

So if there’s been a steady move towards In, why do I refer to instability? The reason is, we’ve still only had a few months of consistent leads for In. Go back just to December and the polls were tied. Go back a year further and Out had a comfortable lead in YouGov’s data. The longer-running Mori data shows Out has been ahead several times since the late ‘70s. Given these shifts, it seems to me too early to say we’ve seen an irreversible shift towards In (though it’s striking that support for In so far seems to have withstood the crises of the last few years).

It’s hard to dispute that views on the EU could well change at least as far as they have over the last eight months. Most people probably haven’t thought about the referendum much yet. As news and arguments develop, In’s lead may still prove to be assailable.

This brings me to my second reason why I’m not so confident of an In victory: opinion about the benefits and costs of EU membership look bad for the In camp. An Opinium poll this weekend showed that some of the underlying views of the EU point more towards Out than to In (I was involved in writing the poll, through my employers, DHA Communications).

read more »