Why neither side in the EU referendum will ever have momentum

One of the most popular concepts in politics is a figment of the imagination.

Momentum is mentioned so often that saying it doesn’t exist seems like claiming gravity doesn’t make things fall. Everyone wants their campaign to have the big mo – and to be seen to have it.

I’m not the first to make this point about momentum. Nate Silver and Mark Pack have shown that it doesn’t exist in US Senate polls and UK Westminster polls, respectively.

The former sums up his findings as: “the direction in which polls have moved is not predictive of the direction in which they will move”. Pretty definitive.

Yet, otherwise-sensible political commentators keep referring to momentum as if it’s a thing.

It certainly feels intuitively right that momentum exists. When a side is on a roll and picking up in the polls, it seems obvious that they should be more likely than not to carry on gaining.

And yet those studies found it’s not real.

But neither looked at referendums. Since they’re a binary choice, referendums seem like the most hospitable environment for momentum – and there have been plenty of references to momentum in the EU campaign so far.

With data from the AV and Scottish votes, I’ve tested whether there’s any evidence that momentum does exist in referendums (using only YouGov data, to avoid any noise from different methodologies).

How would we know momentum if we saw it? Roughly it seems like the idea that, once polls have moved in a particular direction, they should continue to do so (ie they won’t just stay at the same level and certainly they won’t reverse).

But what exactly that means in practice isn’t obvious and I’ll try a few options to see if I can find any evidence for it.

Test 1

Does a movement in a single poll predict what will happen in the next poll?

This is the question Mark Pack answered in the negative. In fact, he found it’s twice as likely that the subsequent poll will be in the opposite direction of the first poll.

My results show the same. Of the 46 pairs of polls in which there was swing in both polls, only 15 swung in the same direction; 31 went in opposite directions.

So: when a referendum poll moves in one direction, there’s a two-to-one chance that the next poll will move in the opposite direction.

Test 2

To be fair, few people would claim to spot momentum on the basis of one poll. So what about those 15 pairs of polls that were in the same direction?

If we look at what happens next, we get the same negative result.  Of the 15 pairs that swung in the same direction, 9 were followed by a poll that swung in the opposite direction compared with 6 in the same direction.

So: after two consecutive referendum polls swing in the same direction, there’s a three-to-two chance that the next poll will move in the opposite direction.

Test 3

But perhaps that was too strict. If there was a general trend in one direction, one measly poll in the opposite direction might not disprove momentum.

Let’s take the pairs of polls that have moved in the same direction – the soonest that someone might reasonably claim to see momentum – and see what happens over the subsequent 3 polls.

This seems a pretty good test of whether momentum exists: it’s what I think people have in mind when they refer to it. Once a side has seen an increase over two polls (and if it had momentum) that side’s position should continue to improve over the next few polls.

We have 14 groups of polls we can test this with (not exactly a great sample size, but the best I have from this set).

Applying the new test to the average of three polls that come after two have moved in the same direction:

7 times, the polls reversed direction (5 within margin of error, 2 outside)

6 times, the polls continued to move in the same direction (all within margin of error)

1 time, there was no change.

Now we’re about evenly split between polls reversing direction and those continuing in the same direction, albeit with a slight lean towards reversing direction.

With this generous, but intuitive, definition of momentum we can say: after two consecutive referendum polls move in the same direction, it’s as likely that the following three polls will reverse that direction as it is that they will continue in the same direction.

This is the best evidence I’ve seen that momentum might exist – and it’s pretty tenuous.

Essentially, if two consecutive referendum polls swing in the same direction, and you use that as evidence that the beneficiary has momentum, you’re as likely to be wrong as you are to be right.

You might be wondering whether three consecutive polls in the same direction are a better predictor. The answer is no. Of the 51 times that could have happened in the YouGov polls, it only showed up three times; after each, the next poll showed a swing in the opposite direction.

What does this mean?

Clearly changes happen in referendum polls: over six weeks, the AV Yes campaign went from a 7pt lead to an 18pt deficit. In Scotland, No’s lead fell from 34pt to 6pt in just over a year.

So, if not momentum, what happened?

The changes usually happened in bursts. The Scottish No campaign had exactly the same lead in September 2013 as it did in August 2014. A month later it had suffered a 12pt swing, in the space of just three polls.

About the same happened with the AV campaign, where the Yes campaign’s lead fell from 17 to 1pt over a single poll. Across the series of referendum polls, leads weren’t whittled away over several polls, they were slashed in just one or two.

Of the 51 pairs of polls, more than two thirds, 35, produced combined swings of less than 4pts – that is, most of the time, not much happened.

But in 6 of the pairs, the polls swung by more than 6pts. These were the moments when the campaigns changed. Every one of these pairs was followed by a swing back the other way (ie they weren’t the start of momentum) but those gains were not completely unwound.

So: referendum polls are characterised by punctuated equilibrium, not momentum.

If you’re looking for change in referendum polls, there’s not much reason to expect either side to steadily accrue gains. Instead, be ready for the polls to change relatively quickly: over the space of a couple of weeks. But when that change happens, don’t expect it to carry on. And ignore anyone who says their side has momentum.

 

In short, Josh was wrong.

Share

Comments are closed.