One of the most popular concepts in politics is a figment of the imagination.
Momentum is mentioned so often that saying it doesn’t exist seems like claiming gravity doesn’t make things fall. Everyone wants their campaign to have the big mo – and to be seen to have it.
I’m not the first to make this point about momentum. Nate Silver and Mark Pack have shown that it doesn’t exist in US Senate polls and UK Westminster polls, respectively.
The former sums up his findings as: “the direction in which polls have moved is not predictive of the direction in which they will move”. Pretty definitive.
Yet, otherwise-sensible political commentators keep referring to momentum as if it’s a thing.
It certainly feels intuitively right that momentum exists. When a side is on a roll and picking up in the polls, it seems obvious that they should be more likely than not to carry on gaining.
And yet those studies found it’s not real.
But neither looked at referendums. Since they’re a binary choice, referendums seem like the most hospitable environment for momentum – and there have been plenty of references to momentum in the EU campaign so far.
With data from the AV and Scottish votes, I’ve tested whether there’s any evidence that momentum does exist in referendums (using only YouGov data, to avoid any noise from different methodologies).
How would we know momentum if we saw it? Roughly it seems like the idea that, once polls have moved in a particular direction, they should continue to do so (ie they won’t just stay at the same level and certainly they won’t reverse).
But what exactly that means in practice isn’t obvious and I’ll try a few options to see if I can find any evidence for it.
Test 1
Does a movement in a single poll predict what will happen in the next poll?
This is the question Mark Pack answered in the negative. In fact, he found it’s twice as likely that the subsequent poll will be in the opposite direction of the first poll.
My results show the same. Of the 46 pairs of polls in which there was swing in both polls, only 15 swung in the same direction; 31 went in opposite directions.
So: when a referendum poll moves in one direction, there’s a two-to-one chance that the next poll will move in the opposite direction.
Test 2
To be fair, few people would claim to spot momentum on the basis of one poll. So what about those 15 pairs of polls that were in the same direction?
If we look at what happens next, we get the same negative result. Of the 15 pairs that swung in the same direction, 9 were followed by a poll that swung in the opposite direction compared with 6 in the same direction.
So: after two consecutive referendum polls swing in the same direction, there’s a three-to-two chance that the next poll will move in the opposite direction.
Test 3
But perhaps that was too strict. If there was a general trend in one direction, one measly poll in the opposite direction might not disprove momentum.
Let’s take the pairs of polls that have moved in the same direction – the soonest that someone might reasonably claim to see momentum – and see what happens over the subsequent 3 polls.
This seems a pretty good test of whether momentum exists: it’s what I think people have in mind when they refer to it. Once a side has seen an increase over two polls (and if it had momentum) that side’s position should continue to improve over the next few polls.
We have 14 groups of polls we can test this with (not exactly a great sample size, but the best I have from this set).
Applying the new test to the average of three polls that come after two have moved in the same direction:
7 times, the polls reversed direction (5 within margin of error, 2 outside)
6 times, the polls continued to move in the same direction (all within margin of error)
1 time, there was no change.
Now we’re about evenly split between polls reversing direction and those continuing in the same direction, albeit with a slight lean towards reversing direction.
With this generous, but intuitive, definition of momentum we can say: after two consecutive referendum polls move in the same direction, it’s as likely that the following three polls will reverse that direction as it is that they will continue in the same direction.
This is the best evidence I’ve seen that momentum might exist – and it’s pretty tenuous.
Essentially, if two consecutive referendum polls swing in the same direction, and you use that as evidence that the beneficiary has momentum, you’re as likely to be wrong as you are to be right.
You might be wondering whether three consecutive polls in the same direction are a better predictor. The answer is no. Of the 51 times that could have happened in the YouGov polls, it only showed up three times; after each, the next poll showed a swing in the opposite direction.
What does this mean?
read more »