Past elections suggest Labour’s prospects are better than most commentators think
Commentary on the difficulties facing the Labour Party – leading in nearly every poll for over a year – sometimes puts me in mind of Bill Shankly, the Liverpool manager of the ‘60s and early ‘70s. Responding to a journalist’s question about a supposed crisis at the club, the great man said, “Ay, here we are with problems at the top of the league”.
Likewise, despite Labour’s consistent polling lead, the impression from the comment pieces, blogs and tweets is of a party struggling to find supporters. The logic is that, firstly, Labour’s lead is smaller than might be expected against a government undertaking such spending cuts, and, secondly, that the government is likely to recover and overhaul that lead between now and the next election.
The first of those points may be subjective, but the second can be tested against what’s happened in the past. Mark Pack has compiled the voting intent results from all opinion polls published since 1945, which allows us to do just that.
November 2011 marked 18 months since the last general election. At that point, Labour were on average 4.6pts ahead of the Tories, a score that has since slightly reduced due to Cameron’s veto bounce.
Comparing this score with opposition parties’ poll scores relative to governments’ 18 months after past general elections allows us to benchmark how Labour are currently doing. It also allows us to estimate Labour’s result in the next general election.
A regression analysis of historical polls 18 months after an election, relative to scores in the next general election can provide a model to analyse where we are now (see below for the detailed methodology). The analysis suggests that performance at this point is in fact a pretty good prediction of performance at the next election: it predicts nearly 60% of the result of the next election, with a very high level of confidence (p=0.002).
If a party is leading at this point, it tends to be leading at the election. However, the gap between the government and the main opposition partly typically halves over the remainder of the parliament.
Therefore, given the current position, the historical data suggest Labour should expect to win the next election by a small margin, of a little over two percent.
However, these polls weren’t conducted equally far before the forthcoming election. The Feb ’74 election, for example, was 44 months after the previous election, while the 2010 election was 60 months after the previous one.
So, as well as comparing election results with polls 18 months after the previous election, we can compare the results with polls exactly two years before that election date.
The chart below shows that oppositions generally (in all but three occasions) have polling leads over governments at this point in the parliament.
Governments often overhaul these leads (in about half the examples) to go on to win the subsequent elections. Two years before an election, the smallest lead that an opposition has had when it’s gone on to win the election, is 6.2 points, before the 1964 election.
If we build another model to predict election results with these data, we get a quite different picture. Although we are now using polls that are closer to the next election, they’re actually less able to predict the result than polls conducted 18 months after the previous election. About 40% of the next election result is predicted by the polls conducted two years before the election, again with a pretty high confidence (p=0.02).
It’s worth noting the oddness of this: polls further out from an election seem to be better at predicting the result than those a bit closer to the election. One possible explanation may be that in the middle of parliaments, governments become relatively unpopular to a fairly consistent degree, so some supporters switch to the opposition. Then as elections approach, voters become more discerning and return to their previous opinions about each party, but the extent to which this happens isn’t predictable.
Regardless, what the model suggests is that two years before an election, an opposition usually needs to be ahead by at least 12 points (the ’64 result is an outlier, reminding us this model isn’t as powerful as the previous one). In the last two years before the election, the gap between the parties has fallen by a factor of about three.
What does this mean for Labour?
The current polling, with Labour slightly ahead of the Tories, is good news for the party. At this point in a parliament, we can predict that it should lead to a small lead at the election. Given the way the electoral system currently favours Labour (subject to boundary changes) this would mean a workable Labour majority at the election.
However, historical trends also suggest that Labour need to extend their lead significantly over the next year or two. The next election will most likely be in May 2015. So by May 2013, Labour should have a double digit lead over the Tories if it is to be confident of winning the election, although the model that predicts this is less reliable than the one looking at Labour’s current position.
So looking at current standings, and ignoring everything except polling scores, Labour’s position seems stronger than much of the current gloomy predictions suggest.
The data are drawn entirely from Mark Pack’s collection of data, available on his website.
Polling data were taken as the average of a minimum of three polls. Where three of more polls were available in the relevant month (18 months after the previous election or two years before the next one), an average of all polls in that month was taken. Where fewer than three polls were available, the closest three polls to the required date were selected.
Given increases in the quantity of polls and advances in polling accuracy, more recent poll data are likely to be more accurate. For example, in ’92 the final polls before the election underestimated Tory support by about 4.5pts. However, this should not have introduced any systematic bias in earlier polls in terms of opposition vs government.
For the sake of clarity, the score comparison is only that between Labour and the Tories, leaving out other parties. In most cases this doesn’t appear to change the results, as changes in their scores seem to affect the main parties fairly equally. The exception to this is in the change between November ’80 polls and the ’83 election. In this instance, the emergence of the new SDP-Liberal Alliance produced a collapse in Labour’s vote in favour of the third party. Therefore, this election is excluded from the regression model, greatly enhancing its predictive power.
This is a very important exclusion for the analysis. Without it, the model based on 18-month data has a lower confidence (p=0.09), and the correlation line shifts to the right – spelling bad news for Labour. The two-year data are also affected, though less dramatically. However, there are good grounds for taking the ’83 data, given the unique circumstances of a new party growing rapidly between polling and the election, and doing so largely at the expense of just one other party.
Three other elections were excluded: October ’51, March ’66, and October ’74. Each of these was sufficiently soon after the previous election for there to be not enough polling to conduct the same analysis as for the other elections.
Additional analysis could include data for exactly 3.5 years before the next election, as a way of removing the effect of varying parliament lengths from the 18-month-after-previous-election analysis. This would be interesting, but my preference for now is to fix the ‘comparison with November 2011’ to 18 months into the election to control for the honeymoon effect of governments generally doing better at the beginning of parliaments.
Two other issues remain. Firstly, the regressions draw on just 13 data points, which isn’t a great deal. Unfortunately we just have to live with this until we’ve had a few more elections to improve the model.
’91 ’97 and ’01 look like outliers. If we decided they were exceptional and excluded them, we would see little change to the 18-month model, but a great deal of weakening of the two-year model and also a shift to the right in it. This strengthens my concern that polls two years before an election aren’t great predictors of results: where the polls are relatively close, the predictive value is pretty weak.
I’m hugely grateful to Dr Josh Pasek at the University of Michigan and Rob Vance for their help with the analysis. Any mistakes are down to my inability to act on their patient explanations.