Past elections suggest Labour’s prospects are better than most commentators think

Commentary on the difficulties facing the Labour Party – leading in nearly every poll for over a year – sometimes puts me in mind of Bill Shankly, the Liverpool manager of the ‘60s and early ‘70s. Responding to a journalist’s question about a supposed crisis at the club, the great man said, “Ay, here we are with problems at the top of the league”.

Likewise, despite Labour’s consistent polling lead, the impression from the comment pieces, blogs and tweets is of a party struggling to find supporters. The logic is that, firstly, Labour’s lead is smaller than might be expected against a government undertaking such spending cuts, and, secondly, that the government is likely to recover and overhaul that lead between now and the next election.

The first of those points may be subjective, but the second can be tested against what’s happened in the past. Mark Pack has compiled the voting intent results from all opinion polls published since 1945, which allows us to do just that.

November 2011 marked 18 months since the last general election. At that point, Labour were on average 4.6pts ahead of the Tories, a score that has since slightly reduced due to Cameron’s veto bounce.

Comparing this score with opposition parties’ poll scores relative to governments’ 18 months after past general elections allows us to benchmark how Labour are currently doing. It also allows us to estimate Labour’s result in the next general election.

A regression analysis of historical polls 18 months after an election, relative to scores in the next general election can provide a model to analyse where we are now (see below for the detailed methodology). The analysis suggests that performance at this point is in fact a pretty good prediction of performance at the next election: it predicts nearly 60% of the result of the next election, with a very high level of confidence (p=0.002).

If a party is leading at this point, it tends to be leading at the election. However, the gap between the government and the main opposition partly typically halves over the remainder of the parliament.

Therefore, given the current position, the historical data suggest Labour should expect to win the next election by a small margin, of a little over two percent.

However, these polls weren’t conducted equally far before the forthcoming election. The Feb ’74 election, for example, was 44 months after the previous election, while the 2010 election was 60 months after the previous one.

So, as well as comparing election results with polls 18 months after the previous election, we can compare the results with polls exactly two years before that election date.

The chart below shows that oppositions generally (in all but three occasions) have polling leads over governments at this point in the parliament.

Governments often overhaul these leads (in about half the examples) to go on to win the subsequent elections. Two years before an election, the smallest lead that an opposition has had when it’s gone on to win the election, is 6.2 points, before the 1964 election.

If we build another model to predict election results with these data, we get a quite different picture. Although we are now using polls that are closer to the next election, they’re actually less able to predict the result than polls conducted 18 months after the previous election. About 40% of the next election result is predicted by the polls conducted two years before the election, again with a pretty high confidence (p=0.02).

It’s worth noting the oddness of this: polls further out from an election seem to be better at predicting the result than those a bit closer to the election. One possible explanation may be that in the middle of parliaments, governments become relatively unpopular to a fairly consistent degree, so some supporters switch to the opposition. Then as elections approach, voters become more discerning and return to their previous opinions about each party, but the extent to which this happens isn’t predictable.

Regardless, what the model suggests is that two years before an election, an opposition usually needs to be ahead by at least 12 points (the ’64 result is an outlier, reminding us this model isn’t as powerful as the previous one). In the last two years before the election, the gap between the parties has fallen by a factor of about three.

What does this mean for Labour?

The current polling, with Labour slightly ahead of the Tories, is good news for the party. At this point in a parliament, we can predict that it should lead to a small lead at the election. Given the way the electoral system currently favours Labour (subject to boundary changes) this would mean a workable Labour majority at the election.

However, historical trends also suggest that Labour need to extend their lead significantly over the next year or two. The next election will most likely be in May 2015. So by May 2013, Labour should have a double digit lead over the Tories if it is to be confident of winning the election, although the model that predicts this is less reliable than the one looking at Labour’s current position.

So looking at current standings, and ignoring everything except polling scores, Labour’s position seems stronger than much of the current gloomy predictions suggest.

 

Methodology

The data are drawn entirely from Mark Pack’s collection of data, available on his website.

Polling data were taken as the average of a minimum of three polls. Where three of more polls were available in the relevant month (18 months after the previous election or two years before the next one), an average of all polls in that month was taken. Where fewer than three polls were available, the closest three polls to the required date were selected.

Given increases in the quantity of polls and advances in polling accuracy, more recent poll data are likely to be more accurate. For example, in ’92 the final polls before the election underestimated Tory support by about 4.5pts. However, this should not have introduced any systematic bias in earlier polls in terms of opposition vs government.

For the sake of clarity, the score comparison is only that between Labour and the Tories, leaving out other parties. In most cases this doesn’t appear to change the results, as changes in their scores seem to affect the main parties fairly equally. The exception to this is in the change between November ’80 polls and the ’83 election. In this instance, the emergence of the new SDP-Liberal Alliance produced a collapse in Labour’s vote in favour of the third party. Therefore, this election is excluded from the regression model, greatly enhancing its predictive power.

This is a very important exclusion for the analysis. Without it, the model based on 18-month data has a lower confidence (p=0.09), and the correlation line shifts to the right – spelling bad news for Labour. The two-year data are also affected, though less dramatically. However, there are good grounds for taking the ’83 data, given the unique circumstances of a new party growing rapidly between polling and the election, and doing so largely at the expense of just one other party.

Three other elections were excluded: October ’51, March ’66, and October ’74. Each of these was sufficiently soon after the previous election for there to be not enough polling to conduct the same analysis as for the other elections.

Additional analysis could include data for exactly 3.5 years before the next election, as a way of removing the effect of varying parliament lengths from the 18-month-after-previous-election analysis. This would be interesting, but my preference for now is to fix the ‘comparison with November 2011’ to 18 months into the election to control for the honeymoon effect of governments generally doing better at the beginning of parliaments.

Two other issues remain. Firstly, the regressions draw on just 13 data points, which isn’t a great deal. Unfortunately we just have to live with this until we’ve had a few more elections to improve the model.

Secondly, ’91 ’97 and ’01 look like outliers. If we decided they were exceptional and excluded them, we would see little change to the 18-month model, but a great deal of weakening of the two-year model and also a shift to the right in it. This strengthens my concern that polls two years before an election aren’t great predictors of results: where the polls are relatively close, the predictive value is pretty weak.

I’m hugely grateful to Dr Josh Pasek at the University of Michigan and Rob Vance for their help with the analysis. Any mistakes are down to my inability to act on their patient explanations.

FacebookShare
  1. cim says:

    One possible explanation may be that in the middle of parliaments, governments become relatively unpopular to a fairly consistent degree, so some supporters switch to the opposition.

    The polling evidence suggests that there are extremely few people who will switch between Labour and Conservative. Voters for one of those parties tend to be at best neutral towards the other party, and are more usually strongly hostile.

    Given that the polls are normalised to either exclude or reallocate “don’t know” and “wouldn’t vote”, if Cameron “wins” a debate with Miliband, some weakly-Conservative voters may start polling Conservative rather than “don’t know” and some weakly-Labour voters may start polling “don’t know” rather than Conservative. The apparent net effect of this on the normalised headline polling figure will be a swing from Labour to Conservative – but that’s not because Labour voters have switched to being Conservative voters.

    A similar effect with Lib Dem voters (though probably not so much this time) – Labour voters dissatisfied with Labour might switch to Lib Dem instead, while at the same time some Conservative-leaning Lib Dem voters might be sufficiently impressed with Cameron to poll Conservative.

    Far from an election, there’s a good chance that many weakly-X voters will be saying “don’t know”, but when there’s an election imminent decide that they have to vote for someone, and if they’ve not been significantly impressed by another party, pick the one they voted for last time.

    • Leo says:

      That sounds entirely fair – and as you say, a better description of what I referred to as government supporters switching to the opposition (ie this happening in effect, but through a chain rather than directly).

      Your last point would certainly explain why fluctuations in the middle of parliaments seem often to be resolved back to positions that polls were in earlier in parliaments.

      In the comments to this post on Liberal Conspiracy there’s been some suggestion that Labour have already taken all the Lib Dem votes they can get, but haven’t shown any ability to get Tory votes. I don’t think this causes any problems for the model, as it doesn’t predict that Labour should gain any votes between now and the election – just that they won’t lose many relative to the Tories, and so will retain their lead.

  1. [...] A longer version of this post and the methodology is here. [...]

  2. opinion poll question…

    [...]Noise of the Crowd » Blog Archive » Past elections suggest Labour???s prospects are better than most commentators think – Interesting things about public opinion[...]…

  3. [...] wrote a post last year about old opinion polls and how well they predict future elections. It compared election results [...]

  4. [...] while ago, I ran an analysis of past election results, and how well they predict future election [...]

  5. [...] two previous posts, I looked at how we can use past polls and elections to shed light on what current polls tell [...]