Noise of the Crowd

I try not to write much about polling methodology. I doubt it’s of interest to many people, and besides, Anthony Wells does it much better than I do.

But there’s been some truly awful reporting today of the latest London mayoral poll, and it’s time to look at weighting and so on.

According to today’s Evening Standard, their new ComRes poll shows “a dramatic slide in Mr Livingstone’s support after his argument with his Tory rival over tax in a radio station lift”.

They go on to say that those “interviewed before “liftgate” last Tuesday morning were split 50/50 between the two candidates. But those surveyed afterwards divided 60/40 in favour of Mr Johnson.” ITV also reported it with the same angle.

This all sounds very plausible and interesting, but it’s in fact a bad misrepresentation of the poll.

The issue is, the poll was never designed to show how opinion changed after shoutyBorisgate. Of course it wasn’t: the poll was set up without anyone knowing there would be any event to compare ‘before’ and ‘after’.

If you do know that an event is coming, say a leaders’ debate, you can run two separate polls, with comparable samples (or even, with the same people), and see how the results compare.

But this ComRes poll doesn’t do that. Instead, a little over three quarters of the poll was conducted before the interview, and the remainder after. Nothing looks to have been done to make sure the samples before and after were comparable.

So we’ve got two groups of people. In terms of how they voted in the last general election (nothing to do with Ken and Boris), the first group has 29% Labour voters and 27% Tory voters. The second group has 26% Labour voters and 32% Tory voters. A Labour 2pt lead vs a Tory 5pt lead.

We then ask them how they’d vote in the London election, and are supposed to be surprised when the group with more Tories say they’re more likely to vote for the Tory candidate!

read more »

This post was written for the Green Alliance blog,to coincide with the launch of a paper on public opinion and the environment. 

Coverage of public opinion on climate change is never just about reporting numbers. Without appreciating the need for journalists to tell a story, we can never really understand why climate change polls are reported as they are.

Over the last decade, two distinct narratives have been told about what the public think of climate change. Each of these narratives has been so dominant for a time that it has been difficult for alternative views of public opinion to get much attention.

The first, which dominated for most of the noughties, was that climate change was increasingly settled in the public’s minds as a great concern. Polls on climate change were rare for much of the decade, but when they did appear in the media, the coverage tended to acknowledge that the public was worried, although perhaps unsure about the risks or about possible solutions, as in this Observer article.

As a result, there was little prominent dissent from the view that climate change was becoming a more important issue for most people, along with a belief that the world needed to take decisive action.

The rise of scepticism

But by the end of 2009, this prevailing narrative about public views on climate change had given way to a very different account.

It happened quite suddenly, around the time of the COP15 in Copenhagen. Now, the dominant frame was that growing numbers of people doubted the existence of serious man-made climate change, and that there was increased resistance to measures to tackle it.

Opinion polls were important to the development of this new account. For about a year, from late 2009, polls were repeatedly used to show the same narrative: that fewer people were now worried about climate change.

The sheer weight of polls, reported across the media, gave the overwhelming impression of an ongoing change in opinion. But this was misleading: in fact, there appears to have been a one-off fall in concern about climate change, which happened between November’09 and January ’10.

The difficulty in understanding opinion lies in the fact that media outlets want to report their own polls, as an exclusive story. They’re much less interested in repeating polls that another newspaper or broadcaster have commissioned.

So over a period of several months, we saw different polls in outlets from the Daily Mail to the BBC and Guardian, which essentially restated the same phenomenon as if it were a new finding. The result was a powerful new narrative, that concern about climate change was experiencing an ongoing decline.

There are two reasons why it’s useful to see this as a new dominant narrative about public opinion, rather than as straight-forward reporting of opinion.

read more »

A week ago the Occupy London protesters at St Paul’s Cathedral had a strong level of public support.

Even just after the Cathedral closed its doors, ICM found a majority agreeing with the protesters’ demands:

This seems impressively high. Only two in five saying that there is no practical alternative to capitalism is certainly at odds with the dominant view in the media.

But this isn’t the same as finding support for the way the protest is unfolding, and opinion may well have changed since the poll was conducted.

The day after the fieldwork, the Telegraph was the first to run with the claim that nine in ten of the tents are empty overnight. Along with repeated stories of ordinary people’s lives being disrupted, the coverage has also targeted the supposed wealth of some of the protesters to suggest that they’re out of touch or hypocritical.

The thing is, this was all utterly predictable. After one day of most protests there will always be some bad headlines for the protesters, but barring disaster there’ll also be some positive ones. But after several days of protest, the fact that there’s a protest will no longer generate news: only the chaos will be of interest.

After the coverage of the UK Uncut occupation of Fortnum and Mason, and the most recent Climate Camp, there should have been nothing surprising in the protests receiving this kind of negative coverage, especially if they went on for several days.

The Fortnum and Mason protest also indicates the impact this can have on opinion.

read more »

Climate change ‘more important than immigration’

Climate change should be a higher priority for the government than immigration, according to findings of a new poll revealed exclusively in Climate Sock. The results will delight environmental campaigners, who have long been calling for climate change to be taken more seriously as a political issue.

According to the poll, 46% more people think that climate change is an important issue in their life than say the same about immigration or asylum. The results will put pressure on the government, which was criticised last week by environmental leaders, who said it was failing to live up to its pledge to be the “greenest government ever”.

The findings will also put an end to doubts about the public’s trust in the work of climate scientists. Following the 2009 hacking and release online of emails from the world-leading Climate Research Unit at the University of East Anglia, there was widespread speculation that public opinion was increasingly turning against the view that climate change was caused by human activity.

Any doubts now appear to have been overcome, with three in four of those surveyed by Ipsos MORI saying that they think human activity has a significant effect on the climate.

Welcoming the results, Eddard Stark, head of the environmental charity Climate Campaigners, said “The government can no longer hide behind the myth that the public have higher priorities. These results send a clear message: the country wants action to stop climate change, and it wants it now”.

Global warming? Bring it on!

Brits are looking forward to the effects of global warming, according to findings of a new poll revealed exclusively in Climate Sock. The results will delight observers who have long argued that environmental pressure groups routinely exaggerate the negative side of climate change.

read more »

Consider this plausible scenario. An airline’s new poll finds that most people want airport capacity to be increased. Two weeks later an environmental NGO announces that their own poll has found two-thirds oppose airport expansion.

Both polls are conducted by reputable agencies, and both interviewed representative samples of over 1,000 people.

How can we reconcile these two polls, and how should journalists report them?

It’s not a problem with polling

The problem is not that polling is inherently untrustworthy. Conducting a poll of 1,000 randomly chosen people means speaking to about 0.002% of the UK adult population. Yet the results are so reliable that, 19 times out of 20, the result you get will be within 3 percentage points of the result you would get if you asked every single person in the country. UK Polling Report offer a good explanation for why this is the case.

Alternatively, if wading through probabilities isn’t your thing, just consider YouGov’s five most recent political polls. For each, they interviewed over 2,000 different people; the proportion who said they would vote Labour were, respectively, 44%, 42%, 45%, 42%, 42%.

If polling itself is untrustworthy, the consistency in these results would require either quite a coincidence or a grand conspiracy. And for anyone tempted to call it a fix, just remember the outraged reaction when, after the second leaders’ debate last year, YouGov’s instant poll found that Cameron ‘won’. It would be a twisted conspiracy indeed if YouGov rigged polls for the Tories last year, and are now doing so in favour of Labour.

So the problem is not that polling is inherently untrustworthy. The problem is this:

read more »

Crikey.  You wait months for fresh data and then two big ones come at once.  And such is life, they say pretty much opposite things. I’ll get to some proper analysis later, but just for now some first thoughts.

Firstly, about the polls. There’s one in the Guardian that apparently shows concern about climate change to be at the same level now as it was in August ’09, i.e. before the UEA emails, the cold winters, Copenhagen, and the relentless stories about how no-one believes in climate change any more.

Then, there’s one in the Mail – which is actually reporting ONS data from August last year – that shows that agreement with climate science is lower now than it’s been at any point since ’06 (when the figures begin).

So, my reactions:

This isn’t a case of the Guardian being climate warriors and the Mail being climate deniers

As far as I can see, both are reporting the data accurately. There’s no apparent cherry picking, and it looks like the comparisons with previous polls are fair. The Guardian’s reporting stands out for linking directly to both data sets, which I don’t remember ever seeing before – round of applause for Damian Carrington – but the Mail’s doesn’t say anything that I don’t think is justifiable (though it took quite a while to find the data – any reason they couldn’t link to it?).

The questions are different and may not be measuring the same phenomenon

I’ve been saying for a while that the decrease in people saying they’re absolutely convinced that the climate is changing/that global warming is a very big problem may be a factor of the way the ‘debate’ between climate warriors and deniers is being conducted. It’s become so vitriolic that many people are heading for the middle ground, on the assumption that both sides are partly right (or because they’re just sick of it).

So a question like ONS’s, whose answer choices are “very convinced/fairly convinced/not very convinced/not at all convinced/don’t know” would tend to lose people from the extremes of the scale to the middle (as happens to an extent: 45% in ’06 to 41% now).

In contrast, the Guardian’s question was on a discrete scale and didn’t present the contrast between firm opinion vs middle ground (climate change already a threat / will be a threat in the future / not a threat / don’t know). Maybe as a result, there’s less of an effect from the way the debate is being conducted and reported.

read more »

I like:

• Trees. Particularly when they’re part of forests.
• People being able to get into forests with as few restrictions as possible.
• People’s views being taken into account when government policy is formed.

Because of that, I’m a bit sad about what I’m about to write.

If you’re in the UK, there’s a good chance you’ve seen or heard coverage of 38 Degrees’ poll, which apparently showed that 75% of the public are against the government’s plans to privatise some forests and change the way it manages the rest. It’s had coverage pretty much everywhere, from the bleeding hearts at the Guardian and BBC to those bastions of anti-green activism at the Sun and Telegraph.

So being a nerd, the first thing I did when I heard the news was to look for the data. And this was when I started getting sad.

1. The data weren’t published when the articles were written

To my knowledge, all the coverage was put together on the basis of what 38 Degrees gave to the media (the data were put up on the YouGov site today, Monday, with the coverage posted on Saturday or Sunday).

We’ve seen several times before why this matters. If journalists cover a poll without seeing the data, they’re often reliant entirely on the word of people who are trying to promote their own interest.

In November, we saw an EDF poll that won coverage of apparent strong support for a new nuclear power station, on the basis of a question that came after respondents had been reminded of the jobs a power station could create.

And we’ve seen other polls reported with absolutely no data ever published, like the claim made in an Easyjet press release last year that a YouGov poll showed that 80% of UK consumers wanted a rethink of Air Passenger Duty. Without the data being available, there’s no way of knowing whether it was true.

Now, this isn’t particular to 38 Degrees: everyone does it. After all, when you’ve got a shiny new poll fresh from the pollsters, why not get coverage for it straight away?  And of course if you’re a journalist and you know that competitors have also got the same story, you’ve got to cover it straight away.

But here’s another reason why that’s a bad idea:

read more »

In this week’s Economist, there’s a bold assertion casually dropped into an article about the cold winter*: “Britain’s scepticism about climate change … [is] already more widespread than in many other European countries”.

Leaving aside the escape route that ‘many’ provides, it’s quite a claim, and one that I’m not sure I’ve seen evidence for. So what defence can we make of it?

The annual HSBC Climate Confidence Monitor is a good source of international data on attitudes to climate change. It asks consistent questions in a decent number of countries (currently 15 countries), so gives us results that can be measured between countries and over time. This year’s results have recently come out, and are available here.

But before we look at the results, there’s something we need to talk about. Across the world, people respond to survey questions in ways that differ consistently from country to country. In some countries, people are generally more likely to choose upper points on a scale, and in other countries, people tend to stay closer to the middle. In my experience, we see a much higher proportion of people choosing upper points on a scale in China than we do in Germany, for example.

This matters a great deal when we’re comparing international data sets. Because of this difference in international scale-usage patterns, it wouldn’t necessarily be fair, for example, to look at a poll that shows 75% in China saying they’re very worried about climate change, and compared that with 60% in Germany who say the same, and conclude that more people in China are worried about climate change than in Germany.

It’s much safer to look at the kinds of questions that avoid scale-usage patterns. While a question like “On a 7-point scale, how worried are you about climate change” would be subject to scale-usage patterns, a question like “Which of these issues are you most worried about” wouldn’t be, because interviewees have to select just one of the issues.

This brings us back to the HSBC data. There is indeed a question in the poll that avoids scale-usage issues: a list of issues, with interviewees asked to select which is their top concern.

read more »

From time to time a news story comes out citing a poll that isn’t in the public domain. These articles are written on the basis of a press release – apparently all the information the journalist has about the poll.

Given that journalists are supposed to be a cynical bunch, this always strikes me as surprising. By writing up the data from the press release without checking the poll themselves, they’re taking a leap of faith that they’ve been given a fair representation of the truth. Since these press releases (of course) show results that are helpful to the organisation that commissioned the poll, you would expect due diligence for a journalist to include checking the data.

A recent poll by EDF Energy, carried out by ICM, shows why this matters.

The research was conducted among 1002 adults living near the Hinkley Point Power Station, and asked about their attitudes to nuclear power and the possible construction of a new plant.

On the strength of the poll, EDF put out this press release, in which they said that “Nearly four times as many local people support plans for a new power station at Hinkley Point than oppose it”, and that “63% support the development of Hinkley Point C”. The press release was picked up quite widely by local media, including the BBC. Nice job by their PR people in winning positive local coverage.

Fortunately, ICM is a member of the British Polling Council (BPC) and abides by its rules. These rules are strongly weighted towards transparency, and include the stipulation that where research findings have entered the public domain – as in this poll – the full data and complete wording of the questionnaire must be made available.

As ever, ICM have done this, and we can look at the data here to test out EDF’s claim.

Firstly, there’s no dispute about the figures they’ve issued. As they say, 63% are “strongly in favour” or “slightly in favour” of the potential development of Hinkley Point C, and only 17% are slightly or strongly opposed.

However, being able to see the complete data also allows us to see the wording of the whole questionnaire.  The sequence of questions runs:

read more »

It looks like we’ve had the starting pistol for the biannual ritual of the season’s change justifying a spate of articles predicting the next few months’ weather. It’s always fun for us Brits, though not exactly harmless. Misreporting of a Met Office’s 2009 seasonal forecast – as a ‘barbecue summer’ – somehow led to serious suggestion that it should be sold off, despite its record as one of world’s most accurate forecasting bodies.

Now this autumn, the Guardian has pitched in with a story about the early arrival of some Bewick’s swans to the UK. Apparently their early departure from Siberia, tied with a cold forecast for the week ahead, was enough to justify an article predicting a cold winter ahead.

Without wanting to take the article too seriously (it is, after all, only a well-executed piece of PR by the Slimbridge Wetland Centre), the prospect of a cold winter should be a worry for anyone campaigning on climate change. Last year, we saw the collapse of talks in Copenhagen; Climategate; Glaciergate (the stories don’t need to be true to have been reported as damaging climate science) – and the coldest winter in the UK for 31 years. Of these, the weather may well have done the most to influence public concern about climate change.

The evidence for this is circumstantial because no-one asked the right questions, but seems fairly strong. A poll in December ’09, when the stories about UEA emails were at their peak, showed no significant movement in agreement with climate science. Yet, another poll, in January ’10, when the UEA stories had died down, but the cold weather was at its most severe, showed a significant drop in agreement that climate change was a reality (though I think methodological problems with this latter poll seriously weaken it). In the other direction, we’ve also seen that confidence in climate science increases when heatwaves or storms cause major disruption, and the media attribute this weather to climate change.

read more »