More bad poll reporting… even when it’s in the name of the forests

I like:

  • Trees. Particularly when they’re part of forests.
  • People being able to get into forests with as few restrictions as possible.
  • People’s views being taken into account when government policy is formed.

Because of that, I’m a bit sad about what I’m about to write.

If you’re in the UK, there’s a good chance you’ve seen or heard coverage of 38 Degrees’ poll, which apparently showed that 75% of the public are against the government’s plans to privatise some forests and change the way it manages the rest. It’s had coverage pretty much everywhere, from the bleeding hearts at the Guardian and BBC to those bastions of anti-green activism at the Sun and Telegraph.

So being a nerd, the first thing I did when I heard the news was to look for the data. And this was when I started getting sad.

1. The data weren’t published when the articles were written

To my knowledge, all the coverage was put together on the basis of what 38 Degrees gave to the media (the data were put up on the YouGov site today, Monday, with the coverage posted on Saturday or Sunday).

We’ve seen several times before why this matters. If journalists cover a poll without seeing the data, they’re often reliant entirely on the word of people who are trying to promote their own interest.

In November, we saw an EDF poll that won coverage of apparent strong support for a new nuclear power station, on the basis of a question that came after respondents had been reminded of the jobs a power station could create.

And we’ve seen other polls reported with absolutely no data ever published, like the claim made in an Easyjet press release last year that a YouGov poll showed that 80% of UK consumers wanted a rethink of Air Passenger Duty. Without the data being available, there’s no way of knowing whether it was true.

Now, this isn’t particular to 38 Degrees: everyone does it. After all, when you’ve got a shiny new poll fresh from the pollsters, why not get coverage for it straight away?  And of course if you’re a journalist and you know that competitors have also got the same story, you’ve got to cover it straight away.

But here’s another reason why that’s a bad idea:

2. This poll doesn’t give a fair representation of public opinion

There were only two questions in the poll, so it won’t take long to look at the wording:

1. To what extent do you agree or disagree with the following statements [sic]?

‘Publicly owned forests and woodlands should be kept in public ownership for future generations.’

So respondents have been prompted with an assertion of one point of view, with some emotive language (“publicly owned”, “for future generations”) that frames the issue in a particular way. After this, their responses will be affected by the first question.

2. Currently, publicly owned forests and woodlands are managed by the Forestry Commission. The government is considering plans to sell off some, or all, of the publicly owned woodlands and forests in England, claiming that forests will be better run if the they [sic] are owned by private companies, charities or individuals rather than a public sector organisation. Others however argue that the selloff is short-sighted and fear that woodland areas will be bought by developers and timber companies who could exploit the forests they own by limiting access to the public and endangering woodland wildlife.

To what extent do you support or oppose the government’s plans to sell publicly owned woodlands and forests in England?

The wording in this question is very long so a lot of people won’t read it, and will say they oppose the sale on the basis of the unbalanced previous question. It also, to my mind, makes the case against the sale much more enthusiastically and using more words than it makes the opposite case.

Most importantly, I don’t see any reason why the questions should be asked in this order.

The key question is the second one, as that asks directly whether people support or oppose the government’s plans. Putting them in this order just means that the main question is influenced by the framing of the first question.

If I really wanted to find out what people thought, I would go for something that’s as balanced as I could, like:

1. The government has recently proposed selling some parts of forests that are publicly owned.

Supporters of the plan say this would give local people and companies more control over how the forests are used. But opponents say that developers would exploit the forests they own by limiting access to the public and endangering wildlife.

To what extent would you support or oppose these plans?

But of course an organisation that’s campaigning on an issue will always hope for poll results that favour their cause.  And they’ll keep on getting away with it until the day that journalists start demanding to see the data before they carry the story.

  1. […] it’s with reluctance I link to this blog-post by the excellent Climate Sock. Read it and weep. Shoddy polling and dodgy numbers are not to be supported – even if they […]