How DECC is wasting money on its new opinion poll

The Department of Energy and Climate Change has started a new tracking poll on public attitudes to a few of their issues. The first wave was out earlier this year (details and results here), and wave 2 should be out soon.

It won’t be much of a surprise that I’m generally in favour of polling. It’s important that people in government (and others with power) should know what the public think about the policies they’re making decisions about, and well-conducted opinion polls are a way of finding this out. They equalise the volume of everyone’s voices so that each opinion counts the same: media mogul or not (of course it doesn’t deal with how those opinions are formed).

DECC’s poll is on an important topic, conducted not to create headlines but so the government can better understand what the public think, so I should be in favour. But I’m increasingly of the view that it’s been badly put together and is costing too much public money.

Confusing questions

The first issue is the quality of the questions. For brevity I’m just going to focus on the two climate change questions, though there are also others I could make the same argument about.

One of the questions asks: “How concerned, if at all, are you about current climate change, sometimes referred to as ‘global warming’?”.

The problem is that word ‘current’. I think it’s intended to distinguish 20th/21st Century climate change from historical climate change: the ice ages and so on.

But when I first read the question, I understood it to mean the climate change we’re experiencing in 2012, as opposed to what we’re going to experience in 20 years’ time. Since I’m only a little concerned about the climate change we’re experiencing in 2012, I would answer the question accordingly.

I don’t know whether others would understand the question as I did, or whether they would think ‘current’ is referring to 20th/21st Century climate change. Given that, I have no idea how to interpret the results of the question, and no-one else can know either.

The other climate change question asks: “Thinking about the causes of climate change, which, if any, of the following best describes your opinion?”, which all seems fine to me.  But then the answer choices are:

  1. Climate change is entirely caused by natural processes
  2. Climate change is mainly caused by natural processes
  3. Climate change is partly caused by natural processes and partly caused by human activity
  4. Climate change is mainly caused by human activity
  5. Climate change is entirely caused by human activity
  6. I don’t think there is such a thing as climate change.
  7. Don’t know
  8. No opinion

What on earth is choice 3 supposed to be doing? If I think that climate change is mainly human but could also be a bit natural, I could pick either choice 3 or choice 4. Someone who thought it was mostly natural but a bit human could pick choice 2 or choice 3. Given these different interpretations, it’s hard to know what the data mean.

At least one wave has been run with these two questions, and it’s always nice to repeat the same questions so results can be compared over time. But since these don’t tell us anything of use, they really should be changed as soon as possible – and all the other questions should be looked at carefully as well.

The most expensive methodology

That brings me to my second concern about the poll: whether it’s worth the money.

Thanks to FOI I know that four waves of the poll – one year’s worth – will cost the taxpayer £130,945 + VAT. The reason it’s so expensive is twofold: it’s being done by face-to-face interview and it’s repeated every three months.

Repeating a poll as often as every three months is great for tracking opinion on things people hear a lot about and where opinion changes often. If you want to know what people think about government cuts vs public spending, regular polling can tell you how things are changing.

But there’s not much point asking more often than, say, once a year, how much people know about carbon capture and storage or whether they like renewable energy. They just aren’t going to change very quickly.

From my understanding of the poll’s tender document, questions will be shuffled so some won’t be asked in every wave. This is sensible, but even so there aren’t many questions in the poll where opinion will regularly change significantly over a three-month period. It’s hard to see why the poll couldn’t be done on a six-monthly basis (or even less often), which presumably would halve the costs and still give up-to-date numbers.

Something similar applies to the use of face-to-face interviewing. As mainstream survey methods go, it’s the best way of getting a representative sample, but also the most expensive way. Doing the survey over the phone or especially online would have been much cheaper: perhaps half the price or less.

This gets us to tricky methodological territory. Face-to-face guarantees some audiences that are hard to reach online. In theory – if both are done as well as they can be and weighted properly – you end up with a more representative sample when you knock on people’s doors. But is this slight benefit worth, at a conservative estimate, £65,000 a year to the taxpayer?

It partly depends what’s going to be done with the data. If the government is going to plan services around the results, especially anything targeted at people who’re hard to reach with online polls, I can see the justification for making sure the sample is absolutely as representative as possible, as a sort of census of people’s energy usage. But given that most of the questions are about opinions, not behaviour – do you support or oppose nuclear energy and so on – it doesn’t look like it will be used to plan services.

Plenty of major companies have marketing strategies that are based on audience segmentations derived from online polls. The fact that online polling has some limitations doesn’t put them off taking advantage of the fact it’s much cheaper. I suspect it’s the assumption of civil service procurement that such polls should be done face-to-face. If so, it’s really time for this to be revised.

At a time when every penny of civil service spending is supposed to be scrutinised, and some important and valuable services are being cut or are on their way out, I’m surprised that this poll has been given so much money. I’m also surprised that it’s ended up including questions that serve no useful purpose – in the process frustratingly missing an opportunity to answer two of the most frequently recurring questions in opinion on climate change.

  1. […] But my main issue is on the quality of the questions. More here on why it has confusing questions. […]

  2. […] climate change is one of the country’s top three challenges, up from 10% two years ago. I have some serious issues with this tracker – it’s overpriced, some of the questions are so badly written as to be […]