Should you believe polls of what China thinks about climate change?

Some years ago I managed a few research projects that used in-depth interviews with businesspeople in various countries. In most places, our interviewers would quickly get the respondents talking, giving me dozens of pages of transcripts to work through.

But when the interviews were in China, we’d get monosyllables. However carefully we set up the questions to invite considered responses, and however much the interviewers probed, the answers were usually of just of one word; occasionally a terse sentence. After a while I began to think it was impossible to do in-depth interviews in China, at least with the approach we were trying.

I mention this because two recent climate change polls have got me wondering again about research in China, and whether the results are reliable.

After a bit of digging, which I describe in this article, I’m increasingly uncertain about the reliability of poll findings from China – specifically here polls on climate change – though I can’t go as far as to say they’re definitely wrong.

The two recent polls both showed the proportion of Chinese people saying they think climate change is a very serious problem. According to YouGov’s poll the figure is 52%, while Pew put it at just 18%.

The latter of these generated headlines about a lack of worry about climate change in China – which of course fits with a sceptic/denier narrative that richer countries shouldn’t cut emissions because China isn’t doing anything. But having looked at the polls, I’m not sure such headlines are warranted.

At first glance, it seems like at least one of these polls must be wrong. The question wording is similar, they were conducted around the same time, yet the results are way beyond the margin of error.

In fact, there may be an easy explanation here: the YouGov poll is of their panellists and was weighted to be representative of the adult online population, while the Pew poll purports to be randomly sampled and nationally representative.

Actually, if YouGov’s panel was typical of the Chinese online population, the online/face-to-face split still couldn’t explain all the difference. Apparently, nearly half of the Chinese population is now online; even if we were to make the heroic assumption that none of the offline population thought climate change is a very serious problem, we’d still end up with nearly 26% saying it was very serious from YouGov’s numbers – again, beyond the Pew margin of error.

We might instead put this down to YouGov’s panel being unrepresentative of the Chinese population in general: perhaps they’re people who are more engaged with the news or international affairs than the average person.

Except, there’s another problem.

The Pew poll is not only out of line with YouGov’s survey, it’s also different from the result Pew found last time it asked the same question: the proportion saying it’s a very serious problem has more than halved:

 China

This seems pretty weird to me. I’m not aware of such a large shift in attitudes towards climate change over five years in any country, including in the UK and US following the UEA email hoax six years ago.

For comparison, here’s a graphic I keep to hand showing the stability of UK opinion on the climate over that period.

 Climate polls

I can think of a few possible explanations for the difference in the Pew results.

The first is that there truly has been a large shift in attitudes to climate change in China. At the moment I have no way of proving this either way, but a shift of this magnitude would be unusual and should have some explanation. I can’t think of any such explanation, though there may well be something that, in my ignorance of Chinese debates, I’m unaware of.

A second possibility is that there isn’t a true public opinion about climate change in China. This would fit with John Zaller’s view that public opinion mostly doesn’t exist: people only have ‘opinions’ on many issues (perhaps not on all issues, though) when they’re asked to express their view, for example by a pollster. At that point, they sample from the, perhaps contradictory, opinions they’ve previously heard and accepted. This is a recipe for volatility: people’s opinion one day may not be the same the next, and they’re easily swayed by what they hear from elites.

This would perhaps explain the volatility from 2007-2009 as well as the change from 2010-2015, but it doesn’t feel seem it can be the full picture. It’s not clear why China’s population would show such volatility while those in other countries would have more stable views. Perhaps climate change is talked about much less often in China than it is in other places, so people have less opportunity to form fixed opinion there. Again, I don’t know enough about Chinese debates to settle this, but given that Chinese respondents seem to say ‘don’t know’ to climate questions less often than people in other countries, I’m not convinced. It’s also not like most people in the UK or US hear or talk about climate change very often, anyway.

A third option is that Pew got super unlucky with their sample. With a perfectly sampled poll, one in twenty will produce a result that’s outside the margin of error, relative to the true value of public opinion (if such a thing exists). Through no fault of the samplers, Pew might have just happened to pick the people who really aren’t worried about climate change. This is possible – and maybe China has such diversity of opinion between areas that the sampling approach is more prone to this kind of bad luck – but they’d have to have been supremely unlucky for this to explain the size of the gap (unless particularly factors in the structure of Chinese opinion make it more likely).

This leaves a fourth option – changes in sampling – which I’ll discuss in more detail.

The 2015 Pew poll claims to be representative of the Chinese adult population. It excludes Tibet, Xinjiang, Hong Kong and Macau, but since they only make up 2% of the population, that shouldn’t affect us much.

In comparison, the 2010 poll (and I assume earlier ones) didn’t claim to be representative. Its sample was about 50% too urban (67% rather than 43%) and is described as representing roughly 42% of the adult population.

The smoking gun? Maybe. The skew would probably be in the direction we’ve seen: people in cities, being better educated and better connected to the outside world would probably be more likely to have heard about, and be worried about climate change. Perhaps the 2015 figure is the true one, and earlier ones were unrepresentative.

Except, the results of another question leave me less convinced.

I’ve gone through the Pew polls looking for another question where I’d expect overall national attitudes to be relatively constant over time and for there to be an urban-rural divide. If such a question showed a similar shift as the sample lost its urban skew, we can be more confident the issue was with the sample. But if it stayed constant, there may be something else going on.

The best I can find is one that asked respondents whether they agreed with the statement that “most people are better off in a free market economy, even though some people are rich and some are poor”. I can’t find 2015 data for this, but 2014 seems to have been conducted with the same sampling approach as 2015, so it should work as a substitute.

Unfortunately, the responses have been largely stable:

China 2

This seems to undermine my theory that the change in the climate-change question is a result of the shifting sample frame.

One way the sample-frame theory might survive is if we disagree that there would be a rural/urban split in the free-market question. Perhaps opinion on that is consistent across China in a way that that views of climate change isn’t. That’s a plausible objection, so perhaps the sample-frame theory is ok after all. This would mean that concern about climate change in China might always have been much lower than we’d realised – but this has, until now, hidden by a disproportionately urban sample. While that wouldn’t fit with any other poll I’ve seen, perhaps all of those might also have the sampling bias that Pew had before 2011.

This could all be resolved if Pew looked at their 2015 climate change data and filtered out everyone who wouldn’t have been sampled in their pre-2011 polls. Then we’d see if the ‘new’ people had markedly different views from the ‘old’ ones. Equally, the 2015 results of the free-market question (if it was asked then) would help us see whether that poll happened to have picked an unusual sample.

Until either of those things happen, though, I would be suspicious of all polls of what people in China think of climate change – at least until they start showing consistent results.

It’s always a warning sign when polls give widely different answers from one another. Combine this with the apparent finding that concern about climate change in China has collapsed and is now lower than any other sampled country – neither of which has been reflected in other polls (though these may have sampling problems). While the latest Pew poll does seem to have a nationally representative sample, the change in their sample frame didn’t produce dramatic shifts of results in other of their questions, which makes me suspicious of the explanation that the old sample frame gave the wrong answer and the new one gives the true opinion.

Ultimately, I can’t see any good explanation for the Pew finding is so far out of line with every other poll. Unless something dramatic has happened in discussions about climate change in China to have turned people against worrying about it, the most likely cause seems to be something about the sampling. But what it is about the sampling, I don’t know.

Share

Comments are closed.