Some years ago I managed a few research projects that used in-depth interviews with businesspeople in various countries. In most places, our interviewers would quickly get the respondents talking, giving me dozens of pages of transcripts to work through.
But when the interviews were in China, we’d get monosyllables. However carefully we set up the questions to invite considered responses, and however much the interviewers probed, the answers were usually of just of one word; occasionally a terse sentence. After a while I began to think it was impossible to do in-depth interviews in China, at least with the approach we were trying.
I mention this because two recent climate change polls have got me wondering again about research in China, and whether the results are reliable.
After a bit of digging, which I describe in this article, I’m increasingly uncertain about the reliability of poll findings from China – specifically here polls on climate change – though I can’t go as far as to say they’re definitely wrong.
The two recent polls both showed the proportion of Chinese people saying they think climate change is a very serious problem. According to YouGov’s poll the figure is 52%, while Pew put it at just 18%.
The latter of these generated headlines about a lack of worry about climate change in China – which of course fits with a sceptic/denier narrative that richer countries shouldn’t cut emissions because China isn’t doing anything. But having looked at the polls, I’m not sure such headlines are warranted.
At first glance, it seems like at least one of these polls must be wrong. The question wording is similar, they were conducted around the same time, yet the results are way beyond the margin of error.
In fact, there may be an easy explanation here: the YouGov poll is of their panellists and was weighted to be representative of the adult online population, while the Pew poll purports to be randomly sampled and nationally representative.
Actually, if YouGov’s panel was typical of the Chinese online population, the online/face-to-face split still couldn’t explain all the difference. Apparently, nearly half of the Chinese population is now online; even if we were to make the heroic assumption that none of the offline population thought climate change is a very serious problem, we’d still end up with nearly 26% saying it was very serious from YouGov’s numbers – again, beyond the Pew margin of error.
We might instead put this down to YouGov’s panel being unrepresentative of the Chinese population in general: perhaps they’re people who are more engaged with the news or international affairs than the average person.
Except, there’s another problem.
The Pew poll is not only out of line with YouGov’s survey, it’s also different from the result Pew found last time it asked the same question: the proportion saying it’s a very serious problem has more than halved:
This seems pretty weird to me. I’m not aware of such a large shift in attitudes towards climate change over five years in any country, including in the UK and US following the UEA email hoax six years ago.
For comparison, here’s a graphic I keep to hand showing the stability of UK opinion on the climate over that period.
I can think of a few possible explanations for the difference in the Pew results.
The first is that there truly has been a large shift in attitudes to climate change in China. At the moment I have no way of proving this either way, but a shift of this magnitude would be unusual and should have some explanation. I can’t think of any such explanation, though there may well be something that, in my ignorance of Chinese debates, I’m unaware of.
A second possibility is that there isn’t a true public opinion about climate change in China. This would fit with John Zaller’s view that public opinion mostly doesn’t exist: people only have ‘opinions’ on many issues (perhaps not on all issues, though) when they’re asked to express their view, for example by a pollster. At that point, they sample from the, perhaps contradictory, opinions they’ve previously heard and accepted. This is a recipe for volatility: people’s opinion one day may not be the same the next, and they’re easily swayed by what they hear from elites.
This would perhaps explain the volatility from 2007-2009 as well as the change from 2010-2015, but it doesn’t feel seem it can be the full picture. It’s not clear why China’s population would show such volatility while those in other countries would have more stable views. Perhaps climate change is talked about much less often in China than it is in other places, so people have less opportunity to form fixed opinion there. Again, I don’t know enough about Chinese debates to settle this, but given that Chinese respondents seem to say ‘don’t know’ to climate questions less often than people in other countries, I’m not convinced. It’s also not like most people in the UK or US hear or talk about climate change very often, anyway.
A third option is that Pew got super unlucky with their sample. With a perfectly sampled poll, one in twenty will produce a result that’s outside the margin of error, relative to the true value of public opinion (if such a thing exists). Through no fault of the samplers, Pew might have just happened to pick the people who really aren’t worried about climate change. This is possible – and maybe China has such diversity of opinion between areas that the sampling approach is more prone to this kind of bad luck – but they’d have to have been supremely unlucky for this to explain the size of the gap (unless particularly factors in the structure of Chinese opinion make it more likely).
This leaves a fourth option – changes in sampling – which I’ll discuss in more detail.
read more »