The lesson is clear. Question wording can have a substantial impact on responses, with complex or confusing wording yielding inaccurate results and false readings of public opinion.
Polling about Holocaust denial has created horror and consternation on multiple occasions, offering lessons applicable to polling more broadly.
In probably the most widely reported poll finding of 1992, more than one in five Americans (22 percent) doubted the Holocaust had occurred, with another 12 percent unsure.
Survivor Eli Weisel was aghast. “What have we done?” the late Nobel Laureate lamented to a reporter. “I’m shocked that 22%–oh my Gd.”
More than a year later, it became clear that 22 percent was a significant exaggeration resulting from a poorly worded question.
Posed by the Roper Organization, the question asked, “Does it seem possible, or does it not seem possible to you that the Nazi extermination of the Jews never happened?”
Flaws in the wording were several. First, it was confusing, a problem compounded by the double negative — impossible that the Holocaust never happened.
Moreover, part of the definition of extermination is the “complete” elimination. The Nazis intended and attempted to kill all of world Jewry and succeeded in killing 6 million but failed to completely eliminate the Jews.
Two tests over a year later clearly revealed the problematic nature of the question.
Roper offered another clearer, but not perfect, version of its original wording. “Does it seem possible to you that the Nazi extermination of the Jews never happened or do feel certain that it happened?”
Asked this question, just 1 percent said it was possible the Holocaust never happened, and 8 percent were unsure.
Rival Gallup conducted its own test, splitting the sample between Roper’s original wording and their own question, “The term Holocaust usually refers to the killing of Jews in Nazi death camps during World War II. Do you doubt the Holocaust actually happened or not?”
Among those who heard Gallup’s version, 87 percent had no doubt about the Holocaust while 9 percent harbored some doubt and 4 percent didn’t know. By contrast, among those who heard the original version, 33 percent said it was possible it never happened, and 2 percent had no opinion.
In short, the original, confusing version of the question consistently produced about 35 percent who either didn’t know or thought the Holocaust possibly never happened, while more comprehensible versions found only 9 percent-13 percent holding those views.
The lesson is clear. Question wording can have a substantial impact on responses, with complex or confusing wording yielding inaccurate results and false readings of public opinion.
Last year, another Holocaust survey made waves.
“Our new poll makes alarming reading,” proclaimed The Economist, “One in five young Americans thinks the Holocaust is a myth.” Once again, many were understandably outraged.
Twenty percent of those under 30 years old agreed with a statement saying, “The Holocaust is a myth,” while 30 percent responded with “don’t know.” This time, the error seemed to be less in question wording (though such agree/disagree questions are notorious for creating acquiescence bias) and more in sampling.
The YouGov poll eliciting this response relied on an opt-in panel for results. That is, those who were sampled had, in response to an ad or other request, volunteered to participate in surveys, usually in exchange for rewards and prizes.
It turns out some of these poll respondents do not answer sincerely but seem bent on completing the survey as quickly and easily as possible. Pew found such respondents are concentrated among those under 30.
For example, in a Pew study, 12 percent of opt-in panel participants under 30 claimed they were licensed to operate a nuclear submarine. Among those 30-60, 8 percent professed holding a nuclear sub license along with .3 percent of those over 60.
It should come as no surprise that about zero percent actually hold such a license.
With a random probability sample, that is without respondents who volunteered to take surveys in exchange for inducements, a more accurate 1 percent or less in each age category claimed a submarine license.
So, when Pew asked an opt-in panel of their own whether the Holocaust was a myth, a similar 20 percent of those under 30 agreed. However, when the question was posed to a random probability panel, only 3 percent of those 18-29 agreed.
Similar differences emerged on questions about abortion.
Under 30s are usually among the most ardent defenders of abortion rights. However, among under 30s in the opt-in sample, 48 percent said abortion should be illegal or legal only in special circumstances. A random sampling method found fewer than half as many, 23 percent, endorsing that restrictive view.
The broader lesson: Be wary of results produced by opt-in panels, especially for subgroups like younger voters.
Mellman is president of The Mellman Group and has helped elect 30 U.S. senators, 12 governors and dozens of House members. Mellman served as pollster to Senate Democratic leaders for over 20 years, as president of the American Association of Political Consultants, a member of the Association’s Hall of Fame, and is president of Democratic Majority for Israel.
