Should Ask About Poll Results
National Council on Public Polls, by Sheldon R. Gawiser, Ph.D. and G. Evans Witt
Polls provide the best direct source of information about public opinion. They are valuable tools for journalists and can serve as the basis for accurate, informative news stories. For the journalist looking at a set of poll numbers, here are the 20 questions to ask the pollster before reporting any results. This publication is designed to help working journalists do a thorough, professional job covering polls. It is not a primer on how to conduct a public opinion survey.
The only polls that should be reported are "scientific" polls. A number of the questions here will help you decide whether or not a poll is a "scientific" one worthy of coverage – or an unscientific survey without value.
Unscientific pseudo-polls are widespread and sometimes entertaining, but they never provide the kind of information that belongs in a serious report. Examples include 900-number call-in polls, man-on-the-street surveys, many Internet polls, shopping mall polls, and even the classic toilet tissue poll featuring pictures of the candidates on each roll.
One major distinguishing difference between scientific and unscientific polls is who picks the respondents for the survey. In a scientific poll, the pollster identifies and seeks out the people to be interviewed. In an unscientific poll, the respondents usually "volunteer" their opinions, selecting themselves for the poll.
The results of the well-conducted scientific poll provide a reliable guide to the opinions of many people in addition to those interviewed – even the opinions of all Americans. The results of an unscientific poll tell you nothing beyond simply what those respondents say.
By asking these 20 questions, the journalist can seek the facts to decide how to report any poll that comes across the news desk.
- Who did the poll?
- Who paid for the poll and why was it done?
- How many people were interviewed for the survey?
- How were those people chosen?
- What area (nation, state, or region) or what group (teachers,lawyers, Democratic voters, etc.) were these people chosen from?
- Are the results based on the answers of all the people interviewed?
- Who should have been interviewed and was not? Or do response rates matter?
- When was the poll done?
- How were the interviews conducted?
- What about polls on the Internet or World Wide Web?
- What is the sampling error for the poll results?
- Who’s on first?
- What other kinds of factors can skew poll results?
- What questions were asked?
- In what order were the questions asked?
- What about "push polls?"
- What other polls have been done on this topic? Do they say the same thing? If they are different, why are they different?
- What about exit polls?
- What else needs to be included in the report of the poll?
- So I've asked all the questions. The answers sound good. Should we report the results?
1. Who did the poll?
What polling firm, research house, political campaign, or other group conducted the poll? This is always the first question to ask.
If you don't know who did the poll, you can't get the answers to all the other questions listed here. If the person providing poll results can't or won't tell you who did it, the results should not be reported, for their validity cannot be checked.
Reputable polling firms will provide you with the information you need to evaluate the survey. Because reputation is important to a quality firm, a professionally conducted poll will avoid many errors.
2. Who paid for the poll and why was it done?
You must know who paid for the survey, because that tells you – and your audience – who thought these topics are important enough to spend money finding out what people think.
3. How many people were interviewed for the survey?
Because polls give approximate answers, the more people interviewed in a scientific poll, the smaller the error due to the size of the sample, all other things being equal. A common trap to avoid is that "more is automatically better." While it is absolutely true that the more people interviewed in a scientific survey, the smaller the sampling error, other factors may be more important in judging the quality of a survey.
4. How were those people chosen?
The key reason that some polls reflect public opinion accurately and other polls are unscientific junk is how people were chosen to be interviewed. In scientific polls, the pollster uses a specific statistical method for picking respondents. In unscientific polls, the person picks himself to participate.
5. What area (nation, state, or region) or what group (teachers, lawyers, Democratic voters, etc.) were these people chosen from?
It is absolutely critical to know from which group the interviewees were chosen.
6. Are the results based on the answers of all the people interviewed?
One of the easiest ways to misrepresent the results of a poll is to report the answers of only a subgroup. For example, there is usually a substantial difference between the opinions of Democrats and Republicans on campaign-related matters. Reporting the opinions of only Democrats in a poll purported to be of all adults would substantially misrepresent the results.
Poll results based on Democrats must be identified as such and should be reported as representing only Democratic opinions.
Of course, reporting on just one subgroup can be exactly the right course. In polling on a primary contest, it is the opinions of those who can vote in the primary that count – not those who cannot vote in that contest. Primary polls should include only eligible primary voters.
7. Who should have been interviewed and was not? Or do response rates matter?
No survey ever reaches everyone who should have been interviewed. You ought to know what steps were undertaken to minimize non-response, such as the number of attempts to reach the appropriate respondent and over how many days.
8. When was the poll done?
Events have a dramatic impact on poll results. Your interpretation of a poll should depend on when it was conducted relative to key events. Even the freshest poll results can be overtaken by events. The President may have given a stirring speech to the nation, pictures of abuse of prisoners by the military may have been broadcast, the stock market may have crashed or an oil tanker may have sunk, spilling millions of gallons of crude on beautiful beaches.
9. How were the interviews conducted?
There are four main possibilities: in person, by telephone, online or by mail. Most surveys are conducted by telephone, with the calls made by interviewers from a central location. However, some surveys are still conducted by sending interviewers into people's homes to conduct the interviews.
Some surveys are conducted by mail. In scientific polls, the pollster picks the people to receive the mail questionnaires. The respondent fills out the questionnaire and returns it.
Mail surveys can be excellent sources of information, but it takes weeks to do a mail survey, meaning that the results cannot be as timely as a telephone survey. And mail surveys can be subject to other kinds of errors, particularly extremely low response rates. In many mail surveys, many more people fail to participate than do. This makes the results suspect.
Surveys done in shopping malls, in stores or on the sidewalk may have their uses for their sponsors, but publishing the results in the media is not among them. These approaches may yield interesting human-interest stories, but they should never be treated as if they represent public opinion.
Advances in computer technology have allowed the development of computerized interviewing systems that dial the phone, play taped questions to a respondent and then record answers the person gives by punching numbers on the telephone keypad. Such surveys may be more vulnerable to significant problems including uncontrolled selection of respondents within the household, the ability of young children to complete the survey, and poor response rates.
Such problems should disqualify any survey from being used unless the journalist knows that the survey has proper respondent selection, verifiable age screening, and reasonable response rates.
10. What about polls on the Internet or World Wide Web?
The explosive growth of the Internet and the World Wide Web has given rise to an equally explosive growth in various types of online polls and surveys.
11. What is the sampling error for the poll results?
Interviews with a scientific sample of 1,000 adults can accurately reflect the opinions of nearly 210 million American adults. That means interviews attempted with all 210 million adults – if such were possible – would give approximately the same results as a well-conducted survey based on 1,000 interviews.
What happens if another carefully done poll of 1,000 adults gives slightly different results from the first survey? Neither of the polls is "wrong." This range of possible results is called the error due to sampling, often called the margin of error.
This is not an "error" in the sense of making a mistake. Rather, it is a measure of the possible range of approximation in the results because a sample was used.
Pollsters express the degree of the certainty of results based on a sample as a "confidence level." This means a sample is likely to be within so many points of the results one would have gotten if an interview were attempted with the entire target population. Most polls are usually reported using the 95% confidence level.
Thus, for example, a "3 percentage point margin of error" in a national poll means that if the attempt were made to interview every adult in the nation with the same questions in the same way at the same time as the poll was taken, the poll's answers would fall within plus or minus 3 percentage points of the complete count’s results 95% of the time.
This does not address the issue of whether people cooperate with the survey, or if the questions are understood, or if any other methodological issue exists. The sampling error is only the portion of the potential error in a survey introduced by using a sample rather than interviewing the entire population. Sampling error tells us nothing about the refusals or those consistently unavailable for interview; it also tells us nothing about the biasing effects of a particular question wording or the bias a particular interviewer may inject into the interview situation. It also applies only to scientific surveys.
Remember that the sampling error margin applies to each figure in the results – it is at least 3 percentage points plus or minus for each one in our example. Thus, in a poll question matching two candidates for President, both figures are subject to sampling error.
12. Who’s on first?
Sampling error raises one of the thorniest problems in the presentation of poll results: For a horse-race poll, when is one candidate really ahead of the other?
13. What other kinds of factors can skew poll results?
The margin of sampling error is just one possible source of inaccuracy in a poll. It is not necessarily the source of the greatest possible error; we use it because it's the only one that can be quantified. And, other things being equal, it is useful for evaluating whether differences between poll results are meaningful in a statistical sense.
14. What questions were asked?
You must find out the exact wording of the poll questions. Why? Because the very wording of questions can make major differences in the results.
Perhaps the best test of any poll question is your reaction to it. On the face of it, does the question seem fair and unbiased? Does it present a balanced set of choices? Would most people be able to answer the question?
On sensitive questions – such as abortion – the complete wording of the question should probably be included in your story. It may well be worthwhile to compare the results of several different polls from different organizations on sensitive questions. You should examine carefully both the results and the exact wording of the questions.
15. In what order were the questions asked?
Sometimes the very order of the questions can have an impact on the results. Often that impact is intentional; sometimes it is not. The impact of order can often be subtle.
During troubled economic times, for example, if people are asked what they think of the economy before they are asked their opinion of the president, the presidential popularity rating will probably be lower than if you had reversed the order of the questions. And in good economic times, the opposite is true.
What is important here is whether the questions that were asked prior to the critical question in the poll could sway the results. If the poll asks questions about abortion just before a question about an abortion ballot measure, the prior questions could sway the results.
16. What about "push polls?"
In recent years, some political campaigns and special-interest groups have used a technique called "push polls" to spread rumors and even outright lies about opponents. These efforts are not polls, but political manipulation trying to hide behind the smokescreen of a public opinion survey.
In a "push poll," a large number of people are called by telephone and asked to participate in a purported survey. The survey "questions" are really thinly-veiled accusations against an opponent or repetitions of rumors about a candidate’s personal or professional behavior. The focus here is on making certain the respondent hears and understands the accusation in the question, not in gathering the respondent’s opinions.
"Push polls" are unethical and have been condemned by professional polling organizations.
"Push polls" must be distinguished from some types of legitimate surveys done by political campaigns. At times, a campaign poll may ask a series of questions about contrasting issue positions of the candidates – or various things that could be said about a candidate, some of which are negative. These legitimate questions seek to gauge the public’s reaction to a candidate’s position or to a possible legitimate attack on a candidate’s record.
A legitimate poll can be distinguished from a "push poll" usually by:
The number of calls made – a push poll makes thousands and thousands of calls, instead of hundreds for most surveys; The identity of who is making the telephone calls – a polling firm for a scientific survey as opposed to a telemarketing house or the campaign itself for a "push poll;" The lack of any true gathering of results in a "push poll," which has as its only objective the dissemination of false or misleading information.
17. What other polls have been done on this topic? Do they say the same thing? If they are different, why are they different?
Results of other polls – by a newspaper or television station, a public survey firm or even a candidate's opponent – should be used to check and contrast poll results you have in hand.
If the polls differ, first check the timing of the interviewing. If the polls were done at different times, the differing results may demonstrate a swing in public opinion.
If the polls were done about the same time, ask each poll sponsor for an explanation of the differences. Conflicting polls often make good stories.
18. What about exit polls?
Exit polls, properly conducted, are an excellent source of information about voters in a given election. They are the only opportunity to survey actual voters and only voters.
19. What else needs to be included in the report of a poll?
The key element in reporting polls is context. Not only does this mean that you should compare the poll to others taken at the same time or earlier, but it also means that you need to report on what events may have impacted on the poll results.
20. So I've asked all the questions. The answers sound good. Should we report the results?
Yes, because reputable polling organizations consistently do good work.