Inside Public-Opinion Polling
“The short memories of the American voters is what keeps our politicians in office.”
― Will Rogers
A Majority of Americans Don't Trust Polling (and a lot of other things)
According to a Maris/McClatchy poll, 60% of Americans either don’t trust polling at all (22%) or very much (38%). Only 37% said they had either a great deal (7%) or a good deal (30%) of confidence in them.
But how can we trust that this information is accurate, since it came from a poll?
As a longtime public-opinion pollster, I think most polls conducted by nonpartisan professionals are largely reflective of public sentiment. However, we need to acknowledge the challenges facing survey research today, and why you shouldn’t bet the rent on a single poll, or sometimes even on many polls.
Skepticism about polling may be part of a broader loss of faith in institutions. This 2021 Gallup Poll measured confidence in 14 institutions. Only three (small business at 70%, the military at 69%, and the police at 51%) enjoyed majority support, with respondents having either a great deal or a lot of confidence in them.
Polling Over Time
To better understand the challenges facing pollsters today, let’s examine how changes in technology and the public’s growing reluctance to take part in polls may impact their accuracy.
To begin with, this is not the golden age of polling. Ironically, in this era of unprecedented connectivity, learning what people believe has actually become more difficult than it was with earlier generations. When public-opinion polling began in earnest in the 1930s, being asked to participate in one was actually considered quite an honor. You’d get an invitation letter or telephone call from the Gallup organization or another one of the few national pollsters of that era. They’d make an appointment to interview you at your home. A well-dressed person would spend the better part of an hour writing down your answers on a paper questionnaire while you served coffee and treats.
However, in-person interviewing, particularly for national polls, was logistically difficult, time-consuming, and expensive. Though phone surveys could be completed faster and at lower costs, it took about 20 years for them to become accepted as a valid means of public-opinion polling. The long road to acceptability was largely based on what percentage of the population had telephones.
Phone polling enjoyed a long period of success. Advances in communications technology, and particularly the advent of Caller ID, required pollsters to make more and more calls in order to complete their surveys. The difficulty in reaching respondents drove up their costs and raised questions about their accuracy.
In the latter half of the 1990s, widespread public use of the Internet opened up a new way to reach potential respondents. But as with the initial usage of the telephone to learn about public opinions, polls conducted online faced a slow road to acceptance by many in the polling industry, as well as those the news media. Online polling is the most common method used today for public-opinion research.
Determining Likely Voters: Part Science, Part Art, Part Luck
Determining which groups or individuals are most likely to vote in a given election is like creating a good recipe. Every pollster has their own secret sauce for making such determinations. I’ll briefly describe a few of those methods here.
Some pollsters use registered voter lists from which to draw their samples. To determine likely voters, many contact only those who have voted in all or the most recent elections. It stands to reason that people who have voted frequently will do so again.
That approach can be problematic, though, when a large number of new or historically infrequent voters show up to vote in a given election. Such intermittent voters may be motivated to cast their ballot by a particular candidate, race, or measure on the ballot.
Another way to determine likely voters is by building a scale of their responses to a series of questions, including such details as how certain they are that they will vote, their level of interest in the election, and so on.
I sometimes add into the mix their level of knowledge of the candidates and measures on the ballot, especially as the election nears. If a person doesn’t know who or what is on the ballot at that point, then it may be an indicator that they are less likely to vote.
Most pollsters adjust the factors they use and the weight attributed to each. As in any field, you learn from past experiences and make adjustments accordingly.
Did You Know…?
What was the average response rate for 2024 Election Polls conducted by The New York Times and Sienna College?
A) 1%
B) 10%
C) 20%
D) 35%
You can find the correct answer at the bottom of this page.
Summing It Up
Given the challenges facing researchers today, it’s usually better to consider the results of a number of polls rather than just a single one. As in nearly every aspect of politics today, there can be a partisan influence in polling.
Most public-opinion polls used to be conducted by independent firms. Today, many are done by firms with partisan leanings or sponsors. Some consolidators, who report and compare the results of different polls, consider the pollsters’ historical accuracy and who sponsors their work.
Polling averages used to be a fairly reliable way to get you in the neighborhood of what people are thinking. But with many polls now being conducted or sponsored by firms with partisan connections, such averages can be misleading.
Some analysts believe examining the results of recent elections may be a better indicator of future outcomes, since you’re dealing with what actual voters did. That makes sense, though at times, there simply may not be enough recent election data from which to draw reliable conclusions.
My advice is to think of polling today as you might a GPS system. They’ll usually get you to your destination, but not necessarily via the best route.
Why Polling Today is So Challenging
A major reason why polling is so challenging today is that far fewer people are willing to participate in them. Do you answer your phone for an unknown caller? Do you respond to voice-mails, e-mails, or texts from organizations or people you don’t know?
Cooperation and response rates to polls have been declining for decades. The cooperation rate is the percentage of people a pollster reaches out to and who actually take the poll. Response rates also factor in other outcomes, such as reaching disconnected numbers and answering machines.
This chart from the Pew Research Center, one of the nation’s most respected pollsters and analysts, illustrates the dramatic drop in responses. In just two decades, telephone survey response rates went from about 36% in 1978 to just 6% in 2018. Cooperation rates tend to move in tandem with response rates.
When a pollster can interview only about one of every 20 people they’ve attempted to reach, it's not hard to see how that would impact a poll’s results. These rates are even lower today.
Though it’s difficult to say much about people you don’t interview, there are indications of a growing none-of-your-beeswax sentiment among folks who just don’t want to share their opinions with people they don’t know. What’s more, the refusal to respond is often not uniform across demographic or political groups. A factor called partisan nonresponse bias is a fancy way of saying voters of different parties may not take polls at similar rates, which would then skew survey results.
What’s the Best Way to do Political Polling Today?
There are three commonly used methods of polling today: in person (also called intercept polling), by telephone, and online. Postal mail polls are still around, but are employed mostly by government agencies, and in declining numbers there as well. In-person polls, while great for conducting interviews among crowds of people—in places such as stadiums—aren’t very practical for election polling across broad geographies, such as a State or the entire country..
If you were polled, how would you like to complete it? From my experience, as long as they’re confident that it’s a legitimate research effort, most people today prefer completing surveys online, in part because they can do so at their convenience.
This informative article from the Pew Research Center discusses how much the methodologies used by pollsters have changed since 2016. One big alteration is the increase in multimodal interviewing—meaning the conducting of polls by more than one method—which spreads out the downside risks of individual methodologies.
Online polls can be more interesting and engaging because they can make use of audio, video, and other media assets. Some people also feel more comfortable expressing their honest opinions without a live interviewer present. (See the Bradley Effect for more on this).
Yet many analysts still refer to telephone polls as the “gold standard” of polling. In a few situations, usually when polling organizations have a lot of money and time, they do remain a viable alternative. However, their high costs and completion times severely limit their usability.
Online polling’s lower costs allow for more frequent research, larger sample sizes, and greater ease in reaching both younger people and those who speak different languages. Clinging to phone polling, a 75-year-old methodology, doesn’t make much sense. Polling needs to look forward, not backward.
Answer to the Did You Know…? Question
The answer is A) 1%.
According to this Scientific American article, the average response rate for the 2024 Election Polls conducted by The New York Times and Sienna College was about 1%. That means that for every hundred people asked to participate in a poll, 99 of them declined. To put this number in context, consider that the Pew Center chart referenced above shows response rates dropping from about 35% in 1997/1998 to about 6% in 2017/2018. Just six years later, that number had shrunk even further.