Survey Bias

Also called Bias in surveys. Also in Business

Quick answer

Survey bias is a systematic error that moves survey results away from the truth in the same direction every time. It enters through who is asked, who chooses to answer, how questions are worded and ordered, and what people decide to say. Because the error always points one way, collecting more responses does not remove it.

~20 sec read

011 min

The 1936 poll that mailed millions and still missed

In autumn 1936, the magazine Literary Digest set out to predict who would win the US presidential election. It mailed a ballot to a very large list of people and asked whom they would choose.

The magazine drew its mailing list from telephone books and car registration records, and about one in four recipients returned a ballot. Almost 2.4 million ballots came back, and the poll put Alfred Landon ahead with 54 percent of the popular vote. Franklin Roosevelt won more than 60 percent instead.

A poll with millions of answers should be precise, and this one was precise. It was also wrong by a wide margin. The cause was survey bias: a systematic error built into who was asked, who replied, or how the questions worked. The next section shows why the size of the sample could not help.

022 min

How survey bias differs from random error

Survey bias differs from random error in one way: it moves every result in the same direction. A survey estimate can be wrong for two separate reasons, and they behave differently.

The first is random error. It comes from asking a sample instead of everyone, so a different random sample would give a slightly different answer. The typical size of this chance variation is called the standard error, and it shrinks as the sample grows. The margin of error reports it.

The second is bias. If a list leaves out poorer households, every sample drawn from that list contains too few of them. A larger sample from the same list gives a more precise estimate of the wrong number.

The 1936 figures show the difference. For a simple random sample of 2.4 million people split near 50-50, the 95 percent margin of error is about 0.06 percentage points. The Digest sample was not random, so this is only a rough guide. Chance cannot explain the miss.

Quantity1936 value
Ballots returnedalmost 2.4 million
Random-sampling margin, 95 percent (calculated)about 0.06 points
Landon's forecast share54 percent
Roosevelt's actual shareover 60 percent, so Landon's was under 40
Gap in Landon's sharemore than 14 points

The gap is more than 200 times the margin. Random error was tiny and bias was large. The rest of this article traces the bias to its sources, in the order a survey meets them.

031 min

Survey bias from who gets asked

Survey bias often starts before any question is written, with the list of people who can be reached. This list is the sampling frame. Anyone missing from it has no chance of being asked, and statisticians call that undercoverage. The Digest's telephone and car lists left out many poorer households.

Convenience sampling has the same effect without a list. The researcher asks whoever is easy to reach. A team that surveys only the readers of its own newsletter hears from people who already follow the product.

Sampling designs protect against this by giving every member of the population a known chance of being picked. Random sampling picks people by chance. Stratified sampling splits the population into groups, such as age bands, and samples each group on purpose, so no group is missed by accident.

The survey mode matters too. The mode is the way people are reached: online, by phone, in person or by mail. An online-only survey misses people without reliable internet, and a phone survey misses people who do not answer unknown numbers. Offering more than one channel narrows the gap.

041 min

Survey bias from who answers

A complete list still produces survey bias when the people who answer differ from the people who do not. This is nonresponse bias. It needs two things: some people skip the survey, and their reason for skipping is linked to the answer they would give. A company that emails an employee survey and hears mostly from satisfied staff has this problem, because unhappy staff are the least likely to reply.

Self-selection is the same problem seen from the other side. When people choose on their own to take part, as with a poll on a website, the respondents are the people who care most. Voluntary bias covers it in detail.

A survey can also miss people who have already gone. A satisfaction survey sent only to current customers cannot hear from customers who cancelled. This is survivorship bias inside a questionnaire.

Falling response rates make many readers expect biased results. Whether that is right depends on the question being asked, as the next section shows.

051 min

When a low response rate does not mean survey bias

Pew Research Center reports that its typical telephone survey had a 36 percent response rate in 1997 and has 9 percent today. Many people read that as proof that surveys are now unreliable. Pew tested the idea, and the answer has two parts.

A low rate is not proof of bias

Pew ran its standard survey, which reached 9 percent of the people it contacted. It ran a second survey with a longer field period, payments to respondents, and letters to households that had declined, and that survey reached 22 percent. Of 40 comparisons between the two, 28 differed by two percentage points or less. The extra effort and expense gave little improvement in the estimates.

Bias follows the topic

The same study found where nonresponse does harm. Pew's survey found that 55 percent of people reported volunteer work in the past year, against 27 percent in a federal household survey. People who do volunteer work also seem more willing to answer a survey. Nonresponse bias therefore concentrates in questions linked to willingness to take part, and a survey can be accurate on one question and far off on the next.

So judge nonresponse bias question by question. One overall response rate does not give the answer.

061 min

Survey bias in online opt-in panels

Online opt-in panels add a newer source of survey bias. In an opt-in panel, people sign themselves up and are rewarded for finishing surveys. Some of them are what Pew Research Center calls bogus respondents: people who do not answer sincerely and try to finish with as little effort as possible to earn the reward.

Why rare views are inflated

A bogus respondent who clicks yes without reading adds false "yes" answers to many questions. When the true share of people holding a view is small, a handful of false answers can multiply it. Pew notes that studies have found opt-in surveys overestimating rare attitudes and behaviors, and that the effect is concentrated among adults under 30.

A case: the Holocaust poll

A December 2023 opt-in survey reported that 20 percent of US adults under 30 agreed with the statement "The Holocaust is a myth." The finding spread widely in the news. Pew asked the same statement of its own panel, which is recruited by mail with probability-based sampling, in January 2024. It found 3 percent. A rare attitude and a young age group are the two conditions where bogus respondents do the most damage.

071 min

Survey bias from question wording

The words of a question can move answers before the respondent has thought about the topic. Pew Research Center has run experiments comparing the word welfare with the phrase assistance to the poor. Support for expanding the programme was much greater under the second name. The policy was the same, and only the label changed. This is close to the framing effect, where the same choice described differently draws a different response.

Some wording faults appear again and again:

  • A leading question signals the expected answer. How helpful was our excellent support team? assumes support was excellent.
  • A loaded question hides an assumption inside it. What do you dislike about the checkout? assumes the respondent dislikes something.
  • A double-barreled question asks two things at once. Pew's example is asking about confidence in a president to handle domestic and foreign policy. One answer cannot be read, because it may refer to either half.
  • A vague question lets each respondent choose a meaning. Do you use the product regularly? means daily to one person and monthly to another.
  • An unbalanced scale offers more positive than negative options, which moves answers upward.

The fix in each case is to ask one concrete thing, in plain words, with balanced choices.

081 min

How to test whether wording causes survey bias

A wording problem is hard to see from the inside, because the writer already knows what the question was meant to say. Pew's method is to write two versions of a question and give each version to half of the sample. Respondents are assigned to a version at random, so the two groups can be treated as essentially identical. A clear difference between the groups' answers then comes from the wording.

A product team can use the same idea on its own survey. It shows version A to a random half of its users and version B to the other half, and compares the answers. If they differ, the question is steering people, and the team should report the result of both wordings or choose the neutral one.

091 min

Survey bias from order and layout

Wording is not the only thing a respondent reads. Survey bias also comes from where a question or an answer option sits. Earlier questions change how later ones are understood. Pew describes two patterns. A contrast effect makes answers more different from each other because of their order. An assimilation effect makes them more alike. Asking about happiness in marriage just before asking about overall happiness can cause a contrast effect, because people tend to leave the marriage out of their overall rating.

The order of answer options matters as well. Research cited by Pew suggests that in telephone surveys people more often choose options heard later in a list. In surveys people read themselves, they more often choose options near the top. Pew's response is to randomize the order of options, so the effect is spread across the sample instead of always favouring the same option.

Length adds to this. A long survey tires people, and tired people settle for the first acceptable answer, such as ticking the same box down a column. Researchers call this satisficing. Forcing an answer to every question makes uncertain people invent one. People who answer the same survey again and again in a panel can also change how they answer, which is called panel conditioning. A short survey with a prefer not to answer choice produces cleaner data.

102 min

Survey bias from how people answer

Even a well-chosen sample and a clear question leave one more source of survey bias: what people decide to say. Researchers call this response bias.

The best known form is social desirability bias. People tend to understate how much they drink or take drugs, cheat on tax and hold racial bias, and to overstate church attendance, donations and voting. The effect can be greater when an interviewer is present than when people answer on paper or on screen. Offering answer options that make an honest answer feel ordinary, such as several choices for infrequent church attendance, reduces it.

Acquiescence bias is the tendency to agree with a statement whatever it says. Pew reports that people with less education and less information tend to agree more, and that the tendency is stronger when an interviewer is present. A better design offers a choice between two statements instead of asking people to agree or disagree.

Other patterns are smaller but common:

  • Extreme responding, where a person picks the end points of every scale, and neutral responding, where a person picks the middle every time.
  • Recall bias, where people reconstruct past events instead of retrieving them, so short and recent time frames give better answers.
  • Demand characteristics, where respondents guess what the researcher wants and provide it. This is the same mechanism as the Hawthorne effect.
  • Interviewer bias, where the tone or expression of the person asking changes the reply.

112 min

What the 1936 survey bias really was

The 1936 poll is often told as a simple lesson about telephone lists. The record is more mixed than that.

The frame, the replies, or both

Gallup blamed the sampling frame in 1938, because it favoured richer households. In 1976, Bryson argued that the frame did not explain the errors and blamed nonresponse. Later reviews used a Gallup survey from 1937 that asked people whether they had taken part. They found that telephone and car owners also supported Roosevelt, so the low response rate and the flawed frame together produced the wrong forecast. Blaming one cause alone is a misreading.

Weighting by past vote

Lohr and Brick added a different point in 2017. The Digest ballot asked how people had voted in 1932. Had the magazine weighted its results by those 1932 answers, it would have forecast an electoral-vote majority for Roosevelt. Weighting gives more influence to respondents from groups that replied less. The choice of weighting model also adds its own uncertainty, which the usual sampling margin does not capture.

The warnings were public

A commentator writing in 1936 noted that less than a quarter of the ballots had been returned and called that quarter self-chosen. The Digest's editorial board strongly opposed weighting, while The New York Times published articles arguing for it. The 24 percent reply rate was also higher than many of today's polls achieve, so a high rate alone never guaranteed accuracy.

121 min

How survey bias shows up in product work

The Digest's problem recurs at a smaller scale whenever a team surveys its users. Take a product manager at a subscription app adds an in-app rating prompt that appears after a user finishes a task. Only active users ever see it, and mostly people with a strong opinion tap it. The average score then describes active, opinionated users. It says nothing about people who never finished onboarding or stopped opening the app.

The decision this changes is concrete. The manager should not present the score as the opinion of all customers. They should report who saw the prompt, how many answered, and who is missing. If the team needs the views of churned users, it must reach them through another channel, such as an email to former customers. They should also check the result against something that is not a survey, such as usage data.

The same applies to a feature vote on a newsletter list, an employee satisfaction form sent by email, and a survey run at an event. Each has a list, a reply pattern, and a wording. Each can be asked the three questions in the next section.

131 min

How to reduce survey bias

Reducing survey bias means working on each source in turn. Start with the population, then the list, then the replies, then the questions.

  1. Define the population before writing a question, and sample it by chance or by strata.
  2. Offer more than one channel, so people without a given device or phone habit can still answer.
  3. Follow up with people who did not reply, and keep the survey short.
  4. Write each question as one concrete, neutral idea, with balanced answer options.
  5. Randomize the order of answer options, and rotate question order where it matters.
  6. For sensitive topics, promise anonymity and offer answer options that make honesty feel ordinary.
  7. Pilot the survey with a small group, and use a split sample to test any wording you doubt.
  8. Weight results to known population totals if the sample is unbalanced, and say that weighting adds uncertainty. Weighting cannot repair a sample that has no one from a group at all.
  9. Publish who was asked, how many answered, and the exact wording.

Before trusting any survey result, ask three questions. Who could have been asked? Who actually answered? How was the question put? A result that cannot answer all three is a rough guide at best.

141 min

Survey bias vs related ideas

Survey bias overlaps with several terms, and the deciding fact is whether the error is systematic and where in the survey it arises.

TermWhat it coversDeciding fact
Sampling errorRandom difference between a sample and the populationIt shrinks as the sample grows, and the margin of error reports it
Selection biasAny non-random way people enter the sampleIt is one source of survey bias, covering who is asked and who answers
Response biasInaccurate answers from people who did respondIt concerns what people say, not who they are
Researcher biasChoices by the analyst, such as favouring evidence that supports a hypothesis or a sponsorIt sits in the analysis, not in the respondents

Survey bias is the wider term. It contains selection bias, nonresponse bias, response bias, and the effects of wording and order.

?4 questions

Questions people ask

Does a bigger sample reduce survey bias?

No. A bigger sample reduces random error only. If the list or the replies are skewed, more responses repeat the same skew with more confidence, as the 2.4 million ballots in 1936 showed.

What is the difference between response bias and nonresponse bias?

Response bias concerns the answers people give, such as saying what looks good. Nonresponse bias concerns the people who give no answer at all, who may differ from the people who do answer.

How can you tell whether a survey is biased?

Compare its results with a trusted benchmark, such as a census or government survey, and check who was asked, who answered and how the questions were worded. Large gaps on the same question suggest bias.

Can weighting fix survey bias?

It can reduce it, but only for the traits used in the weighting, and the choice of weights adds its own uncertainty. In 1936, weighting by earlier vote would have changed the predicted winner.

§6 sources

Sources

Show all 6 sources
  1. Lohr, S. L. and Brick, J. M. (2017). Roosevelt Predicted to Win: Revisiting the 1936 Literary Digest Poll. Statistics, Politics and Policy 8(1).

  2. Pew Research Center. Writing Survey Questions.

  3. Pew Research Center (2012). Assessing the Representativeness of Public Opinion Surveys.

  4. Pew Research Center (2024). Online opt-in polls can produce misleading results, especially for young people and Hispanic adults.

  5. StatTrek. Bias in Survey Sampling.

  6. Qualtrics. What is Response Bias and How Can You Avoid It?

Keep reading

More from Research

All of Research
All of Research

Missing a concept?

Tell us what to write next. We’ll email you when it’s live.

Several? Separate them with commas.

Only to tell you when it’s live.

How we use your email: privacy.