Margin of Error

Also called Margin of sampling error, Sampling error margin. Also in Business

Quick answer

Margin of error is the range, in percentage points, around a survey result within which the true value for the whole population is expected to fall, at a stated confidence level, usually 95%. It measures only random sampling error, which shrinks as more people respond. It does not measure bias from who was asked or how they were asked.

~20 sec read

011 min

Margin of error at a glance

  • What it is: Half the width of the interval around a survey estimate. A result of 48% with a margin of 3 points means the true value is probably between 45% and 51%.
  • Origin: It is half the width of a confidence interval. Jerzy Neyman gave the first thorough general account of confidence intervals in 1937.
  • Confused with: Confidence level. The margin is how wide the range is. The level is how often such ranges contain the true value.
  • Watch for: The margin for a lead between two candidates is about twice the reported margin. A margin of 3 points becomes about 6.

021 min

Why margin of error matters

A survey asks some people and reports a number for everyone. That number is always a little off, because a different set of people would have given a slightly different answer. The margin of error says how far off to expect it to be.

This changes decisions a team really makes. Say a PM reads that 52% of surveyed users prefer the new onboarding flow and ships it. If the margin of error is 5 points, the true preference could be 47%, which is a minority. Without the margin, 52% looks like a clear win.

The opposite mistake also costs money. A team sees a result move from 48% to 51% between two monthly surveys and calls a meeting about the trend. With a margin of 3 points in each survey, that change is well inside what random sampling alone produces. The meeting would discuss a change that may not have happened.

The margin tells you which differences deserve attention and which do not. It also tells you what it cannot tell you, which is covered below.

032 min

What margin of error actually measures

A survey talks to a sample, a small group chosen from a larger population. The result for the sample will almost never match the result for the whole population exactly. The gap between the two is called sampling error.

The margin of error describes how large that gap can reasonably be. Andrew Mercer of Pew Research Center explains it this way: a margin of error of plus or minus 3 percentage points at the 95% confidence level means that if the same survey were run 100 times, the result would be within 3 percentage points of the true population value 95 of those times.

Three quantities set the size of the margin:

  • Sample size (n): the number of people who answered. More answers give a smaller margin.
  • The result itself (p): the share who gave a particular answer. The margin is largest when p is 50% and smaller when the answer is lopsided.
  • Confidence level: how often the range should contain the true value. A higher level gives a wider margin.

For a yes-or-no question in a simple random sample, the standard formula is:

margin of error = z × √( p × (1 − p) ÷ n )

Here z is a number tied to the confidence level. It is about 1.96 for 95% confidence. The part under the square root, divided by n and rooted, is the standard error, which measures how much the result would vary from one sample to the next. The margin is the standard error times z.

The margin is half the width of the confidence interval. A result of 48% with a margin of 3 points gives an interval from 45% to 51%.

041 min

Where the term comes from

The margin of error is the half-width of a confidence interval, so its history is the history of that method. The main ideas of confidence intervals were developed in the early 1930s. The first thorough and general account was given by the statistician Jerzy Neyman in 1937, in a paper titled "Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability" in the Philosophical Transactions of the Royal Society.

Neyman showed how to build a range from sample data that contains the true value with a stated long-run frequency. Survey reports later reduced that range to a single number, the half-width, printed beside a result.

This article does not name who first used the phrase "margin of error". The sources checked here use it as established terminology without saying who coined it, and a claim about its first use would be unverified. The Census Bureau and Pew Research Center define it in the same way: a measure of sampling error.

052 min

A worked example: a survey of 400 users

All numbers in this section are illustrative. They are not from a real product.

A product team emails a survey to its users and gets 400 answers. 40% of respondents say they would use a proposed feature. What is the margin of error?

  1. Standard error = √( 0.40 × 0.60 ÷ 400 ) = √0.0006 = 0.0245, or 2.45 percentage points.
  2. Margin of error at 95% confidence = 1.96 × 2.45 = 4.8 percentage points.
  3. The interval runs from about 35% to 45%.

So the team can say that 40% of respondents would use the feature and that the true share among all users is probably between 35% and 45%. A decision that depends on the true share being above 50% is not supported by this survey.

Now change one input at a time and keep the rest.

ResponsesShare saying yesStandard errorMargin of error (95%)
10040%4.9 points9.6 points
40040%2.45 points4.8 points
40050%2.5 points4.9 points
1,60040%1.2 points2.4 points

Two things follow. First, the result barely changes the margin: moving from 40% to 50% adds only 0.1 point. Because of this, pollsters use p = 50% to report a safe maximum. Second, sample size matters, but with diminishing returns. Going from 400 to 1,600 answers, which is four times the work, only halves the margin from 4.8 to 2.4 points.

The usual benchmark follows from the same formula. Pew Research Center reports that a simple random sample of 1,067 cases has a margin of error of plus or minus 3 percentage points for estimates of overall support for individual candidates. To plan how many people to ask, see sampling.

062 min

Reading a lead or a change against the margin of error

The reported margin of error applies to a single percentage. Most questions people ask of a survey are about a difference between two numbers, and the margin for a difference is larger.

Pew Research Center gives the example of a poll with a margin of 3 points for each candidate. The Republican candidate has 48% and leads by 5 points. The margin for the difference between the two candidates is about twice the individual margin, so about 6 points. The true lead could be anywhere from 1 point behind to 11 points ahead. A lead of 6 points or more would be needed to be confident it is not just sampling error.

The margin for a change between two surveys is larger still. In Pew's example the lead moves from 5 points in Poll A to 8 points in Poll B, a shift of 3 points. The margin of error for that shift is plus or minus 8 points. The observed shift is consistent with anything from a 5-point decline to an 11-point increase.

What you compareRough margin when each result has ±3
One result against a fixed value±3 points
Two results in the same survey (a lead)about ±6 points
The same lead in two different surveys (a change)about ±8 points

When the figure in a report is a gap or a trend, ask for the margin for that gap or trend. If the report gives only the margin for a single result, assume the real uncertainty is two to three times larger.

071 min

Why a subgroup has a larger margin of error

The margin printed for a survey applies to the whole sample. Reports often show a smaller group, such as people under 30 or users of one plan. That group has fewer answers, so its margin is larger.

Pew Research Center works through the case of a poll of 1,067 people. If a group makes up about 15% of the population, it supplies about 160 answers. Its margin of error is about plus or minus 8 percentage points for one candidate and about plus or minus 16 points for the gap between two candidates. That is more than double the margin of the full sample.

For a product team this is a common mistake. As an illustration, a survey of 1,000 customers has a margin near 3 points overall. A chart that splits the same answers into six segments gives each segment a margin of 7 to 12 points or more. A difference of 5 points between two segments is then too small to read as anything. Ask how many people are in each segment before trusting a segment chart.

082 min

What the margin of error does not cover

The margin of error covers one source of error only, which is random variation from asking a sample instead of everyone. It does not cover who was left out, who chose not to answer, or how the questions were worded.

The Census Bureau states this in its guidance on the American Community Survey. Besides sampling error, other errors can come from problems in the sampling frame or the questionnaire, mistakes in reporting or coding, weighting issues, and interviewer bias or nonresponse bias. Nonresponse bias arises when respondents differ in meaningful ways from people who did not respond.

The 1936 Literary Digest poll is the standard demonstration. It mailed ballots to lists of telephone owners, car owners and its own readers. Almost 2.4 million ballots came back, according to Sharon Lohr and J. Michael Brick (2017), who reanalysed the poll. The poll predicted that Alfred Landon would win, and Franklin Roosevelt won in a landslide.

Lohr and Brick calculate that the sampling margins of error for the unweighted 1936 estimates were between 0.18 percentage points for New York and 2.3 points for Nevada, with a median of 0.6 points. They conclude that almost all of the uncertainty comes from nonsampling error. A huge sample gave a tiny margin of error and a very wrong answer.

So a small margin means the result is precise. It does not mean the result is accurate. Precision describes how tightly repeated results would cluster. Accuracy describes whether they cluster around the true value. Bias from who answers is covered in survey bias and voluntary bias.

Pew Research Center names one more adjustment. Pollsters weight their data so the sample matches the population. Weighting reduces bias, but it also increases variability, which statisticians call the design effect. A pollster who reports the simple formula margin without the design effect claims more precision than the survey has.

091 min

How margin of error shows up in product and research work

The same logic appears wherever a team measures a sample and acts on the result.

A PM reviewing a satisfaction survey. Illustratively, the score moved from 71% to 74% after a redesign. If each survey has about 400 responses, each margin is near 4.5 points, so the 3-point rise is inside the range of random variation. The honest summary is that the score did not clearly change.

A researcher reporting to a stakeholder. The stakeholder asks whether 52% preference means the new design wins. The reply that serves the stakeholder is a range: between roughly 47% and 57%, so the survey cannot rule out a minority preference.

A founder reading public data. The Census Bureau's American Community Survey publishes a margin of error with every estimate, at a 90% confidence level. For example, the 2015 one-year estimate for one-person households in Colorado was 564,757 with a margin of 10,127, giving a 90% interval from 554,630 to 574,884. Anyone comparing such figures with a poll needs to know which confidence level each uses, because a 90% margin is narrower than a 95% margin on the same data.

A team running an A/B test on conversion rates. The same arithmetic applies to each variant's rate. See A/B testing for how test design handles the comparison between two variants, which has the same larger-margin problem as a lead in a poll.

101 min

How to use margin of error

Use these checks in order when you read or report a survey result.

  1. Find the sample size and the confidence level.

    If a report gives a margin without them, you cannot judge it. Most polls use 95%. The American Community Survey uses 90%.

  2. Match the margin to the claim.

    A single percentage uses the printed margin. A gap between two answers needs about double. A change between two surveys needs more.

  3. Check the group size.

    If the claim is about a subgroup, find how many people it holds and recompute, or ask the survey owner.

  4. Report the range, not only the point.

    Write "40%, give or take 4.8 points" or "between 35% and 45%".

  5. Ask what the margin leaves out.

    Who could not answer, who chose not to, and what was the response rate. If those answers are poor, the margin overstates how much you know.

  6. Look for repeats.

    Pew Research Center notes that of 590 national polls listed for the 2012 U.S. presidential race, about 5%, or roughly 30, would be expected to miss by more than the margin of error. One surprising poll should wait for confirmation from others.

111 min

When margin of error does not apply

The standard formula assumes a probability sample, where every member of the population had a known chance of being picked. It also assumes the result is a share or an average with a roughly normal spread across repeated samples.

It does not apply, or does not mean what it seems to, in these cases:

  • Convenience or opt-in samples. A survey link shared on social media, or a feedback widget that only some users click, has no known selection chance. A margin printed under such a result gives a false impression of rigour.
  • Very small samples or very lopsided answers. The normal approximation behind the formula works poorly when almost everyone gives the same answer, or when n is only a few dozen. Other interval methods exist for these cases.
  • Whole populations. If you surveyed every user in your database and all of them answered, there is no sampling error to measure.
  • Combined data. Lohr and Brick note that when poll aggregators combine data and estimates, the resulting sampling errors understate the accuracy of the estimates.

When the formula does not apply, say so. A number with no margin is more honest than a number with a margin that does not mean anything.

121 min

Margin of error vs. confidence level and standard error

Three terms travel together and are often swapped.

  • Margin of error: the half-width of the range, in the unit of the result. It answers how far off the result could be.
  • Confidence level: how often a range built this way contains the true value. It answers how sure to be.
  • Standard error: the spread of the result across repeated samples. The margin of error is the standard error times a multiplier that depends on the confidence level.

The deciding fact is that the margin changes when the confidence level changes. For a sample of 1,000 and a 50% result, the margin is about 2.6 points at 90% confidence, 3.1 at 95% and 4.1 at 99%. A margin quoted without its confidence level cannot be compared with any other margin.

The margin of error is also not a probability that the result is wrong. A margin of 3 points at 95% does not say there is a 5% chance the result is wrong by any amount. It says that 5% of repeated surveys of this kind would land outside the range. Both depend on variance, the spread of the answers themselves: the more the answers differ from one person to the next, the larger the standard error.

?7 questions

Questions people ask

What is margin of error?

Margin of error is the amount by which a survey result could differ from the true value for the whole population because only a sample was asked. It is quoted as plus or minus a number of percentage points at a confidence level, usually 95%. A result of 48% with a margin of 3 points means the true value is probably between 45% and 51%.

How do you calculate margin of error?

For a yes-or-no question, multiply 1.96 by the square root of p times (1 minus p) divided by n. Here p is the share giving one answer and n is the number of answers. For 400 answers and a result of 40%, that gives about 4.8 percentage points at 95% confidence.

What sample size gives a margin of error of 3 points?

About 1,067 responses gives a margin of plus or minus 3 percentage points at 95% confidence for a simple random sample, according to Pew Research Center. This assumes the most cautious result of 50%. A smaller margin needs many more responses, because the margin shrinks with the square root of the sample size.

Is a bigger sample always better?

A bigger sample always gives a smaller margin of error, but it does not fix bias. The 1936 Literary Digest poll received almost 2.4 million ballots and still predicted the wrong winner, because the people asked were not representative of voters. Sampling method matters more than size once the sample is moderately large.

What does outside the margin of error mean?

It means a difference is larger than random sampling alone would usually produce. For a lead between two candidates, the lead must be larger than the margin for the difference, which is about twice the printed margin, not just larger than the printed margin. Pew Research Center explains this using a 5-point and 8-point lead.

What is the difference between margin of error and confidence interval?

The confidence interval is the full range, and the margin of error is half its width. A result of 40% with a margin of 4.8 points has a confidence interval of about 35% to 45%. Both refer to the same calculation and the same confidence level.

Why do surveys report a 95% confidence level?

95% is a convention, not a law. It means that about 95 of 100 repeated surveys would give a range that contains the true value. Some sources use other levels: the American Community Survey publishes margins at 90%, which are narrower than 95% margins on the same data.

§6 sources

Sources

Show all 6 sources
  1. Mercer, A. (2016). 5 key things to know about the margin of error in election polls. Pew Research Center.

  2. U.S. Census Bureau (2020). Understanding Error and Determining Statistical Significance, American Community Survey General Handbook, chapter 7.

  3. Lohr, S. L. and Brick, J. M. (2017). Roosevelt Predicted to Win: Revisiting the 1936 Literary Digest Poll. Statistics, Politics and Policy 8(1), 65-84.

  4. Wikipedia. Margin of error.

  5. Neyman, J. (1937). Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability. Philosophical Transactions of the Royal Society of London A 236(767), 333-380.

  6. Wikipedia. Confidence interval.

Keep reading

More from Research

All of Research
All of Research

Missing a concept?

Tell us what to write next. We’ll email you when it’s live.

Several? Separate them with commas.

Only to tell you when it’s live.

How we use your email: privacy.