011 min
Margin of error at a glance
- What it is: Half the width of the interval around a survey estimate. A result of 48% with a margin of 3 points means the true value is probably between 45% and 51%.
- Origin: It is half the width of a confidence interval. Jerzy Neyman gave the first thorough general account of confidence intervals in 1937.
- Confused with: Confidence level. The margin is how wide the range is. The level is how often such ranges contain the true value.
- Watch for: The margin for a lead between two candidates is about twice the reported margin. A margin of 3 points becomes about 6.
021 min
Why margin of error matters
A survey asks some people and reports a number for everyone. That number is always a little off, because a different set of people would have given a slightly different answer. The margin of error says how far off to expect it to be.
This changes decisions a team really makes. Say a PM reads that 52% of surveyed users prefer the new onboarding flow and ships it. If the margin of error is 5 points, the true preference could be 47%, which is a minority. Without the margin, 52% looks like a clear win.
The opposite mistake also costs money. A team sees a result move from 48% to 51% between two monthly surveys and calls a meeting about the trend. With a margin of 3 points in each survey, that change is well inside what random sampling alone produces. The meeting would discuss a change that may not have happened.
The margin tells you which differences deserve attention and which do not. It also tells you what it cannot tell you, which is covered below.
032 min
What margin of error actually measures
A survey talks to a sample, a small group chosen from a larger population. The result for the sample will almost never match the result for the whole population exactly. The gap between the two is called sampling error.
The margin of error describes how large that gap can reasonably be. Andrew Mercer of Pew Research Center explains it this way: a margin of error of plus or minus 3 percentage points at the 95% confidence level means that if the same survey were run 100 times, the result would be within 3 percentage points of the true population value 95 of those times.
Three quantities set the size of the margin:
- Sample size (n): the number of people who answered. More answers give a smaller margin.
- The result itself (p): the share who gave a particular answer. The margin is largest when p is 50% and smaller when the answer is lopsided.
- Confidence level: how often the range should contain the true value. A higher level gives a wider margin.
For a yes-or-no question in a simple random sample, the standard formula is:
margin of error = z × √( p × (1 − p) ÷ n )
Here z is a number tied to the confidence level. It is about 1.96 for 95% confidence. The part under the square root, divided by n and rooted, is the standard error, which measures how much the result would vary from one sample to the next. The margin is the standard error times z.
The margin is half the width of the confidence interval. A result of 48% with a margin of 3 points gives an interval from 45% to 51%.
041 min
Where the term comes from
The margin of error is the half-width of a confidence interval, so its history is the history of that method. The main ideas of confidence intervals were developed in the early 1930s. The first thorough and general account was given by the statistician Jerzy Neyman in 1937, in a paper titled "Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability" in the Philosophical Transactions of the Royal Society.
Neyman showed how to build a range from sample data that contains the true value with a stated long-run frequency. Survey reports later reduced that range to a single number, the half-width, printed beside a result.
This article does not name who first used the phrase "margin of error". The sources checked here use it as established terminology without saying who coined it, and a claim about its first use would be unverified. The Census Bureau and Pew Research Center define it in the same way: a measure of sampling error.
052 min
A worked example: a survey of 400 users
All numbers in this section are illustrative. They are not from a real product.
A product team emails a survey to its users and gets 400 answers. 40% of respondents say they would use a proposed feature. What is the margin of error?
- Standard error = √( 0.40 × 0.60 ÷ 400 ) = √0.0006 = 0.0245, or 2.45 percentage points.
- Margin of error at 95% confidence = 1.96 × 2.45 = 4.8 percentage points.
- The interval runs from about 35% to 45%.
So the team can say that 40% of respondents would use the feature and that the true share among all users is probably between 35% and 45%. A decision that depends on the true share being above 50% is not supported by this survey.
Now change one input at a time and keep the rest.
| Responses | Share saying yes | Standard error | Margin of error (95%) |
|---|---|---|---|
| 100 | 40% | 4.9 points | 9.6 points |
| 400 | 40% | 2.45 points | 4.8 points |
| 400 | 50% | 2.5 points | 4.9 points |
| 1,600 | 40% | 1.2 points | 2.4 points |
Two things follow. First, the result barely changes the margin: moving from 40% to 50% adds only 0.1 point. Because of this, pollsters use p = 50% to report a safe maximum. Second, sample size matters, but with diminishing returns. Going from 400 to 1,600 answers, which is four times the work, only halves the margin from 4.8 to 2.4 points.
The usual benchmark follows from the same formula. Pew Research Center reports that a simple random sample of 1,067 cases has a margin of error of plus or minus 3 percentage points for estimates of overall support for individual candidates. To plan how many people to ask, see sampling.
062 min
Reading a lead or a change against the margin of error
The reported margin of error applies to a single percentage. Most questions people ask of a survey are about a difference between two numbers, and the margin for a difference is larger.
Pew Research Center gives the example of a poll with a margin of 3 points for each candidate. The Republican candidate has 48% and leads by 5 points. The margin for the difference between the two candidates is about twice the individual margin, so about 6 points. The true lead could be anywhere from 1 point behind to 11 points ahead. A lead of 6 points or more would be needed to be confident it is not just sampling error.
The margin for a change between two surveys is larger still. In Pew's example the lead moves from 5 points in Poll A to 8 points in Poll B, a shift of 3 points. The margin of error for that shift is plus or minus 8 points. The observed shift is consistent with anything from a 5-point decline to an 11-point increase.
| What you compare | Rough margin when each result has ±3 |
|---|---|
| One result against a fixed value | ±3 points |
| Two results in the same survey (a lead) | about ±6 points |
| The same lead in two different surveys (a change) | about ±8 points |
When the figure in a report is a gap or a trend, ask for the margin for that gap or trend. If the report gives only the margin for a single result, assume the real uncertainty is two to three times larger.
071 min
Why a subgroup has a larger margin of error
The margin printed for a survey applies to the whole sample. Reports often show a smaller group, such as people under 30 or users of one plan. That group has fewer answers, so its margin is larger.
Pew Research Center works through the case of a poll of 1,067 people. If a group makes up about 15% of the population, it supplies about 160 answers. Its margin of error is about plus or minus 8 percentage points for one candidate and about plus or minus 16 points for the gap between two candidates. That is more than double the margin of the full sample.
For a product team this is a common mistake. As an illustration, a survey of 1,000 customers has a margin near 3 points overall. A chart that splits the same answers into six segments gives each segment a margin of 7 to 12 points or more. A difference of 5 points between two segments is then too small to read as anything. Ask how many people are in each segment before trusting a segment chart.
082 min
What the margin of error does not cover
The margin of error covers one source of error only, which is random variation from asking a sample instead of everyone. It does not cover who was left out, who chose not to answer, or how the questions were worded.
The Census Bureau states this in its guidance on the American Community Survey. Besides sampling error, other errors can come from problems in the sampling frame or the questionnaire, mistakes in reporting or coding, weighting issues, and interviewer bias or nonresponse bias. Nonresponse bias arises when respondents differ in meaningful ways from people who did not respond.
The 1936 Literary Digest poll is the standard demonstration. It mailed ballots to lists of telephone owners, car owners and its own readers. Almost 2.4 million ballots came back, according to Sharon Lohr and J. Michael Brick (2017), who reanalysed the poll. The poll predicted that Alfred Landon would win, and Franklin Roosevelt won in a landslide.
Lohr and Brick calculate that the sampling margins of error for the unweighted 1936 estimates were between 0.18 percentage points for New York and 2.3 points for Nevada, with a median of 0.6 points. They conclude that almost all of the uncertainty comes from nonsampling error. A huge sample gave a tiny margin of error and a very wrong answer.
So a small margin means the result is precise. It does not mean the result is accurate. Precision describes how tightly repeated results would cluster. Accuracy describes whether they cluster around the true value. Bias from who answers is covered in survey bias and voluntary bias.
Pew Research Center names one more adjustment. Pollsters weight their data so the sample matches the population. Weighting reduces bias, but it also increases variability, which statisticians call the design effect. A pollster who reports the simple formula margin without the design effect claims more precision than the survey has.
091 min
How margin of error shows up in product and research work
The same logic appears wherever a team measures a sample and acts on the result.
A PM reviewing a satisfaction survey. Illustratively, the score moved from 71% to 74% after a redesign. If each survey has about 400 responses, each margin is near 4.5 points, so the 3-point rise is inside the range of random variation. The honest summary is that the score did not clearly change.
A researcher reporting to a stakeholder. The stakeholder asks whether 52% preference means the new design wins. The reply that serves the stakeholder is a range: between roughly 47% and 57%, so the survey cannot rule out a minority preference.
A founder reading public data. The Census Bureau's American Community Survey publishes a margin of error with every estimate, at a 90% confidence level. For example, the 2015 one-year estimate for one-person households in Colorado was 564,757 with a margin of 10,127, giving a 90% interval from 554,630 to 574,884. Anyone comparing such figures with a poll needs to know which confidence level each uses, because a 90% margin is narrower than a 95% margin on the same data.
A team running an A/B test on conversion rates. The same arithmetic applies to each variant's rate. See A/B testing for how test design handles the comparison between two variants, which has the same larger-margin problem as a lead in a poll.
101 min
How to use margin of error
Use these checks in order when you read or report a survey result.
Find the sample size and the confidence level.
If a report gives a margin without them, you cannot judge it. Most polls use 95%. The American Community Survey uses 90%.
Match the margin to the claim.
A single percentage uses the printed margin. A gap between two answers needs about double. A change between two surveys needs more.
Check the group size.
If the claim is about a subgroup, find how many people it holds and recompute, or ask the survey owner.
Report the range, not only the point.
Write "40%, give or take 4.8 points" or "between 35% and 45%".
Ask what the margin leaves out.
Who could not answer, who chose not to, and what was the response rate. If those answers are poor, the margin overstates how much you know.
Look for repeats.
Pew Research Center notes that of 590 national polls listed for the 2012 U.S. presidential race, about 5%, or roughly 30, would be expected to miss by more than the margin of error. One surprising poll should wait for confirmation from others.
111 min
When margin of error does not apply
The standard formula assumes a probability sample, where every member of the population had a known chance of being picked. It also assumes the result is a share or an average with a roughly normal spread across repeated samples.
It does not apply, or does not mean what it seems to, in these cases:
- Convenience or opt-in samples. A survey link shared on social media, or a feedback widget that only some users click, has no known selection chance. A margin printed under such a result gives a false impression of rigour.
- Very small samples or very lopsided answers. The normal approximation behind the formula works poorly when almost everyone gives the same answer, or when n is only a few dozen. Other interval methods exist for these cases.
- Whole populations. If you surveyed every user in your database and all of them answered, there is no sampling error to measure.
- Combined data. Lohr and Brick note that when poll aggregators combine data and estimates, the resulting sampling errors understate the accuracy of the estimates.
When the formula does not apply, say so. A number with no margin is more honest than a number with a margin that does not mean anything.
121 min
Margin of error vs. confidence level and standard error
Three terms travel together and are often swapped.
- Margin of error: the half-width of the range, in the unit of the result. It answers how far off the result could be.
- Confidence level: how often a range built this way contains the true value. It answers how sure to be.
- Standard error: the spread of the result across repeated samples. The margin of error is the standard error times a multiplier that depends on the confidence level.
The deciding fact is that the margin changes when the confidence level changes. For a sample of 1,000 and a 50% result, the margin is about 2.6 points at 90% confidence, 3.1 at 95% and 4.1 at 99%. A margin quoted without its confidence level cannot be compared with any other margin.
The margin of error is also not a probability that the result is wrong. A margin of 3 points at 95% does not say there is a 5% chance the result is wrong by any amount. It says that 5% of repeated surveys of this kind would land outside the range. Both depend on variance, the spread of the answers themselves: the more the answers differ from one person to the next, the larger the standard error.
?7 questions
Questions people ask
What is margin of error?
How do you calculate margin of error?
What sample size gives a margin of error of 3 points?
Is a bigger sample always better?
What does outside the margin of error mean?
What is the difference between margin of error and confidence interval?
Why do surveys report a 95% confidence level?
§6 sources
Sources
Show all 6 sourcesHide sources
Mercer, A. (2016). 5 key things to know about the margin of error in election polls. Pew Research Center.
U.S. Census Bureau (2020). Understanding Error and Determining Statistical Significance, American Community Survey General Handbook, chapter 7.
Lohr, S. L. and Brick, J. M. (2017). Roosevelt Predicted to Win: Revisiting the 1936 Literary Digest Poll. Statistics, Politics and Policy 8(1), 65-84.
Wikipedia. Margin of error.
Neyman, J. (1937). Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability. Philosophical Transactions of the Royal Society of London A 236(767), 333-380.
Wikipedia. Confidence interval.





