Voluntary bias

Also called Voluntary response bias, Volunteer Bias. Also in Psychology, Product

Quick answer

Voluntary bias is a distortion in research results that appears when people decide for themselves whether to take part. Volunteers usually differ from people who stay out, because they feel strongly about the topic or have more time, interest or better health. The results then describe the volunteers, not the wider group the study meant to learn about.

~20 sec read

012 min

How voluntary bias appears in a feedback pop-up

Voluntary bias often starts with a feedback pop-up. Picture a product manager at a subscription app who wants to know how users feel. They add a pop-up with one question: how likely are you to recommend this app, on a scale from 0 to 10? Nobody has to answer. After a month there are many replies and a clear average, and the manager tells the leadership team that this is what users think.

The users who love the app tap 10, the users who are angry tap 0, and everyone in the middle closes the pop-up. A user who thinks the app is fine has no reason to spend a minute on a reply. So the average describes the people who felt strongly enough to answer, and most users are not among them.

The manager did nothing careless. The method has a flaw from the start: nobody chose the sample. The users chose themselves, and that flaw is what this article is about. Waiting for more replies does not fix it, because every new reply still comes from a user who decided to give one.

Teams use this design anyway, and for good reasons. It costs almost nothing, it runs by itself and it reaches a very large number of people. The real question is what the answers can safely be used for. That depends on how far the volunteers differ from everyone else, and researchers have measured that gap in several settings.

pop-up0 to 10100
Look at who is missing from the replies. The faded figures with dashed outlines are the users in the middle, and they did not answer.

022 min

Why voluntary bias happens

Voluntary bias happens when the people who choose to take part are not a small copy of the group the study is about. Two terms make this exact. The target population is everyone the study wants to learn about, here all users of the app. The sample is the part of that group whose answers are actually collected.

In good sampling, chance decides who enters the sample. Each person in the population has a known probability of being picked, so the sample tends to look like the population. This is called probability sampling. A voluntary sample is a non-probability sample, because the people decide and not chance. Call-in radio polls and open online polls work this way. So does a convenience sample, such as whoever happens to visit a page, which has the same weakness.

A person's decision is a problem when the reason for deciding is linked to the thing being measured. Three links come up again and again.

  • Feeling. A person with a strong opinion, good or bad, gets more out of replying than a person with no opinion.
  • Time and interest. People with spare time, or with a personal interest in the topic, are more likely to join.
  • Circumstances. In health research, volunteers are often healthier and better off than the people who were invited. The next section measures this.

The size of the bias depends on how strongly the decision to join is linked to the answer. If the link is weak, volunteers answer much like everyone else and the bias is small. If the link is strong, as with a satisfaction question and a furious user, the bias is large. A bigger sample cannot repair this, because a large voluntary sample is built in the same way as a small one. A small random sample can be safer than a large voluntary one. Researchers describe the underlying risk as a threat to external validity, which means the findings may not apply beyond the people who were studied.

Three names for one problem

Survey guides call it voluntary response bias. Trial reports call it volunteer bias. Many methods papers call it self-selection bias, and a 2008 paper from Statistics Netherlands on web surveys warns that self-selection leads to biased estimates. All of these describe people choosing themselves into the data.

032 min

Voluntary bias in the UK Biobank

The UK Biobank shows how voluntary bias can remain in a study that was built with great care. It collects body measurements, blood samples and health records from volunteers so that researchers can look for the causes of disease. Its organisers invited about 9.2 million people aged 40 to 69 who lived near an assessment centre, and 5.5% of them took part in the first visit.

In 2017 a team led by Anna Fry compared those volunteers with the general population of the same age. Among men aged 45 to 54, 25.6% of the volunteers were obese, against 31.5% of men in the general population. The study has about half a million participants, and it still differs from the people who were invited.

Volunteers do not always hold strong opinions

Many guides describe voluntary bias as a surplus of angry and delighted people. The Biobank shows a different pattern. A study that asks for time, a visit and a blood sample attracts people who are more careful about their health than the people who say no. Researchers call this the healthy volunteer effect: people who sign up for research tend to be more health-conscious than people who do not. So the direction of the bias depends on the topic. In a satisfaction survey it pushes answers to the two ends of the scale, and in a health study it pushes them toward good health.

UK Biobankinvited: 9.2 milliontook part: 5.5%
Look at the length of the second bar against the first. Only a small part of the invited group took part.

041 min

Voluntary bias in a Dutch health survey

A second case isolates voluntary bias by changing mainly one thing: how people were recruited. Dutch researchers led by Kei Long Cheung used two health surveys of adolescents and compared them in 2017. In the Twente region, adolescents answered during class time at school, so almost all of them took part. In the IJsselland region, adolescents received an invitation by post and chose whether to reply.

The two samples gave different pictures. The authors describe the voluntary sample's results as an underestimate of how common health behaviours were, and they report that alcohol use came out as much as four times lower. The voluntary group also reported better mental health and more positive school experiences, and the gap stayed after the authors controlled for demographic differences.

Two caveats limit what this comparison shows. The surveys ran in two different regions, so the comparison is not a perfect experiment. The authors also call the problem non-response bias, a close relative covered later in this article. Still, the direction matches the Biobank: the adolescents who chose to reply looked healthier and more positive than the adolescents who had no choice.

The two cases teach different things. The Biobank shows the gap against national figures for a very large study. The Dutch surveys show what a nearly complete sample would have said, which is the benchmark most product teams never have.

051 min

How voluntary bias grows during a study

Voluntary bias does not stop at sign-up, because staying in a study is a second choice. Most research asks volunteers to come back, answer more questions or give samples, and each request is another choice that removes some people.

A team led by Sue Jordan documented this in a trial of probiotic supplements for childhood atopy, which is a tendency toward allergic disease. Compared with mothers in the same area who gave birth in the same period, the 454 recruits were less deprived. As the trial went on, the most deprived mothers made up a smaller share of those still taking part.

Researchers call dropout of this kind attrition, and attrition that is not random is selective attrition. The group becomes a little less like the population at each stage: invitation, sign-up, return visits and sample donation. At the end, a finding is known to hold for the volunteers who stayed, and the researchers still have to show that it holds beyond them.

062 min

Voluntary bias in product and design work

Product and design teams meet voluntary bias in more places than the pop-up.

Ratings and reviews

A star rating comes only from the customers who decided to write a review. A 2020 study by Verena Schoenmüller, Oded Netzer and Florian Stahl of more than 280 million reviews on 25 platforms found that people with extreme opinions are more likely to write a review, and that this is one reason review scores sit mostly at the two ends of the scale. A rating therefore describes the reviewers more than the average buyer.

Surveys, betas and panels

An in-app survey reaches only people who open the app, so users who have already left never see it. A beta programme or a research panel holds people who volunteered, and these users are likely to be more engaged than a typical customer. Their feedback is real, but it comes from the most engaged part of the user base. A survey link shared in a community or group chat has a further problem, because the group that shares it can decide the result.

Feature adoption

A related form appears when a team measures an optional feature. Users who switch it on are not a random group. If they stay longer than other users, the feature may be a sign of an engaged user and not the reason that user stays. Assigning users to the feature at random, as in A/B testing, removes the choice and so removes the problem.

When volunteers are paid

Paying volunteers brings a different problem. Pew Research Center, a research organisation that studies survey methods, describes online opt-in surveys that recruit through ads and social media. Studies cited by Pew Research Center have found large errors in online opt-in surveys because of bogus respondents, who answer without sincerity in order to collect rewards. Pew adds that these respondents can make opt-in surveys overestimate rare views, and that young adults are a subgroup they often enter.

072 min

How to spot and reduce voluntary bias

Voluntary bias is easier to reduce before data collection than to repair afterwards, but it can often be measured.

Spot it

Four checks take little time.

  1. Compare the replies with a known benchmark.

    Fry's team compared Biobank volunteers with national figures. A product team can compare the users who replied with the full user list on plan, region and time since sign-up.

  2. Look at the shape of the scores.

    Many scores at 9 and 10, many at 0 and 1, and few in the middle are a sign that strong feelings are choosing to reply.

  3. Track the reply rate by segment.

    If one group of users answers far more often than another, the replies over-represent that group.

  4. Check against behaviour.

    If users who gave a high rating cancel as often as everyone else, the rating does not predict what users do.

Reduce it

  1. Choose people instead of waiting for volunteers.

    Draw a random set of users from the full user list and contact them directly. Stratified sampling goes one step further: it splits users into groups, such as plan type, and draws at random from each group. Survey design covers how to build such a plan.

  2. Follow up with people who did not reply.

    A reminder through a second channel, or a one-question message asking why someone declined, brings back part of that group.

  3. Make replying cheap.

    Short surveys with plain questions, tested first on a few people, take away the effort that keeps neutral people out. Anonymity helps where honest answers are sensitive.

  4. Check why people joined.

    A screening question about motivation lets you compare keen volunteers with the rest. Give the same small reward to every invited person, and still check the quality of the answers, as the paid-panel problem above shows.

What weighting can and cannot do

Weighting gives replies from under-represented groups more influence, so that the sample matches the known shares of the population. In the probiotic trial, weighting the data to account for volunteer bias changed one important result. Weighting has a limit, though. It corrects differences that were measured, such as age or region. It cannot correct a difference that was not measured, such as how strongly users feel. A sensitivity analysis helps there: try several assumptions about the users who did not reply and see whether the conclusion changes.

Say what the data covers

The last step is wording. A report that names its volunteers is honest about the limit.

Among the users who chose to answer, [X]% gave a score of 9 or 10. This survey cannot show how the users who did not answer feel.

082 min

When voluntary bias matters less

Voluntary bias matters most when the goal is to count something, and less when the goal is to compare. It does the most damage when the question is how common something is or what most people think. It does less damage when the question is how two things are related, or when the goal is to discover problems and not to measure them.

The Biobank authors draw this line themselves. They write that the study cannot give reliable rates of disease, yet it can still support valid conclusions about the links between exposures and health conditions. Kenneth Rothman, John Gallacher and Elizabeth Hatch made the same point in 2013 for science in general: representativeness may be essential for opinion polls or public-health uses, but they do not see it as a sensible goal for a scientific study.

In product work, the same logic gives a rule. A volunteer who describes a confusing checkout has found a real problem, whether or not other users share the view. What the volunteer cannot tell you is how many users share it. Use volunteers to find problems, and use a random sample to size them.

When selection changes the links

The exception to that rule is a link between two traits that both affect who volunteers. Simulations by Marcus Munafò and colleagues show that even modest differences in who joins or stays in a study can produce misleading estimates of the links between two traits. The reason is collider bias. A collider is a variable that two other variables both influence, and taking part in a study can be one. Say a usability study recruits volunteers, and both patient users and tech-confident users are more likely to volunteer. Among the volunteers, the two traits can then show a link that does not exist across all users.

092 min

Voluntary bias vs nearby biases

Voluntary bias is easy to confuse with other biases that also leave people out of the data. The difference is the step at which the people are lost.

A response bias concerns how people answer, for example giving the answer they think is expected, so it affects people who are already in the data. A survivorship bias concerns which cases remain visible after the failures drop out. Voluntary bias concerns who decided to take part at all.

ConceptWho ends up in the dataWhat goes missing
This pageVoluntary biasPeople who did not choose, who often differ in feeling, time or healthWho ends up in the data: People who chose to take part
Not in the library yetNon-response biasPicked people who stayed silentWho ends up in the data: People picked by the researcher who replied
ResearchResponse BiasHonest answers, lost to wording or social pressureWho ends up in the data: Everyone who answered
BusinessSurvivorship BiasCases that failed and left the dataWho ends up in the data: Cases that lasted

The closest relative is non-response bias. There, the researcher picks a sample and some of the chosen people do not reply, so the replies differ from the chosen group. In voluntary bias nobody was picked, because people picked themselves. In practice the two overlap, and the Dutch authors call their postal result non-response bias even though replying was voluntary. The practical difference is where the fix starts. For non-response, chase the people who were picked. For voluntary bias, change the way people enter the sample.

?4 questions

Questions people ask

Is voluntary bias the same as self-selection bias?

Nearly. Both describe people choosing themselves into the data. Voluntary bias is the usual name when people volunteer to answer or join a study, while self-selection bias also covers choosing a program or feature before being measured.

What is a voluntary response sample?

A voluntary response sample is a group made of people who chose to respond, such as callers to a radio poll or visitors who click a survey link. The researcher picked nobody in it.

Does offering a reward remove voluntary bias?

No. A reward can widen who replies, but people still choose whether to take it. Paid online panels also draw respondents who rush for the reward, so incentives need checks on answer quality.

Does voluntary bias affect A/B tests?

Not when chance assigns users to each version, because nobody chooses a group. It can affect a test where users pick their own version, such as opting in to a beta, since the groups differ before the test starts.

§8 sources

Sources

Show all 8 sources
  1. Fry, A., Littlejohns, T. J., Sudlow, C., et al. (2017). Comparison of Sociodemographic and Health-Related Characteristics of UK Biobank Participants With Those of the General Population. American Journal of Epidemiology. pmc.ncbi.nlm.nih.gov

  2. Cheung, K. L., ten Klooster, P. M., Smit, C., et al. (2017). The impact of non-response bias due to sampling in public health studies: A comparison of voluntary versus mandatory recruitment in a Dutch national survey on adolescent health. BMC Public Health. pmc.ncbi.nlm.nih.gov

  3. Jordan, S., Watkins, A., Storey, M., et al. (2013). Volunteer Bias in Recruitment, Retention, and Blood Sample Donation in a Randomised Controlled Trial Involving Mothers and Their Children at Six Months and Two Years: A Longitudinal Analysis. PLoS ONE. journals.plos.org

  4. Schoenmueller, V., Netzer, O. and Stahl, F. (2020). The Polarity of Online Reviews: Prevalence, Drivers and Implications. Journal of Marketing Research, abstract on the Bocconi University Department of Marketing site. marketing.unibocconi.eu

  5. Munafò, M. R., Tilling, K., Taylor, A. E., Evans, D. M. and Davey Smith, G. Collider scope: when selection bias can substantially influence observed associations. Preprint of the paper in the International Journal of Epidemiology. biorxiv.org

  6. Rothman, K. J., Gallacher, J. E. and Hatch, E. E. (2013). Why representativeness should be avoided. International Journal of Epidemiology 42(4), 1012-1014, as listed by the authors' institute. rti.org

  7. Mercer, A., Kennedy, C. and Keeter, S. (2024). Online opt-in polls can produce misleading results, especially for young people and Hispanic adults. Pew Research Center. pewresearch.org

  8. Statistics Netherlands (2008). How accurate are self-selection web surveys? Discussion paper. cbs.nl

Keep reading

More from Research

All of Research
All of Research

Missing a concept?

Tell us what to write next. We’ll email you when it’s live.

Several? Separate them with commas.

Only to tell you when it’s live.

How we use your email: privacy.