011 min
The problem volunteer bias creates
A research team wants to know how a whole group behaves, such as all patients with a condition, all customers of a service or all adults in a country. It cannot study everyone, so it studies a sample. A sample describes the group well only if it looks like the group.
When participation is voluntary, the sample is chosen in part by the participants themselves. People who agree to take part can differ from people who refuse in health, education, attitudes, free time or interest in the topic. Any result that depends on those traits is then measured on the wrong people.
The error is hard to see. The study can be well designed, carefully run and correctly analysed, and still describe volunteers and not the population. A larger sample does not fix it, because the extra volunteers differ from the group in the same direction.
021 min
How volunteer bias works
The Catalogue of Bias, which is maintained by the Catalogue of Bias Collaboration, describes the mechanism in one line: "When the sample consists of volunteers, the risk is that they are not representative of the general population."
Three points explain how it grows.
It can arise at every stage. The Catalogue says "Volunteer bias can occur at all stages of the trial from recruitment, retention through to follow-up." Someone may agree to join and then drop out. The people who stay differ from those who leave.
The differences are not only demographic. They can include attitudes toward the study and the institutions behind it. They can also relate to the topic: volunteers may be less willing to come forward for studies about behaviour regarded as less socially acceptable.
Direction is hard to predict. The Catalogue notes that volunteers tend to be more educated, come from higher social class and be more approval motivated. But it also says it is difficult to estimate the impact of volunteer bias and the direction of its effect. The bias can push a result up or down, depending on the question.
032 min
Named cases of volunteer bias
UK Biobank. UK Biobank is a large health study in the United Kingdom that recruited participants between 2006 and 2010. About 9.2 million people aged 40 to 69 were invited to take part, and 5.5% participated in the baseline assessment. Anna Fry and colleagues compared the participants with the general population in a 2017 paper in the American Journal of Epidemiology.
Participants were less likely to be obese, to smoke and to drink alcohol daily, and had fewer self-reported health conditions. At ages 70 to 74, rates of all-cause mortality and total cancer incidence were 46.2% and 11.8% lower in men, and 55.5% and 18.1% lower in women, than in the general population of the same age. The authors concluded that "UK Biobank is not representative of the sampling population; there is evidence of a 'healthy volunteer' selection bias."
The authors added a limit: "Nonetheless, valid assessment of exposure-disease relationships may be widely generalizable and does not require participants to be representative of the population at large." The bias affects how well the cohort shows how common a disease is. It may affect a comparison between exposed and unexposed people less.
A study of sexuality. According to the Catalogue of Bias, a study by Strassberg and colleagues in 1995 found that volunteers reported more positive attitudes toward sexuality, less sexual guilt and more sexual experience than non-volunteers. The authors wrote that these findings had "sobering implications" for how far such research could be generalised.
A probiotic trial. A trial of probiotic supplements for childhood allergy, reported by Jordan and colleagues in 2013, found that representation of the most deprived participants decreased as the trial progressed, because they were more likely to be lost to follow-up.
A related case: the Literary Digest poll. Before the 1936 United States presidential election, the magazine The Literary Digest mailed 10 million questionnaires, and 2.38 million came back. It predicted that Alf Landon would win. Landon received 37.54% of the popular vote. Later research concluded that non-response bias was the primary source of the error, although the sampling frame was also quite different from the vast majority of voters. This is a close relative of volunteer bias, because the people who returned the form chose to.
041 min
Volunteer bias in product and research work
Product teams meet this bias whenever participation is optional.
A user research panel (illustrative). A researcher recruits interview participants through a banner in the product. The people who click are more engaged than the typical user. The interviews report that the onboarding flow is clear, and the result is true for engaged users and uncertain for the rest.
An in-app survey (illustrative). A product manager sends a satisfaction survey and 8% answer. Those who answer include very happy and very unhappy users. A rise in the average score may mean that the group of people who answer has changed.
A beta programme (illustrative). A team opens a beta to anyone who signs up. Sign-ups come from users who like to try new things. A feature that tests well with them may confuse the broader base.
An opt-in experiment (illustrative). If users choose to join a new feature and are then compared with users who did not, the comparison mixes the feature's effect with the traits of people who opt in. A random assignment, as in an A/B test, avoids this. Without random assignment, the result is a correlation that cannot be read as cause.
A related error is survivorship bias, where only the cases that remain are studied. Volunteer bias and survivorship bias often appear together: those who volunteer and those who stay are both filtered groups.
051 min
How to reduce and report volunteer bias
Prefer random assignment and random sampling where possible.
Random selection from the full list of the population avoids letting people choose themselves in.
Lower the refusal rate.
The Catalogue of Bias says "The likelihood of volunteer bias increases as the refusal rate to volunteer increases." Anything that raises participation is likely to reduce the bias.
Protect privacy.
The Catalogue says ensuring anonymity and confidentiality is essential to increase participation, which decreases volunteer bias.
Compare responders with the invited list.
Fry and colleagues did this: they had demographic data on everyone invited and could compare those who took part with those who did not.
State the limit in the report.
Say who volunteered, how many refused, and which conclusions might not apply to the people who did not take part.
Ask what the result is used for.
The UK Biobank authors' point is that a comparison between groups can hold even when a sample is not representative. An estimate of how common something is cannot.
061 min
When the volunteer bias idea is misapplied
A low response rate does not prove bias. It is easy to treat a low response rate as the measure of bias. A meta-analysis of 30 methodological studies by Robert M. Groves, as summarised on Wikipedia, found that the coefficient of determination for variance in non-response bias by response rate was only 0.11, which makes response rate a weak predictor of non-response bias. A survey with 10% response can be less biased than one with 60%. What matters is whether responders differ from non-responders on the question being asked.
A representative sample is not always needed. The UK Biobank authors' conclusion shows that when the goal is to compare exposed and unexposed people, a sample that does not represent the population can still give valid results.
Volunteer bias is not the only selection error. Calling every skewed sample "volunteer bias" hides the cause. A sample can be skewed because of who was invited, who answered or who stayed. Name which one happened.
Compensation changes who volunteers. Offering a payment or a gift raises participation, but it can also change which people volunteer. This is reasoned from the mechanism and was not tested in the sources used here.
071 min
Volunteer bias versus similar errors
Several neighbouring terms are easy to mix up.
| Term | What goes wrong | How it differs |
|---|---|---|
| Volunteer bias | People who choose to join differ from those who do not | The cause is the choice to take part |
| Non-response bias | People who answer differ from people who do not | It covers anyone invited who did not answer, including people who never chose |
| Survivorship bias | Only cases that remain are studied | The cause is what remains, not what was chosen |
| Self-selection bias | People place themselves in a group | It is the general form. Volunteer bias is the case of choosing to join a study |
Wikipedia describes self-selection bias as closely related to non-response bias. The axis that separates them is which step filtered the sample: the decision to join, the decision to answer, or what remained.
?6 questions
