011 min
How recall bias distorted the MMR and autism question
In the early 2000s, news reports said that the measles, mumps and rubella (MMR) vaccine might cause autism. The claim came from a paper that was later withdrawn. Researchers then wanted to know how long after the vaccination parents of children with autism first noticed signs.
Andrews and colleagues reported in 2002 that, among parents of children with autism who had regressive symptoms, those whose child was diagnosed after the MMR news coverage more often placed the first signs soon after the vaccination than parents whose child was diagnosed before it.
The vaccine was the same for both groups of children. The difference was in the parents' memory. After weeks of news coverage, a parent had a ready explanation for what happened, and the vaccination date became the event the parent noticed most. That pattern has a name: recall bias.
The next question is why memory moves in this way, and why it matters more in some studies than in others.
022 min
How recall bias works
People do not retrieve a stored copy of the past. A person rebuilds an answer from fragments, from what seems likely, and from what they believe today. This is harmless when everyone rebuilds with the same errors, because the errors are then the same size in every group. Researchers call that a non-differential error. It adds noise and usually makes a real effect look weaker.
Recall bias is a differential error. One group remembers or reports more completely than the other, so the gap between groups no longer reflects the truth alone.
The usual cause is a difference in motivation. People who developed the outcome look harder through their memories for something that could explain it than people who did not. Researchers sometimes call this effort after meaning. A parent of a child with a diagnosis reviews the pregnancy, the diet and the illnesses. A parent of a healthy child has no reason to.
The error does not always push a result upward. Which way it moves depends on what people are asked about, and the section on study designs gives an example.
A worked example with invented numbers
A case-control study compares two groups. The cases are mothers of children born with a birth defect. The controls are mothers of children born without one. Each mother is asked whether she took a common medicine in early pregnancy. These numbers are made up to show the mechanism.
In truth, the medicine has no effect. Exactly 200 of the 1,000 case mothers and 200 of the 1,000 control mothers took it. The cases remember and report their use 90 percent of the time. The controls report it 60 percent of the time. Nobody invents a use that did not happen. People only forget.
| Group | Mothers | Truly took the medicine | Say they took it |
|---|---|---|---|
| Cases | 1,000 | 200 | 180 |
| Controls | 1,000 | 200 | 120 |
In this example, 180 of the 200 exposed case mothers say so, but only 120 of the 200 exposed control mothers do. The study records 180 exposed and 820 unexposed cases against 120 exposed and 880 unexposed controls. The odds ratio is (180 × 880) ÷ (820 × 120), which is about 1.6. With perfect memory it would be exactly 1.0, which means no link.
A reader who sees only the final number of 1.6 would conclude that the medicine raises risk by about 60 percent. Nothing in the result shows that the cause is memory. That is why a strong-looking association is a reason to ask how the exposure was measured. The same limit applies to any claim that two things go together, and correlation and causation covers it in general.
031 min
Which study designs recall bias affects
Recall bias needs two things: a question about the past, and a reason for groups to answer differently. Some study designs supply both.
A case-control study is the most exposed. It starts from people who already have the outcome and from similar people who do not, and then asks both groups about earlier exposures. The outcome is known before the question is asked, so the groups have different reasons to remember.
A retrospective cohort study and a survey of past behaviour carry the same risk, because both rely on self-reports about earlier times.
A prospective study avoids most of the problem. The Catalogue of Bias names a prospective design as the most suitable way to avoid recall bias. It records exposure at the start, before anyone knows who will develop the outcome, so there is no outcome yet to shape memory. The protection is not complete. Many prospective studies also ask people about their history when they join, and those answers can still be biased.
The direction of the error depends on the exposure. Habits people see as undesirable, such as smoking, tend to be underreported, which can hide a real risk. Habits people see as healthy can be overreported, which can make a protective effect look larger.
041 min
Where the term recall bias comes from
The term comes from epidemiology, the study of how disease spreads and what causes it. It arose because case-control studies depend on what people remember. No single founder is usually credited with it.
Raphael published a proposal for assessing and controlling recall bias in the International Journal of Epidemiology in 1987. Coughlin's 1990 review of the research then separated recall that is biased from recall that is merely inaccurate.
That separation matters. A person who forgets a visit to the doctor is inaccurate. Only if one group forgets more often than another does the error become a bias. The next section shows what researchers found when they tested for that difference.
052 min
When recall bias turns out to be small
If memory bias were always large, every case-control study would be doubtful. Two research teams tested the question directly, and neither found much.
A repeat-interview test in Montreal
A team at a Montreal hospital asked women about exposures that might affect pregnancy at two points: early in pregnancy, and again after delivery. The women gave birth between September 1983 and May 1985. The team then compared mothers of healthy babies with mothers of babies who had problems, and checked 39 exposure variables.
The women did change some answers. Deleting an earlier report was more common than adding a new one. The changes were similar in all groups. The odds ratios built from the early answers and the after-birth answers did not differ in any important way, so the study found no sign of biased reporting.
A diet test in Sweden
A Swedish team studied diet and breast cancer. Women had filled in a diet questionnaire before any diagnosis, as part of mammography screening. Later, 265 cases and 431 controls were interviewed about the same habits. The team compared the two sets of answers for each woman.
Agreement between the questionnaire and the interview was often low. For different food groups it ranged from 31 to 57 percent. Memory was clearly imperfect. Even so, the team saw little sign of recall bias, and the handful of group differences it did find were in foods that the public had not been warned about in connection with breast cancer.
What these tests show
In both studies people were inaccurate, but they were not inaccurate in a way that favoured one group. Recall bias is a risk that depends on the exposure, the question and the people, not a certainty. These two results also come from specific topics, so they do not clear every case-control study. The MMR example shows what happens when the topic carries a strong public story.
062 min
Recall bias in surveys and user research
Recall bias is not limited to medical research. A product team meets it whenever a survey asks about the past, such as a product manager asking customers how many times they contacted support in the last year.
Bradburn, Rips and Shevell showed in 1987 that people often cannot list every past event a survey asks about. When this happens, respondents often fall back on estimates built from the few details they do remember. A customer might remember two recent support calls and what a normal year feels like, and then guess. The answer is a mix of memory and estimation. A survey bias check should ask which of the two a question is really measuring.
Telescoping, the error in dating events
People also misplace events in time. This is called telescoping. An old event can seem more recent than it was, which is forward telescoping. A recent event can seem older, which is backward telescoping. A question about the last 30 days then counts old events that fall outside the window and misses recent events that people think are older.
Researchers usually trace the study of telescoping to a 1964 article by Neter and Waksberg.
Where telescoping changes direction
The two directions are not equally common at every age of an event. A summary of the research puts the switch point at about three years. Events older than that tend to be reported as more recent than they were. Events more recent than that tend to be reported as older than they were. A question about a time window longer than a year or two will therefore mix both errors in one number.
When a shorter window backfires
The standard advice is to ask about a shorter period. A short window reduces telescoping and the amount of guessing a person must do. It has a cost for rare events. Shorter reference periods give more accurate recall but can miss rare events. A question about the last week returns zero for most people if the behaviour happens twice a year. The window should match how often the behaviour occurs.
071 min
Recall bias vs nearby concepts
Four nearby ideas are often mixed up with recall bias. Social desirability bias is the closest in effect. Social desirability bias happens when people give the answer that looks good, even when they remember correctly. Response bias is the general term for any pattern of answers that differs from the truth, and recall bias is one kind. Hindsight bias is about judging the past: once people know an outcome, they believe they predicted it. The machine learning measure called recall shares only the word. It scores how many true cases a classifier finds.
| Concept | What goes wrong | What separates it from recall bias |
|---|---|---|
| ResearchSocial desirability bias | ||
| ResearchResponse bias | ||
| PsychologyHindsight bias | ||
| Not in the library yetSelection bias |
The deciding question is simple. Ask whether the error comes from remembering, from choosing what to say, or from who was asked.
082 min
How to reduce recall bias
Researchers cannot remove memory error, but they can limit how much it differs between groups. These steps are listed from the strongest to the most practical.
Record exposure before the outcome is known.
A prospective design is the strongest option.
Use records instead of memory.
Pharmacy data, medical charts, registries and product logs do not depend on recollection.
Ask about a short, anchored period.
"Since the start of the quarter" works better than "in the past year", and the length should match how often the behaviour occurs.
Ask every group the same questions in the same way.
Keep interviewers unaware of which group a person is in and of what the study expects.
Pick a comparison group with a similar reason to remember.
Controls who have another illness search their past as hard as cases do.
Test it.
Re-ask a small sample, or compare answers with records, as the Montreal and Swedish teams did.
Use a diary when the behaviour is frequent.
A diary avoids the memory problem but asks much more effort from each participant.
When you read a retrospective result, ask three questions. Was the exposure recorded before the outcome? Did the groups have different reasons to remember? And if memory did differ, which way would it move the number? If you cannot answer the third, treat the result as a hypothesis to check, not a finding to act on.
?4 questions
Questions people ask
Is recall bias the same as forgetting?
Does recall bias affect randomized trials?
What is another example of recall bias?
Can recall bias hide a real effect?
§10 sources
Sources
Andrews N, Miller E, Taylor B, et al. Recall bias, MMR, and autism. Archives of Disease in Childhood, 2002.
Mackenzie SG, Lippman A. An investigation of report bias in a case-control study of pregnancy outcome. American Journal of Epidemiology, 1989.
Holmberg L, Ohlander EM, Byers T, et al. A search for recall bias in a case-control study of diet and breast cancer. International Journal of Epidemiology, 1996.
Coughlin SS. Recall bias in epidemiologic studies. Journal of Clinical Epidemiology, 1990.
Show all 10 sourcesShow fewer sources
Raphael K. Recall bias: a proposal for assessment and control. International Journal of Epidemiology, 1987.
Bradburn NM, Rips LJ, Shevell SK. Answering autobiographical questions: the impact of memory and inference on surveys. Science, 1987.
Catalogue of Bias Collaboration. Recall bias. Catalogue of Bias, 2017.
CASRAI. Recall bias: causes, direction, and how to reduce it.
Lensym. Recall bias: memory distortion and temporal telescoping in surveys.
Telescoping effect. Wikipedia, a summary of the research.
