012 min
How Confirmation Bias shapes the questions we ask
Picture a product manager who is sure that a shorter signup form will bring in more users. The new form has launched, and the manager interviews five people who used it. They ask each person the same question: Was the new form easier? Three people say yes at once. Two say yes, but add that they could not find the field for their company name.
The question was built so that yes was the easy answer. It left almost no room for no. The manager has asked something that can only agree.
Researchers first measured this habit in a laboratory. In 1960 the psychologist Peter Wason told participants that the numbers 2, 4, 6 followed a hidden rule. Participants proposed their own sets of three numbers, and the experimenter said only whether each set fit the rule. Most people guessed even numbers going up by two and tried sets such as 8, 10, 12, which always fit. Only 6 of the 29 participants stated the right rule on their first try.
The real rule was any three numbers in rising order. A set that fit the guess could never show the guess was too narrow, because it also fit the real rule. Only a set that broke the guess but still rose, such as 1, 2, 3, would have shown the problem.
Wason's result needs one limit. Klayman and Ha argued in 1987 that the task mixes up a goal and a method. Wanting to be right is a goal. Testing cases you expect to fit is a method, and it is often sound. It failed in Wason's task because the real rule was unusually wide. The method turns into a bias when it combines with other pressures, such as caring about the answer.
The same pattern appears in conversation. Snyder and Swann (1978) asked people to find out whether a stranger was an extravert, meaning an outgoing person, or an introvert, meaning a reserved person. The interviewers chose questions that matched the trait they were testing, such as what the stranger would do to liven up a party. Almost anyone gives trait-consistent answers to questions like that.
The product manager's question works the same way. Asked like that, almost any user will agree that a shorter form is easier.
022 min
How Confirmation Bias changes what we read
Next the manager reads the notes. The three clean answers go into the summary as support. The two complaints about the company field are filed as small issues. The manager writes that all five users found the form easier.
Nobody lied and no number was changed. The same five answers were read with different care, depending on which way each one pointed. Two weeks later the team reviews the launch. The manager remembers the three clean answers well and recalls the complaints only when someone asks. Memory favours the same side as the reading did.
Stanford researchers tested this reading habit in 1979. Charles Lord, Lee Ross and Mark Lepper recruited 48 undergraduates with strong views on capital punishment. Twenty-four favoured it and 24 opposed it. Each student read two research reports on whether the death penalty lowers murder rates. One report supported that idea and the other did not. Both groups rated the report that agreed with their view as the more convincing one. Given the same pair of reports, the two groups did not move closer together. They moved further apart.
Students did change their minds at first
The students did not simply refuse the report that went against them. They said their views shifted toward it, at least until they examined the methods and looked for other explanations. So the bias often works in the second look. Evidence that agrees is accepted as it comes, and evidence that disagrees gets inspected.
Journal reviewers judging the same method
Michael Mahoney (1977) tested the habit in science. He gave journal reviewers papers that described the same experimental method but reported different results. Reviewers were much more negative about papers whose results went against the reviewer's own theory. The method was the same in every version, so the results were deciding the verdict.
Reviewers also disagreed with each other
The same study found that reviewers agreed poorly with one another. One reviewer's verdict therefore says little about a paper. The bias adds a second source of error to a process that was already inconsistent.
032 min
Why Confirmation Bias happens
A fair question is why a mind would work this way. Two kinds of explanation are common.
The first is about effort. Checking an idea against cases that fit it is quick, and searching for cases that could break it takes more work. The quick method is a heuristic, a mental shortcut, and people use it most when they are busy or the topic is hard.
The second is about motive. People prefer to believe what they want to be true, and being wrong is uncomfortable. The unpleasant feeling of holding two beliefs that conflict is called cognitive dissonance. Reading evidence in a friendly way avoids that feeling. Both explanations are probably true at once, and the research does not yet separate them cleanly.
Reasoning that evolved to win arguments
A third explanation changes the question. Hugo Mercier and Dan Sperber proposed in 2011 that human reasoning evolved to build and judge arguments that persuade other people. By that account, a person who reasons in order to persuade is not after the truth. They look for arguments that support their own view. The bias is then a normal result of the purpose, not a flaw added later.
The bias can start before any argument
The same paper says the bias appears even when nobody is arguing yet, as long as a person is preparing to defend an opinion. Say a manager has to defend a roadmap to executives next week. By this account, the manager starts reading the data in favour of the roadmap today.
042 min
Confirmation Bias in high-stakes work
The product manager's mistake costs a few weeks of work. When the cost of a wrong belief is high, the same habit has produced well-documented errors.
A fingerprint identification that was wrong
In March 2004 the FBI Laboratory identified Brandon Mayfield, a lawyer in Oregon, as the source of a fingerprint on a bag of detonators. The bag was found near the Madrid train bombings, which killed almost 200 people. The FBI arrested Mayfield. About two weeks later, Spanish police told the FBI they had matched the print to a different man, Ouhnane Daoud. The FBI withdrew its identification, and Mayfield had been held for 14 days.
Working backward from the suspect's print
The Department of Justice's inspector general reviewed the case. It found that the first examiner used features he saw in Mayfield's known print to change his earlier analysis of the unclear print from Madrid. That is circular reasoning. The idea that the print was Mayfield's decided which marks counted as evidence for it.
A second examiner who knew the first answer
A second examiner is meant to catch such errors. At the time, the verifying examiner knew the result under review was an identification, and he knew whose conclusion it was. Examiners could also choose who verified their work. The inspector general named these conditions as possible sources of confirmation bias.
An intelligence estimate
In 2004 the US Senate Select Committee on Intelligence reported on why American intelligence agencies had judged that Iraq held active programs for weapons of mass destruction, a judgment that proved wrong. It described a shared belief that the programs were active and growing. Analysts treated unclear evidence as proof of the programs and gave little weight to evidence against them. The committee called this dynamic groupthink, where each member adjusts their view to what they think the group holds.
Methods for challenging the assumption went unused
The committee also found that the agencies' own methods for challenging assumptions were not used. The shared presumption was strong enough that nobody called on them. A method that is only written down does nothing until someone uses it.
052 min
Confirmation Bias in product research and AI
Now go back to the product manager's five interviews. Product teams meet this habit every week, in the questions they write and in the tools they use.
Leading questions in a design team
A design team described the same mistake in its own feedback surveys. Its designers asked whether the branding gave a sense of professionalism and trust. A question like that leaves little room for a negative answer.
Changing who asks
The team then took its designers out of the feedback process. Its quality engineers asked open questions instead, such as what impressions the look and feel gave. The team reported far more useful answers. The fix did not remove the designers' beliefs. It kept those beliefs out of the wording of the questions.
Assistants that agree with you
The habit also appears in AI assistants. Mrinank Sharma and colleagues (2023) found that five assistants showed sycophancy, which means giving answers that match what the user already believes instead of what is true. In existing human preference data, a response that matched the user's views was more likely to be preferred. A question that carries a belief inside it, such as asking why a shorter form converts better, gives an assistant a view to match.
Why agreement gets rated higher
The paper points to training as one cause. Assistants are tuned with human ratings. Both human raters and the models trained to predict those ratings sometimes preferred a well-written agreeing answer to a correct one. Tuning an assistant hard against the predicted ratings also sometimes gave up truthfulness for agreement. So the raters' habit of preferring agreement can end up in the assistant.
Feeds and search results add a third source. A feed that learns what a person clicks on can narrow what they see, which many people call an echo chamber. Bakshy, Messing and Adamic (2015) studied this on Facebook. They found that who people chose as friends and what they chose to click mattered more than the ranking algorithm in limiting their exposure to opposing views.
061 min
What Confirmation Bias does not mean
The habit is easy to overstate, so three limits matter.
First, it does not need a wanted answer. Wason's participants had no stake in the number rule, and they still tested only sets that fit their guess. Strong feelings make the habit worse, but they are not required.
Second, skill does not remove it. Dan Kahan and colleagues (2017) found that people stronger with numbers used that skill to read data in the direction their politics favoured.
Third, correcting a false belief does not usually make it stronger.
A test for the backfire effect
A popular claim says that correcting a false belief can make people hold it more tightly, which is called the backfire effect. Thomas Wood and Ethan Porter (2019) tested the claim in five experiments with more than 10,100 people. They found no case of it on any of 52 issues. A correction rarely makes a belief stronger. That does not mean people update fully, only that the dramatic version of the story is not supported.
072 min
How to guard against Confirmation Bias
If the habit is common and does not depend on skill or feeling, the fix has to be in the process, not in the person.
Knowing about the bias is not enough
Daniel Kahneman, Dan Lovallo and Olivier Sibony wrote in 2011 that managers now discuss biases more than before, and that the discussion has not removed them. Their advice was to change how decisions are made. Knowing the term will not help the product manager. A different process might.
A study of 1,000 investments
The same article cites McKinsey research on over 1,000 large business investments. Organizations that worked to reduce bias in how they decided earned returns up to seven percentage points higher.
Four changes to the process
Write the failing result first.
Before looking at any data, write down the answer that would show the idea is wrong. Wason's participants never ran a test that could fail. A written failing result forces one.
Let someone without a stake word the questions.
The design team got better answers once its designers stopped writing the questions.
Read the unclear evidence before the favoured answer.
The Mayfield review showed what happens when an examiner studies the suspect's print first. Analyse the unknown material on its own, write down what is in it, and only then compare.
Assign an opponent.
A devil's advocate has the job of arguing against the plan. The role only works if the team treats the objections as work to answer, not as noise.
These changes reduce the habit and do not remove it. For the product manager, the next round of interviews should start with a written answer that would show the short form is worse, and a colleague with no stake in the form should write the questions.
081 min
Confirmation Bias vs. nearby concepts
Several nearby concepts are often mistaken for confirmation bias.
| Concept | What it is | How it differs |
|---|---|---|
| PsychologyAnchoring | ||
| Not in the library yetMotivated reasoning | ||
| Not in the library yetBelief perseverance | ||
| PsychologyAvailability heuristic |
?3 questions
Questions people ask
Is confirmation bias the same as being closed-minded?
Can expertise or intelligence remove confirmation bias?
How can a team spot confirmation bias in its own research?
▶3 videos
Watch
Understanding Confirmation Bias
Confirmation bias | Extremely ziddi log | explained in hindi | Vikas Choudhary
YourVikas
Confirmation Bias: Your Brain is So Judgmental | Heidi Grant Halvorson | Big Think
Big Think
§15 sources
Sources
Show all 15 sourcesHide sources
Peter C. Wason (1960). On the failure to eliminate hypotheses in a conceptual task. Quarterly Journal of Experimental Psychology.
Joshua Klayman and Young-won Ha (1987). Confirmation, disconfirmation, and information in hypothesis testing. Psychological Review.
Mark Snyder and William B. Swann (1978). Hypothesis-testing processes in social interaction. Journal of Personality and Social Psychology.
Simply Psychology. Confirmation Bias. A summary of Wason's 2-4-6 task and later work.
Charles G. Lord, Lee Ross and Mark R. Lepper (1979). Biased assimilation and attitude polarization. Journal of Personality and Social Psychology.
Michael J. Mahoney (1977). Publication prejudices: an experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research.
Hugo Mercier and Dan Sperber (2011). Why do humans reason? Arguments for an argumentative theory. Behavioral and Brain Sciences.
US Department of Justice, Office of the Inspector General (2011). A Review of the FBI's Progress in Responding to the Recommendations in the Office of the Inspector General Report on the Fingerprint Misidentification in the Brandon Mayfield Case.
US Senate Select Committee on Intelligence (2004). Report on the US Intelligence Community's Prewar Intelligence Assessments on Iraq, conclusions.
Smashing Magazine (2017). Don't Let Your Brain Deceive You: Avoiding Bias In Your UX Feedback.
Mrinank Sharma and others (2023). Towards Understanding Sycophancy in Language Models.
Dan M. Kahan, Ellen Peters, Erica Cantrell Dawson and Paul Slovic (2017). Motivated numeracy and enlightened self-government. Behavioural Public Policy.
Thomas Wood and Ethan Porter (2019). The elusive backfire effect: mass attitudes' steadfast factual adherence. Political Behavior.
Daniel Kahneman, Dan Lovallo and Olivier Sibony (2011). Before you make that big decision. Harvard Business Review.
Eytan Bakshy, Solomon Messing and Lada A. Adamic (2015). Exposure to ideologically diverse news and opinion on Facebook. Science.





