The Experimental Mindset

Also called Experiment Mindset, Test-and-Learn Mindset. Also in Research, Business

Quick answer

The experimental mindset is the habit of treating a decision as a question to test instead of a plan to defend. A person states what they expect, runs the smallest test that could show whether they are right, reads the result honestly, and lets it change the next step. A test that fails still teaches something.

~20 sec read

011 min

Why a plan feels safer than a test

Picture a product team whose sign-up numbers have gone flat. Everyone agrees that the onboarding flow is the cause, so the team plans to rebuild the whole flow. The rebuild will take six months. It has an owner, a deadline and a place on the roadmap. Nobody has tested the belief underneath it: that a better flow will bring more people through.

The plan feels safe because it looks like progress. Each week adds visible work. A test looks like a delay, and it might say that the belief was wrong. Fear of that answer is real. People worry about looking wrong in front of the team, and about the time already spent. Once months have gone into the rebuild, dropping it means admitting that those months went to a guess. That pull to keep going after money and time are spent is called sunk cost.

Now suppose the rebuild ships and sign-ups do not move. The team finally has its answer, but it paid six months for it. The experimental mindset asks for the same answer sooner and at a much lower price.

022 min

How the experimental mindset works

The mindset borrows the loop that scientists use and shrinks it to fit ordinary work. It has five moves.

  1. Notice a question. Sign-ups are flat, and the team does not know why.
  2. Write a hypothesis. A hypothesis is a guess stated so that evidence can show it is wrong: "If we remove the second screen, more visitors will finish signing up."
  3. Choose the smallest test that could show the guess is wrong, and fix the pass mark before it starts.
  4. Run the test and read the whole result, including the part you hoped not to see.
  5. Decide: repeat it, scale it, change it or drop it.

Step 3 does most of the work. "Smallest" means the cheapest version that still gives a real answer from real people. A model of the answer, or a meeting about it, does not count. Fixing the pass mark first matters just as much. Without it, a mixed result is easy to read as a win after the fact.

The strongest form of this step is a controlled experiment, where one group sees the change and a similar group does not. Web teams usually run that as A/B testing. The mindset is wider than that tool. A one-week trial, five customer calls and a pilot with a single team all count, as long as each one is set up so that it can come back with "no".

032 min

Where the experimental mindset comes from

No single paper defines the experimental mindset. The practice is the scientific method, which is far older than the phrase, and many writers on work and self-improvement have used the phrase loosely. The version in this article comes from the neuroscientist Anne-Laure Le Cunff, who made it the centre of her book Tiny Experiments. In a Big Think interview she says the idea is based on the scientific method.

She sorts the ways people respond to uncertainty by two questions: how curious they are, and how ambitious.

MindsetCuriosityAmbitionWhat the person does with uncertainty
CynicallowlowMocks people who still try
EscapisthighlowExplores, but avoids committing effort
PerfectionistlowhighWorks harder to avoid not knowing
ExperimentalhighhighTurns it into a question and tests it

Le Cunff describes four mindsets: cynical (low curiosity, low ambition), escapist (high curiosity, low ambition), perfectionist (high ambition, low curiosity) and experimental (high in both).

Le Cunff adds that these are not personality types. They shift with the situation, so one engineer can be experimental about a new feature and perfectionist about a performance review. That matters for teams, because the mindset has to be chosen again each time a decision comes up.

The same attitude has been run at much larger scale inside technology companies, as a programme of thousands of tests. Those companies publish measured results, so the next sections use their cases.

042 min

The Bing headline test

Bing, Microsoft's search engine, gives the clearest case of a cheap test finding value that nobody had expected. In 2012, a Microsoft employee proposed a change to how Bing showed the headlines of search ads. The change needed only a few days of an engineer's time. It was one of hundreds of ideas on the list.

The program managers who ranked the ideas placed this one low on the list, and it waited more than six months. An engineer who saw how cheap the change would be to build then ran it as a live test. The cost of building it was small, so a test was easier than another discussion.

Within hours the system was reporting unusually high revenue for the new headline version.

A result that looked too good

The test system raised a "too good to be true" alert within hours, and the team had to check for a bug before believing it. Such alerts usually point to a bug, but Kohavi and Thomke say this one did not. The test showed that revenue rose by 12%, and Kohavi and Thomke estimate the change was worth more than $100 million a year in the United States alone. The user-experience measures did not get worse.

Why the idea waited

The managers were not careless. A ranked list of ideas is a set of predictions about value, and predictions about new ideas are weak. Here the prediction was wrong by a wide margin, and only the test could show it. A ranking costs nothing to make, so it is easy to trust. A test costs a few days, and it was the cheaper way to learn.

the ideawaits six monthslive testAB12%
Look at the left window: the idea was ready to build for months. The right window shows how little it took to find out what it was worth.

052 min

Sizing the experimental mindset to the decision

Back to the sign-up team. Before it picks a test, it has to ask how much it can lose if the guess is wrong, and how easily it could undo the work.

Jeff Bezos wrote in his 2016 letter to Amazon shareholders that many decisions can be reversed, and he called these two-way doors. A team can walk back through such a door, so it can be made quickly by a small group. He also wrote that most decisions should be made with about 70% of the information you wish you had. Waiting for certainty costs time that a reversible decision does not need to spend.

The mindset turns that idea into a method. A six-month rebuild looks like a decision that cannot be undone. The team can split it into steps that can be undone, and run the first one. The team sets the full rebuild aside and tests one change on a share of new visitors for one week. The result then shows whether the full rebuild is worth planning at all.

Letting the test set the size of the investment

Tests can also answer how much to spend. Microsoft ran tests that added small delays to Bing, and they showed that each 100 milliseconds of speed was worth 0.6% of revenue. With a price for each 100 milliseconds, the team that worked on Bing's speed could justify its own size, and could delay features that slowed the page.

full rebuildwhole flowone weekone change
The faded window is the plan the team set aside. The solid window is the one change they test instead.

061 min

What tests find that opinions miss

The sign-up team might guess that a small change cannot matter much. Microsoft's records say otherwise.

A small change with a large effect

In 2008, a test with about 900,000 users in the United Kingdom found that opening Hotmail in a new tab raised engagement by 8.9%. Engagement is the number of clicks on the home page. Most changes to engagement move it by less than 1%, so this was large. The idea was controversial, because few sites opened links in new tabs at that time, so Microsoft released it only in the United Kingdom first. Microsoft repeated the test in June 2010 with 2.7 million American users, saw similar results, and then released the change worldwide.

Amazon found something similar. Its tests showed that moving credit card offers from the home page to the shopping cart page raised profits by tens of millions of dollars a year. The case shows two habits. The team ran a test instead of settling an argument, and it repeated the test in a new country before it spread the change. A result that holds in a second sample is stronger than one that held once.

072 min

Most ideas will not win

The Hotmail case is the kind of story people remember. Stories like it hide how often tests come back empty.

Kohavi and Thomke report that, across the tests run at Google and Bing, roughly one in five to one in ten gives a positive result. Kohavi and Thomke also write that the vast majority of new ideas fail when they are tested, and that even experts often misjudge which ones will pay off. A team that treats a failed test as a mistake has the wrong expectation.

This changes what counts as success. A team that judges itself by the share of tests that win will either stop testing or start stretching its readings. A better measure is how many questions it answered, and how cheaply. Each losing test also stops a larger build that would have lost the same way.

Why cheap tests change the numbers

Low win rates are only bearable when a test is cheap. Kohavi and Thomke point out that once a company has built systems to run tests, one more test costs very little. Several large technology companies each run more than 10,000 online controlled experiments a year. At Bing, about 80% of proposed changes are first run as controlled experiments. Kohavi and Thomke say dozens of revenue changes found this way each month raised Bing's revenue per search by 10% to 25% every year.

A company without those systems pays the full price for every test, and its managers soon ask for fewer of them. A small team can start with spreadsheets and a share of traffic, and move to shared tools as the number of tests grows.

081 min

Booking.com and testing as the default answer

Booking.com, the travel booking site, shows what happens when a company applies the mindset to almost everything. Thomke reports that Booking.com runs about 25,000 tests a year.

Stefan Thomke opens his 2020 article about it with a proposal. In December 2017, the design director at Booking.com proposed testing a far simpler home page than the one in use. The new page had a small window asking where the customer was going, the dates and the number of people, plus three simple options. It would remove content that the company had spent years refining.

That is the kind of idea that teams fight over for months. People who built the old page defend it, and people who like the new one attack its critics. At a company that tests by default, the argument ends in a test.

A test can only replace an argument if people can report a loss without blame. Teams that punish a failed test soon stop reporting them, and the data becomes a record of good news. This is why psychological safety comes before any tool.

092 min

When the experimental mindset breaks down

The mindset fails in four known ways. Each has a sign that the team can watch for.

A test that costs too much

Not every test is small. Microsoft spent more than $25 million on a third pane of social media content in Bing, and it produced almost no gain in engagement or revenue. Large bets still need a cheap test first, at the lowest cost that can answer the question. Tests are most useful when an idea is costly to build.

An average that hides a broken group

A total can mislead when it adds up groups that behave differently. In one Microsoft test, a JavaScript bug stopped Internet Explorer 7 users from clicking Bing results, and the overall result turned from positive to negative. A team that read only the total would have dropped a good idea. A team that also looked at each group found the bug. The surprising result in the Bing headline case worked the same way: an unusual number is first a reason to check the setup.

A measure that rewards the wrong thing

Choosing the number that decides success is hard. At Bing, a worse set of search results would make people search and click on ads more, so a team that counted only those two numbers would reward the damage. Bing's team therefore picked measures such as fewer queries per task and more tasks completed, and Kohavi and Thomke advise reviewing the choice every year.

Experiments used to avoid deciding

A team can test forever. Each test needs an end: before it starts, write down the date it will be read and the four choices that follow, which are to repeat, scale, change or drop. Stories about big wins, such as the Bing headline, are also told far more often than the many tests that found nothing. Remember survivorship bias when reading them.

Some decisions cannot be tested fairly at all, such as one hire or one merger. The mindset still helps there in a smaller form: write down what you expect, and compare it with what happened.

101 min

The experimental mindset and nearby ideas

Three ideas are often mixed up with the experimental mindset. The deciding fact for each is what it describes.

IdeaWhat it isWhat separates it
PsychologyGrowth mindsetIt is about how you think of your own ability. The experimental mindset is about how you make decisions under uncertainty. A person can hold one and not the other.What it is: A belief that abilities can improve with effort
ProductA/B testIt is one tool for one kind of question. The mindset also covers trials, pilots and interviews that have no split.What it is: A test that splits users between two versions
Not in the library yetFailing fastIt describes speed only. The mindset first writes the guess and the pass mark, so that a failure is recognisable.What it is: Stopping quickly when something does not work

112 min

Running a first experiment

Begin with one belief that a plan depends on, and write it in this form.

We believe [change] will lead to [result] for [group]. We will test it by [smallest version] for [time]. It passes if [measure] reaches [number set now]. If it passes, we will [next step]. If not, we will [other step].

Then set the edges of the test.

  1. One change.

    If two things change at once, the result cannot say which one mattered.

  2. A time limit.

    Choose it before you start, and read the result on that date, not earlier.

  3. A limit on loss.

    Decide the most money, time or goodwill you can lose, and keep the test inside it.

  4. One measure that matters,

    plus a look at each group, so that a broken segment shows up.

  5. A written outcome.

    Record what you expected, what happened and what you will do, including when the guess was right.

The last item is also how teams detach their ideas from the people who proposed them. When the record shows many guesses, wrong and right, no single guess is a verdict on its author.

Back to the sign-up team one last time. After the week, the shorter flow either clears the pass mark the team set or it does not. If it does, the team plans the next change and knows why. If it does not, five months and a rebuild are still available for a better idea.

121 min

Questions about the experimental mindset

Is the experimental mindset the same as trial and error?

No. Trial and error tries things and notes what happened. The experimental mindset states a guess and a pass mark first, so the result can say the guess was wrong. That makes a failed try easier to learn from.

How long should a personal experiment run?

Long enough to see the thing you are testing more than once. A daily habit needs at least a week or two. Choose the length before you begin, and read the result on that date.

Can I use it in a team that punishes mistakes?

Start with a small test whose only purpose is to answer a question, and report it the same way whether it wins or loses. Teams that punish failed tests stop reporting them, so the manager’s reaction matters most.

Does it mean I should never commit to a long plan?

No. A long plan is fine when its riskiest belief has been tested, or when the work cannot be split into steps. The mindset asks that the riskiest belief be tested first.

?4 questions

Questions people ask

Is the experimental mindset the same as trial and error?

No. Trial and error tries things and notes what happened. The experimental mindset states a guess and a pass mark first, so the result can say the guess was wrong. That makes a failed try easier to learn from.

How long should a personal experiment run?

Long enough to see the thing you are testing more than once. A daily habit needs at least a week or two. Choose the length before you begin, and read the result on that date.

Can I use it in a team that punishes mistakes?

Start with a small test whose only purpose is to answer a question, and report it the same way whether it wins or loses. Teams that punish failed tests stop reporting them, so the manager’s reaction matters most.

Does it mean I should never commit to a long plan?

No. A long plan is fine when its riskiest belief has been tested, or when the work cannot be split into steps. The mindset asks that the riskiest belief be tested first.

§4 sources

Sources

Show all 4 sources
  1. Kohavi, R., & Thomke, S. (2017). The Surprising Power of Online Experiments. Harvard Business Review, September–October (reprint).

  2. Thomke, S. (2020). Building a Culture of Experimentation. Harvard Business Review, March–April.

  3. Bezos, J. (2017). 2016 Letter to Shareholders. Amazon.

  4. Le Cunff, A.-L. (n.d.). A neuroscientist's guide to fixing your mind in 15 mins. Big Think interview.

Keep reading

More from Product

All of Product
All of Product

Missing a concept?

Tell us what to write next. We’ll email you when it’s live.

Several? Separate them with commas.

Only to tell you when it’s live.

How we use your email: privacy.