011 min
Why a plan feels safer than a test
Picture a product team whose sign-up numbers have gone flat. Everyone agrees that the onboarding flow is the cause, so the team plans to rebuild the whole flow. The rebuild will take six months. It has an owner, a deadline and a place on the roadmap. Nobody has tested the belief underneath it: that a better flow will bring more people through.
The plan feels safe because it looks like progress. Each week adds visible work. A test looks like a delay, and it might say that the belief was wrong. Fear of that answer is real. People worry about looking wrong in front of the team, and about the time already spent. Once months have gone into the rebuild, dropping it means admitting that those months went to a guess. That pull to keep going after money and time are spent is called sunk cost.
Now suppose the rebuild ships and sign-ups do not move. The team finally has its answer, but it paid six months for it. The experimental mindset asks for the same answer sooner and at a much lower price.
022 min
How the experimental mindset works
The mindset borrows the loop that scientists use and shrinks it to fit ordinary work. It has five moves.
- Notice a question. Sign-ups are flat, and the team does not know why.
- Write a hypothesis. A hypothesis is a guess stated so that evidence can show it is wrong: "If we remove the second screen, more visitors will finish signing up."
- Choose the smallest test that could show the guess is wrong, and fix the pass mark before it starts.
- Run the test and read the whole result, including the part you hoped not to see.
- Decide: repeat it, scale it, change it or drop it.
Step 3 does most of the work. "Smallest" means the cheapest version that still gives a real answer from real people. A model of the answer, or a meeting about it, does not count. Fixing the pass mark first matters just as much. Without it, a mixed result is easy to read as a win after the fact.
The strongest form of this step is a controlled experiment, where one group sees the change and a similar group does not. Web teams usually run that as A/B testing. The mindset is wider than that tool. A one-week trial, five customer calls and a pilot with a single team all count, as long as each one is set up so that it can come back with "no".
032 min
Where the experimental mindset comes from
No single paper defines the experimental mindset. The practice is the scientific method, which is far older than the phrase, and many writers on work and self-improvement have used the phrase loosely. The version in this article comes from the neuroscientist Anne-Laure Le Cunff, who made it the centre of her book Tiny Experiments. In a Big Think interview she says the idea is based on the scientific method.
She sorts the ways people respond to uncertainty by two questions: how curious they are, and how ambitious.
| Mindset | Curiosity | Ambition | What the person does with uncertainty |
|---|---|---|---|
| Cynical | low | low | Mocks people who still try |
| Escapist | high | low | Explores, but avoids committing effort |
| Perfectionist | low | high | Works harder to avoid not knowing |
| Experimental | high | high | Turns it into a question and tests it |
Le Cunff describes four mindsets: cynical (low curiosity, low ambition), escapist (high curiosity, low ambition), perfectionist (high ambition, low curiosity) and experimental (high in both).
Le Cunff adds that these are not personality types. They shift with the situation, so one engineer can be experimental about a new feature and perfectionist about a performance review. That matters for teams, because the mindset has to be chosen again each time a decision comes up.
The same attitude has been run at much larger scale inside technology companies, as a programme of thousands of tests. Those companies publish measured results, so the next sections use their cases.
042 min
The Bing headline test
Bing, Microsoft's search engine, gives the clearest case of a cheap test finding value that nobody had expected. In 2012, a Microsoft employee proposed a change to how Bing showed the headlines of search ads. The change needed only a few days of an engineer's time. It was one of hundreds of ideas on the list.
The program managers who ranked the ideas placed this one low on the list, and it waited more than six months. An engineer who saw how cheap the change would be to build then ran it as a live test. The cost of building it was small, so a test was easier than another discussion.
Within hours the system was reporting unusually high revenue for the new headline version.
A result that looked too good
The test system raised a "too good to be true" alert within hours, and the team had to check for a bug before believing it. Such alerts usually point to a bug, but Kohavi and Thomke say this one did not. The test showed that revenue rose by 12%, and Kohavi and Thomke estimate the change was worth more than $100 million a year in the United States alone. The user-experience measures did not get worse.
Why the idea waited
The managers were not careless. A ranked list of ideas is a set of predictions about value, and predictions about new ideas are weak. Here the prediction was wrong by a wide margin, and only the test could show it. A ranking costs nothing to make, so it is easy to trust. A test costs a few days, and it was the cheaper way to learn.
052 min
Sizing the experimental mindset to the decision
Back to the sign-up team. Before it picks a test, it has to ask how much it can lose if the guess is wrong, and how easily it could undo the work.
Jeff Bezos wrote in his 2016 letter to Amazon shareholders that many decisions can be reversed, and he called these two-way doors. A team can walk back through such a door, so it can be made quickly by a small group. He also wrote that most decisions should be made with about 70% of the information you wish you had. Waiting for certainty costs time that a reversible decision does not need to spend.
The mindset turns that idea into a method. A six-month rebuild looks like a decision that cannot be undone. The team can split it into steps that can be undone, and run the first one. The team sets the full rebuild aside and tests one change on a share of new visitors for one week. The result then shows whether the full rebuild is worth planning at all.
Letting the test set the size of the investment
Tests can also answer how much to spend. Microsoft ran tests that added small delays to Bing, and they showed that each 100 milliseconds of speed was worth 0.6% of revenue. With a price for each 100 milliseconds, the team that worked on Bing's speed could justify its own size, and could delay features that slowed the page.
061 min
What tests find that opinions miss
The sign-up team might guess that a small change cannot matter much. Microsoft's records say otherwise.
A small change with a large effect
In 2008, a test with about 900,000 users in the United Kingdom found that opening Hotmail in a new tab raised engagement by 8.9%. Engagement is the number of clicks on the home page. Most changes to engagement move it by less than 1%, so this was large. The idea was controversial, because few sites opened links in new tabs at that time, so Microsoft released it only in the United Kingdom first. Microsoft repeated the test in June 2010 with 2.7 million American users, saw similar results, and then released the change worldwide.
Amazon found something similar. Its tests showed that moving credit card offers from the home page to the shopping cart page raised profits by tens of millions of dollars a year. The case shows two habits. The team ran a test instead of settling an argument, and it repeated the test in a new country before it spread the change. A result that holds in a second sample is stronger than one that held once.
072 min
Most ideas will not win
The Hotmail case is the kind of story people remember. Stories like it hide how often tests come back empty.
Kohavi and Thomke report that, across the tests run at Google and Bing, roughly one in five to one in ten gives a positive result. Kohavi and Thomke also write that the vast majority of new ideas fail when they are tested, and that even experts often misjudge which ones will pay off. A team that treats a failed test as a mistake has the wrong expectation.
This changes what counts as success. A team that judges itself by the share of tests that win will either stop testing or start stretching its readings. A better measure is how many questions it answered, and how cheaply. Each losing test also stops a larger build that would have lost the same way.
Why cheap tests change the numbers
Low win rates are only bearable when a test is cheap. Kohavi and Thomke point out that once a company has built systems to run tests, one more test costs very little. Several large technology companies each run more than 10,000 online controlled experiments a year. At Bing, about 80% of proposed changes are first run as controlled experiments. Kohavi and Thomke say dozens of revenue changes found this way each month raised Bing's revenue per search by 10% to 25% every year.
A company without those systems pays the full price for every test, and its managers soon ask for fewer of them. A small team can start with spreadsheets and a share of traffic, and move to shared tools as the number of tests grows.
081 min
Booking.com and testing as the default answer
Booking.com, the travel booking site, shows what happens when a company applies the mindset to almost everything. Thomke reports that Booking.com runs about 25,000 tests a year.
Stefan Thomke opens his 2020 article about it with a proposal. In December 2017, the design director at Booking.com proposed testing a far simpler home page than the one in use. The new page had a small window asking where the customer was going, the dates and the number of people, plus three simple options. It would remove content that the company had spent years refining.
That is the kind of idea that teams fight over for months. People who built the old page defend it, and people who like the new one attack its critics. At a company that tests by default, the argument ends in a test.
A test can only replace an argument if people can report a loss without blame. Teams that punish a failed test soon stop reporting them, and the data becomes a record of good news. This is why psychological safety comes before any tool.
092 min
When the experimental mindset breaks down
The mindset fails in four known ways. Each has a sign that the team can watch for.
A test that costs too much
Not every test is small. Microsoft spent more than $25 million on a third pane of social media content in Bing, and it produced almost no gain in engagement or revenue. Large bets still need a cheap test first, at the lowest cost that can answer the question. Tests are most useful when an idea is costly to build.
An average that hides a broken group
A total can mislead when it adds up groups that behave differently. In one Microsoft test, a JavaScript bug stopped Internet Explorer 7 users from clicking Bing results, and the overall result turned from positive to negative. A team that read only the total would have dropped a good idea. A team that also looked at each group found the bug. The surprising result in the Bing headline case worked the same way: an unusual number is first a reason to check the setup.
A measure that rewards the wrong thing
Choosing the number that decides success is hard. At Bing, a worse set of search results would make people search and click on ads more, so a team that counted only those two numbers would reward the damage. Bing's team therefore picked measures such as fewer queries per task and more tasks completed, and Kohavi and Thomke advise reviewing the choice every year.
Experiments used to avoid deciding
A team can test forever. Each test needs an end: before it starts, write down the date it will be read and the four choices that follow, which are to repeat, scale, change or drop. Stories about big wins, such as the Bing headline, are also told far more often than the many tests that found nothing. Remember survivorship bias when reading them.
Some decisions cannot be tested fairly at all, such as one hire or one merger. The mindset still helps there in a smaller form: write down what you expect, and compare it with what happened.
101 min
The experimental mindset and nearby ideas
Three ideas are often mixed up with the experimental mindset. The deciding fact for each is what it describes.
| Idea | What it is | What separates it |
|---|---|---|
| PsychologyGrowth mindset | ||
| ProductA/B test | ||
| Not in the library yetFailing fast |
112 min
Running a first experiment
Begin with one belief that a plan depends on, and write it in this form.
We believe [change] will lead to [result] for [group]. We will test it by [smallest version] for [time]. It passes if [measure] reaches [number set now]. If it passes, we will [next step]. If not, we will [other step].
Then set the edges of the test.
One change.
If two things change at once, the result cannot say which one mattered.
A time limit.
Choose it before you start, and read the result on that date, not earlier.
A limit on loss.
Decide the most money, time or goodwill you can lose, and keep the test inside it.
One measure that matters,
plus a look at each group, so that a broken segment shows up.
A written outcome.
Record what you expected, what happened and what you will do, including when the guess was right.
The last item is also how teams detach their ideas from the people who proposed them. When the record shows many guesses, wrong and right, no single guess is a verdict on its author.
Back to the sign-up team one last time. After the week, the shorter flow either clears the pass mark the team set or it does not. If it does, the team plans the next change and knows why. If it does not, five months and a rebuild are still available for a better idea.
121 min
Questions about the experimental mindset
Is the experimental mindset the same as trial and error?
No. Trial and error tries things and notes what happened. The experimental mindset states a guess and a pass mark first, so the result can say the guess was wrong. That makes a failed try easier to learn from.
How long should a personal experiment run?
Long enough to see the thing you are testing more than once. A daily habit needs at least a week or two. Choose the length before you begin, and read the result on that date.
Can I use it in a team that punishes mistakes?
Start with a small test whose only purpose is to answer a question, and report it the same way whether it wins or loses. Teams that punish failed tests stop reporting them, so the manager’s reaction matters most.
Does it mean I should never commit to a long plan?
No. A long plan is fine when its riskiest belief has been tested, or when the work cannot be split into steps. The mindset asks that the riskiest belief be tested first.
?4 questions
Questions people ask
Is the experimental mindset the same as trial and error?
How long should a personal experiment run?
Can I use it in a team that punishes mistakes?
Does it mean I should never commit to a long plan?
§4 sources
Sources
Show all 4 sourcesHide sources
Kohavi, R., & Thomke, S. (2017). The Surprising Power of Online Experiments. Harvard Business Review, September–October (reprint).
Thomke, S. (2020). Building a Culture of Experimentation. Harvard Business Review, March–April.
Bezos, J. (2017). 2016 Letter to Shareholders. Amazon.
Le Cunff, A.-L. (n.d.). A neuroscientist's guide to fixing your mind in 15 mins. Big Think interview.





