011 min
Why Field Testing exists
Picture a team that builds a delivery app for couriers and has not yet used Field Testing. In the office, every test passes. The phone has a strong signal, the screen is clean, and the tester holds it with both hands. Then a courier opens the app in a stairwell with no signal, in bright sun, wearing gloves, while a customer waits at the door. The confirm button fails, and the courier writes the delivery on paper.
The team did not test badly. Nobody in the office could have copied that stairwell. The usual answer is Field Testing: instead of rebuilding the courier’s world in a lab, the team gives the app to couriers and watches what happens.
021 min
Two meanings of Field Testing
The courier story shows Field Testing as product teams use the words, but software engineers use them for a second practice. Product and research teams mean a late round in which people from the target market use a near-final product on their own, while the team collects feedback over an extended period. Software engineers mean something narrower. They mean running test cases inside software that is already installed and in use, so that failures can show up before the user meets them. Both meanings share one idea: the real setting is where the product gets judged. This article follows the first meaning, and it borrows the second to explain why a lab cannot find everything.
031 min
How a Field Testing round runs
A Field Testing round starts when the product is almost ready, so the courier team first finishes its earlier rounds. Alpha testing is the first round, run inside the company. Beta testing gives a near-final version to outsiders, usually with features to try. Field Testing comes after both. The trade also calls this stage limited availability, early release or soft launch.
Now the team hands the build to a group of couriers and asks them only to do their normal work. There is no task list. The team’s job changes from guiding to watching.
Two kinds of information come back. Usage data shows which features people use, how often, and where the app crashes. What the couriers say explains why. Many companies also use this stage to try live marketing material and the customer support process before the general release, because the product is stable enough that those systems can be judged too.
041 min
Field Testing vs beta testing
Field Testing is easy to mix up with beta testing, because both give an almost finished product to outsiders. The difference is who steers. In a beta test, the team points testers at specific features and asks for feedback on them, while in Field Testing it lets testers loose and sees which features they pick up on their own.
| Method | Who steers | Main question |
|---|---|---|
| This pageField Testing | ||
| Not in the library yetBeta testing | ||
| ResearchUsability Testing | ||
| ResearchField study |
Usability testing can also happen in the field, as the dairy case below shows. A field study has a different target. It watches how people work today, usually before there is a product to judge, while Field Testing judges a product that already exists.
051 min
What 119 bug reports showed about Field Testing
The courier gap is invented, but researchers have counted gaps like it in real software. In 2017, Luca Gazzola, Leonardo Mariani, Fabrizio Pastore and Mauro Pezzè studied 119 bug reports from three software projects: 63 for Eclipse, 26 for OpenOffice and 30 for Nuxeo. For each problem a user hit, they asked whether a team could have found the cause before release by testing in-house.
They found that 70% of the faults behind those problems were intrinsically hard to find before release. Of the problems that were hard to find in-house, 78% could be seen only with resources that exist in the field. Examples of such resources are new plug-ins and files that users add to their own installation. The authors call faults like these field-intrinsic. They describe the work as initial evidence from three projects, not a general law.
062 min
Four conditions that Field Testing covers and a lab cannot
Counting the gap is one step, and the next question is why a lab cannot close it. Gazzola’s team sorted the field-only faults by what an in-house test would need and lacks. Combinatorial explosion was the most frequent cause of these faults, and conditions that cannot be created in-house were the least common.
Combinatorial explosion
The app and its setting can be fully known and still be too large to test. Settings, inputs, devices and system versions combine in more ways than a team can run. The courier team cannot test every phone, every signal strength and every stored delivery in every order.
Unknown environment conditions
Some faults appear only with facts about the user’s setting that the team cannot know before release. Which other apps the courier has installed is one such fact.
Unknown application conditions
Some inputs can be run in-house, but the fault depends on a state of the app that the team did not know existed. No test suite covers a case nobody has imagined.
Conditions that cannot be created in-house
A few faults need a setting the team simply cannot build. These were the rarest in the study.
Silent failures
A failure is silent when the program does not notice the problem and the user cannot tell either without knowing what the right result should be. Of the 83 failures the team analysed in detail, 44 were silent. So a field test that only counts crashes would miss most of what went wrong. Testers need a way to say what looked off, and the team needs a sense of the right result to compare against.
071 min
What Field Testing showed at a dairy repair centre
Faults are one thing a lab misses, and the way people really work is another. Steven Hoober, who writes the Mobile Matters column on UXmatters, met the second by accident. He was running a pilot test of a mobile app at a repair centre for a local dairy. In one session, a participant became very suspicious of the mileage figures that the app showed.
When the sessions ended, Hoober asked the shop supervisor about the doubt. The supervisor showed him a back office. Behind a desk lay a heap of electronic boxes gathering dust. The supervisor said the boxes were data loggers that the company had taken off its trucks because they were not accurate. The office walls were lined with cubbyholes, like a mail room.
Nothing in the test plan asked about any of this. The supervisor’s answer does not prove where the app’s numbers came from. It does show that the company had already stopped trusting some of its own data devices, which is a reason for doubt that no lab could have supplied. Hoober calls the quick, informal observation he does in cases like this lean ethnography, and he says that field sessions with a team of observers are rare for him.
081 min
When a realistic lab does the work of Field Testing
Field Testing costs more than a lab session, so one research team asked whether the extra work pays off. In 2004, Jesper Kjeldskov, Mikael Skov, Benedikte Als and Rune Høegh tested MOBILEWARD, a context-aware handheld patient record for hospital nurses. They ran the test twice. Six nurses used the system in a realistic laboratory setup, and another six used it during real work at the Hospital of Frederikshavn.
The laboratory found the same problems as the field, except for one. The laboratory round took 34 man-hours. The field round took 65 man-hours. The authors’ conclusion was plain: for this system, the field added little. Their paper’s title asks whether it was worth the hassle.
What the hospital result does not say
The result covers one system, with six nurses in each setting, and the laboratory was built with the ward in mind. A lab that ignores the setting may give a different answer. Another team, Baillie and Schatz, found that testers rated the system higher in the field, and the 2004 paper had not examined that difference. The two results fit one rule. A field round earns its cost when the setting changes what you need to learn, and not when a good copy of the setting is cheap to build.
091 min
The costs and limits of Field Testing
The 65 hours in that study point to the first cost of Field Testing, which is time. A field round takes longer to plan, run and analyse than a lab session. Four other limits matter as much.
- Less control. Signal, weather, noise and interruptions differ for every tester, so two results are hard to compare and a failed run is hard to repeat.
- Observer effect. People act differently when they know someone is watching, a pattern known as the Hawthorne effect. A researcher sitting beside the tester can trigger it. Usage data collected in the background draws less attention than a person in the room.
- Risk to the product’s name. Testers who meet bugs may form a poor opinion before launch, so the product needs to be free of major problems first.
- Privacy and secrecy. Testers use the product in homes and workplaces, and the product is not public yet. The team needs consent for what it records and an agreement about what testers may share.
101 min
How to run Field Testing well
A team that accepts these costs can still run Field Testing well, and the order of the steps matters.
Write the question first.
Name the decision the round will inform, such as which features to keep, and the conditions you worry about, such as weak signal.
Recruit people who match your buyers.
Give them their real work, not a task list, and ask for their consent to be observed or recorded.
Build in the instruments before release.
Crash reports, counts of feature use and a short way for testers to say what went wrong all have to exist on day one.
Pilot the field test itself.
Hoober advises running a pilot as though it were a real session, so the team has practised its note-taking, its route and its equipment before the real round.
Debrief soon after each session,
while testers still remember what they did.
Turn each field-found fault into a lab test.
A fault that can be repeated at a desk can be checked after every later change.
If you want to follow one tester through a full working day, a contextual inquiry fits that goal better than a general release round.
111 min
When to use Field Testing
These steps lead to one last decision, which is whether a given product needs Field Testing at all. Use it when the product’s success depends on conditions you cannot copy, such as signal, noise, interruptions and the other apps on the device. Use it only after the known problems are fixed in earlier rounds, because testers who hit major bugs stop using the product and teach you nothing about everyday use.
Skip it for a rough idea. A prototype test is the right tool at that stage. Skip it too when you need a controlled comparison of two designs, because uncontrolled settings blur the difference. An A/B test answers that question better. And when the setting is easy to copy, a realistic lab may give you the same list of problems for much less effort.
?5 questions
Questions people ask
How many testers does Field Testing need?
Can Field Testing be done remotely?
Can Field Testing be automated?
What tools does Field Testing use?
Does Field Testing need a finished product?
§7 sources
Sources
Show all 7 sourcesHide sources
Luca Gazzola, Leonardo Mariani, Fabrizio Pastore and Mauro Pezzè, "An Exploratory Study of Field Failures" (2017). arxiv.org
Luca Gazzola, "Field Testing of Software Applications", doctoral symposium paper (2017). arxiv.org
Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als and Rune T. Høegh, "Is It Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field", Mobile HCI 2004. vbn.aau.dk
Jesper Kjeldskov and Mikael B. Skov, "Was it Worth the Hassle? Ten Years of Mobile HCI Research Discussions on Lab and Field Evaluations", MobileHCI 2014. vbn.aau.dk
Steven Hoober, "Succeeding with Field Usability Testing and Lean Ethnography", UXmatters (7 November 2016). uxmatters.com
Centercode, "What is Field Testing?", glossary entry. centercode.com
Austin Meyer, "Field Testing vs Beta Testing: Key Differences Explained", Centercode (updated 18 July 2023). centercode.com





