Field Testing

Also called field test, field trial. Also in AI & Engineering, Product

Quick answer

Field testing is a way of checking a nearly finished product by letting real users run it in their own settings, instead of in a lab or test environment. The team watches what happens under real conditions, such as weak signal, interruptions and different devices. It finds problems that in-house testing misses and usually follows alpha and beta testing.

~20 sec read

011 min

Why Field Testing exists

Picture a team that builds a delivery app for couriers and has not yet used Field Testing. In the office, every test passes. The phone has a strong signal, the screen is clean, and the tester holds it with both hands. Then a courier opens the app in a stairwell with no signal, in bright sun, wearing gloves, while a customer waits at the door. The confirm button fails, and the courier writes the delivery on paper.

The team did not test badly. Nobody in the office could have copied that stairwell. The usual answer is Field Testing: instead of rebuilding the courier’s world in a lab, the team gives the app to couriers and watches what happens.

021 min

Two meanings of Field Testing

The courier story shows Field Testing as product teams use the words, but software engineers use them for a second practice. Product and research teams mean a late round in which people from the target market use a near-final product on their own, while the team collects feedback over an extended period. Software engineers mean something narrower. They mean running test cases inside software that is already installed and in use, so that failures can show up before the user meets them. Both meanings share one idea: the real setting is where the product gets judged. This article follows the first meaning, and it borrows the second to explain why a lab cannot find everything.

031 min

How a Field Testing round runs

A Field Testing round starts when the product is almost ready, so the courier team first finishes its earlier rounds. Alpha testing is the first round, run inside the company. Beta testing gives a near-final version to outsiders, usually with features to try. Field Testing comes after both. The trade also calls this stage limited availability, early release or soft launch.

Now the team hands the build to a group of couriers and asks them only to do their normal work. There is no task list. The team’s job changes from guiding to watching.

Two kinds of information come back. Usage data shows which features people use, how often, and where the app crashes. What the couriers say explains why. Many companies also use this stage to try live marketing material and the customer support process before the general release, because the product is stable enough that those systems can be judged too.

041 min

Field Testing vs beta testing

Field Testing is easy to mix up with beta testing, because both give an almost finished product to outsiders. The difference is who steers. In a beta test, the team points testers at specific features and asks for feedback on them, while in Field Testing it lets testers loose and sees which features they pick up on their own.

MethodWho steersMain question
This pageField TestingWhich features do people really use, and what breaks?Who steers: The tester, using the product freely
Not in the library yetBeta testingIs each feature ready to launch?Who steers: The team, which points testers at features
ResearchUsability TestingCan people finish the tasks?Who steers: A moderator, who sets tasks
ResearchField studyHow do people work today?Who steers: The observer, who follows the participant

Usability testing can also happen in the field, as the dairy case below shows. A field study has a different target. It watches how people work today, usually before there is a product to judge, while Field Testing judges a product that already exists.

051 min

What 119 bug reports showed about Field Testing

The courier gap is invented, but researchers have counted gaps like it in real software. In 2017, Luca Gazzola, Leonardo Mariani, Fabrizio Pastore and Mauro Pezzè studied 119 bug reports from three software projects: 63 for Eclipse, 26 for OpenOffice and 30 for Nuxeo. For each problem a user hit, they asked whether a team could have found the cause before release by testing in-house.

They found that 70% of the faults behind those problems were intrinsically hard to find before release. Of the problems that were hard to find in-house, 78% could be seen only with resources that exist in the field. Examples of such resources are new plug-ins and files that users add to their own installation. The authors call faults like these field-intrinsic. They describe the work as initial evidence from three projects, not a general law.

062 min

Four conditions that Field Testing covers and a lab cannot

Counting the gap is one step, and the next question is why a lab cannot close it. Gazzola’s team sorted the field-only faults by what an in-house test would need and lacks. Combinatorial explosion was the most frequent cause of these faults, and conditions that cannot be created in-house were the least common.

Combinatorial explosion

The app and its setting can be fully known and still be too large to test. Settings, inputs, devices and system versions combine in more ways than a team can run. The courier team cannot test every phone, every signal strength and every stored delivery in every order.

Unknown environment conditions

Some faults appear only with facts about the user’s setting that the team cannot know before release. Which other apps the courier has installed is one such fact.

Unknown application conditions

Some inputs can be run in-house, but the fault depends on a state of the app that the team did not know existed. No test suite covers a case nobody has imagined.

Conditions that cannot be created in-house

A few faults need a setting the team simply cannot build. These were the rarest in the study.

Silent failures

A failure is silent when the program does not notice the problem and the user cannot tell either without knowing what the right result should be. Of the 83 failures the team analysed in detail, 44 were silent. So a field test that only counts crashes would miss most of what went wrong. Testers need a way to say what looked off, and the team needs a sense of the right result to compare against.

071 min

What Field Testing showed at a dairy repair centre

Faults are one thing a lab misses, and the way people really work is another. Steven Hoober, who writes the Mobile Matters column on UXmatters, met the second by accident. He was running a pilot test of a mobile app at a repair centre for a local dairy. In one session, a participant became very suspicious of the mileage figures that the app showed.

When the sessions ended, Hoober asked the shop supervisor about the doubt. The supervisor showed him a back office. Behind a desk lay a heap of electronic boxes gathering dust. The supervisor said the boxes were data loggers that the company had taken off its trucks because they were not accurate. The office walls were lined with cubbyholes, like a mail room.

Nothing in the test plan asked about any of this. The supervisor’s answer does not prove where the app’s numbers came from. It does show that the company had already stopped trusting some of its own data devices, which is a reason for doubt that no lab could have supplied. Hoober calls the quick, informal observation he does in cases like this lean ethnography, and he says that field sessions with a team of observers are rare for him.

back officedata loggerscubbyholes
Look at what fills the office: devices the company stopped using on one side and cubbyholes on the walls on the other.

081 min

When a realistic lab does the work of Field Testing

Field Testing costs more than a lab session, so one research team asked whether the extra work pays off. In 2004, Jesper Kjeldskov, Mikael Skov, Benedikte Als and Rune Høegh tested MOBILEWARD, a context-aware handheld patient record for hospital nurses. They ran the test twice. Six nurses used the system in a realistic laboratory setup, and another six used it during real work at the Hospital of Frederikshavn.

The laboratory found the same problems as the field, except for one. The laboratory round took 34 man-hours. The field round took 65 man-hours. The authors’ conclusion was plain: for this system, the field added little. Their paper’s title asks whether it was worth the hassle.

What the hospital result does not say

The result covers one system, with six nurses in each setting, and the laboratory was built with the ward in mind. A lab that ignores the setting may give a different answer. Another team, Baillie and Schatz, found that testers rated the system higher in the field, and the 2004 paper had not examined that difference. The two results fit one rule. A field round earns its cost when the setting changes what you need to learn, and not when a good copy of the setting is cheap to build.

laboratory34man-hoursfield65man-hours
Both bars share one scale, so the taller bar shows how much more effort the field setting took.

091 min

The costs and limits of Field Testing

The 65 hours in that study point to the first cost of Field Testing, which is time. A field round takes longer to plan, run and analyse than a lab session. Four other limits matter as much.

  • Less control. Signal, weather, noise and interruptions differ for every tester, so two results are hard to compare and a failed run is hard to repeat.
  • Observer effect. People act differently when they know someone is watching, a pattern known as the Hawthorne effect. A researcher sitting beside the tester can trigger it. Usage data collected in the background draws less attention than a person in the room.
  • Risk to the product’s name. Testers who meet bugs may form a poor opinion before launch, so the product needs to be free of major problems first.
  • Privacy and secrecy. Testers use the product in homes and workplaces, and the product is not public yet. The team needs consent for what it records and an agreement about what testers may share.

101 min

How to run Field Testing well

A team that accepts these costs can still run Field Testing well, and the order of the steps matters.

  1. Write the question first.

    Name the decision the round will inform, such as which features to keep, and the conditions you worry about, such as weak signal.

  2. Recruit people who match your buyers.

    Give them their real work, not a task list, and ask for their consent to be observed or recorded.

  3. Build in the instruments before release.

    Crash reports, counts of feature use and a short way for testers to say what went wrong all have to exist on day one.

  4. Pilot the field test itself.

    Hoober advises running a pilot as though it were a real session, so the team has practised its note-taking, its route and its equipment before the real round.

  5. Debrief soon after each session,

    while testers still remember what they did.

  6. Turn each field-found fault into a lab test.

    A fault that can be repeated at a desk can be checked after every later change.

If you want to follow one tester through a full working day, a contextual inquiry fits that goal better than a general release round.

111 min

When to use Field Testing

These steps lead to one last decision, which is whether a given product needs Field Testing at all. Use it when the product’s success depends on conditions you cannot copy, such as signal, noise, interruptions and the other apps on the device. Use it only after the known problems are fixed in earlier rounds, because testers who hit major bugs stop using the product and teach you nothing about everyday use.

Skip it for a rough idea. A prototype test is the right tool at that stage. Skip it too when you need a controlled comparison of two designs, because uncontrolled settings blur the difference. An A/B test answers that question better. And when the setting is easy to copy, a realistic lab may give you the same list of problems for much less effort.

?5 questions

Questions people ask

How many testers does Field Testing need?

No source gives one number for Field Testing. Guides to field studies, a related method, name 4 to 8 or 10 to 12 participants. Choose enough testers to cover each setting that worries you.

Can Field Testing be done remotely?

Yes, in part. Crash reports and usage data already arrive from a distance, and Nielsen Norman Group describes remote field studies that use video when a site visit is not possible. A remote view shows less of the room.

Can Field Testing be automated?

Not the field part, because its value comes from real people using the product freely. The problems it finds can become automated regression tests afterwards, so the same fault is caught on every later release.

What tools does Field Testing use?

Crash reporting, counts of feature use and network records are the usual set. Session replay shows what a tester saw on screen. Teams then reproduce field bugs in the lab with tools that slow or cut the network.

Does Field Testing need a finished product?

It needs an almost finished one. Major defects should be gone before the round starts, so the team learns how people use the product day to day and not just where it breaks.

§7 sources

Sources

Show all 7 sources
  1. Luca Gazzola, Leonardo Mariani, Fabrizio Pastore and Mauro Pezzè, "An Exploratory Study of Field Failures" (2017). arxiv.org

  2. Luca Gazzola, "Field Testing of Software Applications", doctoral symposium paper (2017). arxiv.org

  3. Jesper Kjeldskov, Mikael B. Skov, Benedikte S. Als and Rune T. Høegh, "Is It Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field", Mobile HCI 2004. vbn.aau.dk

  4. Jesper Kjeldskov and Mikael B. Skov, "Was it Worth the Hassle? Ten Years of Mobile HCI Research Discussions on Lab and Field Evaluations", MobileHCI 2014. vbn.aau.dk

  5. Steven Hoober, "Succeeding with Field Usability Testing and Lean Ethnography", UXmatters (7 November 2016). uxmatters.com

  6. Centercode, "What is Field Testing?", glossary entry. centercode.com

  7. Austin Meyer, "Field Testing vs Beta Testing: Key Differences Explained", Centercode (updated 18 July 2023). centercode.com

Keep reading

More from Research

All of Research
All of Research

Missing a concept?

Tell us what to write next. We’ll email you when it’s live.

Several? Separate them with commas.

Only to tell you when it’s live.

How we use your email: privacy.