Back to Basics 2: Selecting the Right User Research Method

If you’ve been following along with us, you’ll know that the Human Interfaces team has been using the extra time we have as much of the country is still emerging from COVID-19 shutdowns to take a look back at the basics of user testing (and UX research more broadly). Last week, we explored what user testing is and why you might want to conduct such an activity.

At Human Interfaces, we believe that no two research efforts look exactly alike. The success of a given research project is largely dictated by posing the right questions to the right users, and doing so via the appropriate method. In subsequent installments of this series, we will provide tips for selecting and recruiting the right users. However, if the right research method is not selected, none of that will matter.

When it comes to user testing, there are two primary categories of methods that are commonly used: (1) Field-based and (2) Lab-based. Each type of user test has its own place, with specific benefits and limitations.

Field-based or lab-based?

Some of the primary considerations for choosing between field-based and lab-based methods include:

• User environment – The degree to which one’s environment changes their interaction with the product
• Disrupting the user – The level of disruption that can be tolerated during testing
• Practicality of access – Researcher accessibility to a user’s natural environment
• Other limitations – Time & resources required to conduct the study

In this installment of our Back to Basics series, we will discuss the nature of each approach, as well as key considerations in deciding which type of user testing may be most appropriate in various circumstances. Ultimately, what you have to decide is whether to take your research to the user, or bring the user to your research.

Taking the research to your user

There are times when the best approach to user testing means going into the field (either literally or figuratively). Some of the most common field-based user testing methods include contextual inquiries and longitudinal or “live with” studies, often captured via journals and diaries.

Central to all of these field methods is the idea of observing users in their “natural habitat”. Rather than bringing representative users into a “lab” setting and observing product use in a controlled scenario, field methods seek to understand how users interact with a product in real environments. This includes considering how real contexts impact what features should be included in a product and how they can be best implemented in the interface. It also provides a way to elicit unique use cases of the product that would not be possible to capture if the user was not in their natural environment.

Part of the power of field methods is the ability to understand what people actually do with your product and how they go about those tasks. Because these observations happen in the wild, the researcher can also understand aspects of the environment that impact what they do and how they do it. Such an understanding can impact design decisions that better account for real-world usage. For example, if a touch screen interface is being considered, designers must keep in mind that users may be wearing gloves while using the device in a cold environment.

Contextual inquiries can provide a first-hand view of how a system works, providing insight into the processes, tasks, pain points and challenges involved in that system. In this method, a researcher observes users interacting with a product or system within a natural environment. Often, this approach also sheds light on workflows and work-arounds that users come up with to deal with limitations of a product or system (e.g., ad-hoc spreadsheets or post-it notes being used to account for a poor information flow in an interface).

The types of insights provided by contextual inquiries can often be used to inform next-gen product development as interface limitations and opportunities are exposed in the context of real-world usage.

Longitudinal field-methods, such as journal or diary studies, take a similar approach of understanding product interactions via real-world usage. A primary difference is that, as compared to a contextual inquiry where the researcher is likely to be present, diary studies often rely on a higher-level of participant self-reporting. One best practice to help ensure engagement is to provide the diary in digital format and alert the participant at regular intervals that an entry is required.

A common approach to a longitudinal study is to provide a product to participants and have them “live with” the product over a period of time while keeping a journal of what they use the product for (tasks), how they go about those tasks, and what they like and do not like about how the product works or functions. Such research is often performed as beta testing, prior to product launch. It is also common to use this type of method as a post-launch evaluative mechanism to determine whether or not the design goals for the interface were achieved or supported during ongoing use.

Pros & cons of field-based methods

In all cases, field-based methods for user testing have the potential of delivering rich qualitative insights into how your users interact with your product, system or interface. These insights often speak to the bigger picture, or how your product fits in to the user’s ecosystem.

These field-methods can be particularly relevant if dealing with highly specialized user types, who might be difficult to recruit for lab-based studies, as well as when the context is critical to how the product is used and cannot be easily simulated in a lab environment.

Field-based methods carry with them particular challenges from a research perspective. For contextual inquiries, for example, it is not always easy to find users (or businesses) who are open to having a researcher invade their space. By its very nature, these methods are also less controlled than lab-based user testing. While you can attempt to focus on particular tasks or areas of an interface, it is not guaranteed that you will actually capture that behavior in the real world (however, this can be insightful in and of itself).

In longitudinal field research such as journal or diary studies, participant attrition can also be a big challenge, requiring significant investment in time and money to keep participants engaged.

Field-based methods generate large amounts of qualitative and quantitative data. Because there is an inherent lack of control in these methods, analyzing the data and drawing conclusions across sessions or participants can also be more of a challenge. In many cases, however, these challenges may be outweighed by the potential to gather rich insights into your users, the environments they operate in, and how those environments might dictate design decisions for your product or interface.

Bringing the user to your research

As with field-based methods, conducting research in the lab brings with it specific benefits and limitations. Probably the most common form of lab-based testing is a traditional task-based usability study.

At its core, task-based usability studies involve presenting representative users with tasks or scenarios associated with a product or interface, having those users attempt those tasks, and identifying issues and opportunities for refinement in the interface based on both observation and discussion.

While this type of user testing can be conducted in a range of environments, whether a formal research facility, ad-hoc venue or remotely, it is inherently less focused on the context or environment than field-based methods. It is also typical that task-based usability studies are more controlled (and potentially constrained) than field-based methods such as a contextual inquiry.

The primary focus of task-based usability studies is on identifying usability issues. Based on observations and participant feedback, recommendations for design changes can be made to mitigate or eliminate the issues. This type of user testing provides for a controlled environment where specific areas of a product or interface are investigated (often via tasks). Whereas contextual inquiries are often used to inform product design from a blank-slate, task-based user testing is inherently a refining mechanism.

Pros & cons of lab-based methods

Task-based usability studies can be conducted at virtually any point within development, using varying fidelities of prototypes or functioning interfaces. Usability studies can also be performed rapidly and iteratively. A small number of participants, often 6-8, can be run through the study and yield a high-percentage of potential issues, for which fixes can be prioritized into the development cycle.

While control is one of the greatest benefits of lab-based usability studies, it also represents a limitation. For example, while task-based testing can tell you whether or not it is easy or intuitive to use a given feature in your product, because you are controlling the tasks, it is not as good at informing whether or not it will actually be used in the real-world for that purpose.

Whether you decide to pursue a field-based approach, or focus more on lab-based user testing, it is important to understand the benefits and limitations of each approach. As we said before, no two research efforts look exactly alike. Sometimes you may want to explore all possible ways a new device could be used, so field-based methods would be most appropriate. Other times, you may want to identify pain points that take away from a user’s ability to carry out a specific task, in which case lab-based methods should be used. Choosing the right method is an essential step in ensuring that your findings will be relevant, meaningful, and actionable.

In our next installment of our Back to Basics series, we will focus on the importance of, and strategies for, identifying, selecting and recruiting the right participants to be part of your user testing. In the meantime, drop us a note and tell us about experiences you’ve had in selecting and executing the “right” method for your research goals!