Back to Basics 10: Understanding User Behaviors (Part 1)

Welcome back for another installment of our “Back to Basics” blog series, focused on the ins and outs of conducting impactful user research. Some of our more recent posts in the series have explained advanced user experience research methods that can be used to determine how users interact with a product and whether a product design meets established usability guidelines.

In this article, we will discuss how researchers can better understand user behaviors and attitudes through moderated in-person usability study sessions with users.

Moderated In-Person Usability Testing

Moderated in-person usability studies can be used in the discovery phase of product development, to evaluate the current state of a system in the middle of the development cycle, to evaluate the final version of a system at the end of the development cycle, or to gain insights into users’ interactions with a competitor’s system.

Talking to potential users of a product or service is the single most important thing that researchers can do to improve its overall usability. Why? Without talking to users, you will never be able to uncover the true reasons that they behave the way they do when using your technology. These underlying causes are the “holy grail” for usability and will provide the most valuable insights when designing your products.

What is it?

Usability testing may be implemented early on to inform the design process. Usability testing – arguably one of the most tried and true UX methods – essentially involves observing users as they attempt to perform specific tasks. But don’t wait until you have a fully functioning product to get started; low-fidelity prototypes can be used to gather tons of information from your users. You’ll be amazed at the level of understanding you can walk away with after a session of watching real or intended users interact with your product or interface.

Usability testing is the most important tool in the UX researcher’s toolbox because it allows them to uncover users’ needs without trying to infer where problems will occur. To achieve this, researchers directly observe (and interact with) a number of representative users performing a set of realistic tasks. While moderating the sessions with participants, researchers typically collect both behavioral (observable) and attitudinal (unobservable, from rating scales, etc.) data to cast a wider net and provide more robust support for their findings. Ultimately, the goal of usability testing is to collect defendable data for design recommendations, so that designers can confidently address them before the product is deployed. These finding typically fall into one of three categories: effectiveness (i.e., the user’s ability to perform a task), efficiency (i.e., the resources required to perform a task, such as time and effort), and satisfaction (i.e., how pleasant it is for the user to use a system). If the researcher has done his/her job right, all barriers to quickly and effectively using the interface will be removed, and the users will actually enjoy (rather than tolerate) their experience.

Strengths

• Provides rich data and actionable feedback from real end users
• Fundamental means of user testing backed by numerous references and resources
• Can include both qualitative and quantitative data, and allows for subtle behavioral cues (e.g., eye movement, body language) to be observed
• Provides reliable and defendable data, as well as qualitative support that brings the issues to life (e.g., user quotes)

Weaknesses

• Can be more costly and resource-intensive due to multiple study participants and laboratory needs as compared to other methods (e.g., heuristic evaluation)
• Scheduling issues can arise (e.g., no-shows, inclement weather, difficulty recruiting the right users)
• Design recommendations take longer to deliver; data needs to be synthesized and key findings need to be deduced

How to do it?

Usability tests are usually conducted with a moderator present in the

testing environment who ensures that the testing sessions run smoothly, builds rapport with the user to encourage the most useful feedback, and uses their knowledge of qualitative and quantitative methods to ensure valid data collection.

However, for those familiar with usability testing, you may have come across unmoderated testing and wondered if it was a good idea or not. Unmoderated usability testing involves users completing usability sessions via a web app in their own environments. If you are considering this approach, you need to be aware of a number of pitfalls that can easily render your study data invalid. First, users who answer questions in the absence of a moderator can easily misinterpret the meaning of the question, or the instructions for using the interface, without anyone ever catching it. Second, participants are much less likely to think carefully about the task at hand when left to their own devices, and could breeze through the session inattentively to pick up the incentive check. Third, technical challenges that inevitably arise from conducting remote sessions may render some or all of your data useless. Finally (and most importantly) carefully executed follow-up and probing questions that drill down into the underlying cause of behaviors and attitudes are not possible in unmoderated sessions. As previously mentioned, uncovering these “why” questions is what makes usability testing so incredibly valuable, so to eliminate this all together means that you are not capitalizing on the valuable resource you have available – your users’ behaviors, needs, and expectations.

To get the most out of your moderated user testing, it is helpful to have already conducted an expert review to remove glaring issues. In addition, be sure to complete the following steps:

1. Clearly define your test objectives (e.g., do you care to know how valuable the product is apart from its ease of use?)
2. Create 4-6 task scenarios for participants to complete
3. Choose data collection procedures ahead of time so that the data are structured appropriately (e.g., answer choices can be categorical, ordinal, or continuous, which has implications on data analysis and interpretation on the back end)
4. Choose your data collection procedures (e.g., Likert-type rating scales, card sorting, structured interview questions, typing tasks)
5. Prepare the test materials (e.g., participant packet, data collection sheets), test protocol (including scripts, researcher instructions, and interview questions), and the test environment (e.g., laptop, prototype, lighting, cameras)
6. Qualify and recruit test participants from the target populations of interest (representative users)
7. Conduct usability sessions in a controlled environment and collect data real-time (e.g., moderator notes) and for review (e.g., video and audio)
8. Data is analyzed, and test results and design recommendations are communicated

The goal of a usability test is always to uncover the most design flaws as possible within a limited amount of time, although this doesn’t mean that a huge number of representative users are always necessary. In fact, 5 users has been a popular number for in-person moderated usability testing, as some studies have empirically determined that just 5 test subjects are able to uncover around 80% of usability problems. However, this only applies to target populations that are homogenous, which is rarely the case. More often than not, you want to test with participants from various demographics (e.g., age, experience with product, level of education) that could be using your products. Given this distinction, 10-20 participants should allow you to capture the varying perspectives of your entire target population.

Usability testing is typically not a “one and done” situation. In order to refine your product to achieve the optimal experience for your users (and adoption in the market), iterative testing is required. Just as the name implies, you test a group of users, make design changes, and then repeat until you are satisfied with the results. Budget constraints and product launch dates will inevitably play a role in how long testing will be allowed to go on, so keep in mind that the law of diminishing returns also plays a role. The majority of issues will be discovered in the first couple of iterations, at which point you will be fine-tuning your design or testing out potential additional features.

What is the output?

Testing sessions are often audio and video-recorded by professional software and mounted cameras, which can help to more precisely capture certain usability metrics (e.g., time spent completing a task, number of errors, facial expressions, and user comments). Screen and room recordings can be shared with clients and stakeholders who are able to livestream the user sessions or watch the recordings at a later time. The moderator will typically take detailed notes during the user session, documenting both their observations of user behaviors and reactions, as well as user comments.

Typically, several types of data will be collected, including some qualitative and some quantitative. During the data analysis process, themes and insights are synthesized from the qualitative data, and calculations are made to summarize any quantitative metrics. The final usability report may include the usability problems discovered throughout the study, the associated severity levels of the problems, and recommendations for suggested fixes or design improvements. All of the cited findings and recommendations will be rooted in the user data to allow for meaningful improvements to be made.

Final Thoughts

Testing design prototypes with real end users yields rich insights and reveals the ways in which real people interact with a system. In-person moderated usability testing is commonly employed iteratively from the very beginning stages of product development to the final stages of product deployment to identify user frustrations and preferences. Usability testing establishes the “whys” behind design preferences and usability issues.