Thoughts on Unmoderated Studies: Trendy but Is It Appropriate?

Here at Human Interfaces, we hold by the belief that one of the main keys to successful research is properly matching the method being used to the objectives and questions at hand. We spend a considerable amount of time working with clients to understand those objectives, sorting through our toolbox to decide which tools are best suited to answer their research questions, and figuring out how to apply those tools in an efficient and effective manner.

You can get a sense for just how much time we spend thinking about the ins and outs of different methodologies by checking out other topics we’ve written about on our blog. In the last couple years, we’ve covered everything from a general overview of different methods and when to use them, taken a deep dive into field methods such as Contextual Inquiries, and explored how to get ahead of the usability curve with expert and heuristic reviews (including some fun real-world examples). Heck, we’ve even published an entire eBook digging into strategies for success in usability and user testing (check it out if you’re interested).

One recent trend we are starting to see from our clients is an interest in exploring the possibility of unmoderated user testing, and the potential uses of this method. You’re probably familiar with, or at least have heard of, sites such as UserTesting and UserZoom. Unmoderated user testing certainly has its benefits. With it, you have the potential to:

• Quickly collect feedback and data without the need for a physical facility or professional moderator
• Reach across geographical regions with quant-level sample sizes
• Achieve both qualitative and quantitative results with a relatively low touch

Before deciding to go this route, however, it is important to understand exactly what unmoderated testing is… and equally as important, what it is not. Understanding the benefits, limitations and potential pitfalls of unmoderated user testing can help you decide if it is the right method for you. Let’s look at a couple of key considerations to help you decide if unmoderated user testing might be right for you.

Do you need to know the “why’s”?

With moderated qualitative user testing, one of the most powerful tools in the researcher’s toolbox is one simple question – “Why?” Often, the ability to ask follow-up questions based on an observed behavior or a given piece of user feedback can yield more insights than the original observation or feedback itself. This is particularly helpful when your testing is focused on informing design decisions and iterating towards a positive user experience. During early and mid-stage development, a small sample, moderated usability study usually works best.

“Often, the ability to ask follow-up questions based on an observed behavior or a given piece of user feedback can yield more insights than the original observation or feedback itself.”

Unmoderated testing, however, may fit the bill if you already understand the “why’s”, and are more interested in validating or benchmarking your interface. The larger sample sizes achievable through unmoderated testing can also make it a great platform for A/B testing, where you are more interested in an outcome than in understanding the “Why’s” behind that outcome.

Do you have specialized users or a complex tasks in the interface?

Typically, with online unmoderated testing platforms, your ability to define and control participant screening criteria is more limited than when using a traditional qualitative research recruiting service. It’s much more similar to fielding a survey via a panel than it is conducting targeted recruiting for a traditional usability study. If you’re testing a simple e-commerce site, this might not matter much. However, if your interface is designed for highly specialized users, executing an online unmoderated test will be much more problematic.

“Without a moderator to clarify task intent or dynamically adjust for participant uncertainty or errors, complex interface tasks are problematic with unmoderated testing”

Similarly, if your interface involves complex tasks, unmoderated testing may not be for you. In practice, properly executing an unmoderated test requires you to present clear-cut tasks that have straightforward goals and easily-communicated ending points. Without a moderator to clarify task intent or dynamically adjust for participant uncertainty or errors, complex interface tasks are problematic with unmoderated testing.

Carefully consider efficiencies

One of the big potential benefits of unmoderated online testing is efficiency, both in terms of time and resources. Depending on the nature of your research questions, however, it may not be as efficient as it seems. For example, if your goal is gaining qualitative feedback on your website, identifying potential usability issues or trying to drive design changes, someone is going to have to watch and analyze the session recordings. That quick “hands-off” user test just became a lot more labor intensive. Now you’re in a position where you still need to have a resource “virtually” attend the test sessions, but without the ability to ask the “why’s” which can lead to so many of the true user experience insights.

Know your objectives before you decide which method to use

All of this is not to say that unmoderated testing doesn’t have its place – but rather that it is not a Panacea that can simply replace traditional user experience research. It can, however, serve as a valuable supplement, as long as you understand its limitations. One way I’ve seen it used effectively is as a validation tool for design decisions that have been driven by traditional user experience research. In these cases, you might gain your deep insights through moderated test sessions, come up with a new design, and validate it via online testing (think A-B testing).

“As with all methods, the important thing is to make sure that the approach matches your research questions and objectives.”

As with all methods, the important thing is to make sure that the approach matches your research questions and objectives. In the case of unmoderated testing, this means understanding limitations you have in the control of the participants sample, your ability to understand the “why’s” of user feedback, and in a recognition that it is not always more efficient than traditional methods.