Back to Basics 4: The Place & the Tools

Welcome back for another installment of our “Back to Basics” blog series, focused on the in-s and out-s of conducting impactful user research. If you’ve been following along, you’ll recall that we have tackled a couple big topics in recent posts, including how to go about selecting the right methods to address your research questions, and how to find and recruit the right usersto participate in the research.

In this post, we are going to explore two critical things to consider before user testing begins, including (1) where to hold the study sessions and (2) which tools or services will be needed at those places. Determining a suitable venue depends on several factors including study type, cost, accessibility, and facility features.

Naturally, if we are conducting a remote study the choice of venue becomes much less important, because we can use our own offices or homes. Similarly, field studies and site visits typically mean the “venue” is already predetermined. However, in other cases, when participants are invited to show up at a set location, we get a choice of using ad-hoc facilities (e.g. conference rooms at the research sponsor’s offices temporarily repurposed for the study) or dedicated ones (e.g. purpose-built user testing labs).

Venue Pros & Cons

The most obvious benefit of using ad-hoc facilities, especially at the research sponsor’s own premises, is cost. Many studies can be carried out in any available room, and the sponsor typically has the required equipment (i.e., product, GUI, etc.) to be tested. In this way, researchers do not need to occupy dedicated research facilities that may be used for more complex testing (e.g., video recording of tasks being carried out on a mobile device). Sponsors may also find this option convenient, because it makes it easy for them to participate in the research. All they have to do is walk down the hall and chat with the researcher.

However, there are several significant advantages to using dedicated lab facilities over ad-hoc rooms. First, purpose-built labs at testing sites are typically equipped with specialty technical hardware for research purposes. These include split rooms with the lab space and a one-way mirror on one side and an observation room behind the mirror on the other side; high-definition video cameras, room microphones, and recording equipment for test sessions; and custom networking setups to enable screen-sharing for computers and mobile devices.

All of these systems are familiar to the researcher because they use them regularly, which cuts down on setup time and reduces the likelihood of an error occurring during testing, all of which translate to additional costs incurred by the sponsor.

Dedicated facilities also typically offer a variety of value-added services and conveniences that take away much of the stress and logistics of hosting research sessions. These depend on the facility, but some common services include live-streaming audio-video feeds, which are useful for stakeholders who are too busy to attend sessions in person or are based elsewhere; online archiving of recorded sessions for easy sharing; and refreshments and snacks for participants as well as catered meal options for observers. Additionally, this setup allows any of the stakeholders to explore topics that are of personal interest by asking the researcher to ask a follow-up question at the end of a session. Some facilities may also offer participant recruiting or even research services as part of a one-stop-shop setup.

Sometimes, especially when there is a need to present a more professional atmosphere than ad-hoc facilities can provide (e.g. when 3rd-party clients, executive stakeholders, or high-profile participants are involved); dedicated facilities are a great way to leave a positive impression. Although using dedicated facilities is naturally going to cost more up-front than trying to conduct research from a makeshift meeting room, for many people the long-term benefits far outweigh the costs. Often, relieving the stress of spending valuable time gathering all of the required resources to conduct a study is not worth it when professional solutions are already available. Furthermore, the added benefits of using reliable equipment, watching live sessions from behind the glass, and creating an experience that your stakeholders will enjoy is seen as money well spent.

Online vs. In-Person Testing

In recent years, online testing has been becoming increasingly popular. Specialty research-oriented websites and software that allow both unmoderated and moderated remote testing online have provided researchers greater flexibility in conducting UX research.  In the last few months, in particular, online or remote testing has been the only option for many as we’ve dealt with restrictions due to COVID-19.  Like most tools, online testing has its place and purpose, and we’ll compare it with traditional in-person testing to provide a better idea of the strengths and weaknesses of each method.

Certainly, one of online testing’s key advantages is allowing participants to join a session from potentially anywhere in the world. This may be useful in studies where the test material is software or web-based or if the test can be presented through the computer screen. However, when hardware is being tested, or if user interaction with physical components are important (e.g. testing a fingerprint reader hardware and software components on a smartphone), it becomes quickly apparent that online remote testing is less suitable than in-person sessions.

Another related point to consider is the ease of observing participant reactions and responses beyond the responses they provide directly through questionnaires or verbal feedback. With remote online testing, researchers have limited ability to observe how users “really feel” (e.g., non-verbal cues) versus what they self-report. In other words, they could rate the product highly because they truly like it, or because they are simply trying to be agreeable (i.e., demand characteristics), the latter of which is difficult to detect if you are not physically with the participant. Although online testing attempts to mitigate this shortcoming by incorporating webcam feeds to capture facial expressions, in-person sessions simply provide a richer source of non-verbal observational data. That said, sometimes remote studies are a necessary compromise when weighing the cost of easy data collection versus data quality.

Online testing also arguably lacks the stricter rigor and safeguards inherent with in-person testing by identifying unsuitable participants who “game the system” to get through screeners. Clearly, it is easier to detect unqualified participants when sessions are in-person than online, which provides some level of anonymity. This makes in-person studies a better bet when there is a stronger need for data validity (which is always a major concern).

Another commonly-cited strength of online testing is its ability to handle unmoderated testing, which can free up the researcher from having to run time-consuming sessions. Technically, like online tests, some in-person studies can also be configured to allow participants to complete the study without a moderator. Regardless of whether it is an online or in-person test, unmoderated sessions are often unsuitable for studies that require user interaction beyond the computer screen, or when potential for complications or misinterpretations exist. The catch here is that, although it seems like unmoderated sessions would be a time-saver, this is not always the case in the long-run because reviewing, coding, interpreting, and analyzing video can take just as much (if not more) time than actually running the session.

There are two other related advantages often associated with online testing compared with in-person studies: lower cost and faster results. Although that is true, a more nuanced view will reveal that these benefits are really tradeoff gains over aspects like data richness and confidence in data validity. Often, when one adds in more robust screening criteria, the required webcam use, participant verification, longer individual sessions, and so on, costs increase and data collection slows. Despite all these considerations and comparisons with in-person sessions, online testing can still be a viable option but, as mentioned earlier, it is important for the researcher and stakeholders to be aware and make conscious decisions to choose a method with the right balance of pros and cons to match study needs and resource constraints.

Key Features of a UX Research Lab

First and foremost, a traditional usability or user research lab should provide a way to directly observe participants, typically through a one-way mirror in an adjoining room. Often, the reason for allowing direct observation is not for the researcher-moderator or the participant, but for stakeholders. It is often very useful for designers, developers, and other key project personnel to see first-hand how real-world users interact with the test material or product. Not only does this foster a greater sense of involvement and participation in the research process with stakeholders, but also provides them an opportunity to form insights based on unfiltered observations.

Another key component found in modern labs today is audio-video recording equipment. Central to this are specialty room cameras that can allow panning, zooming and viewing from various angles to suit specific room setups. The ability to capture professional-looking videos showing multiple viewpoints within the room (and of the device under test) are often only achievable in dedicated labs.

For example, in a roundtable focus group setup, the camera would ideally show all participants from the front, emulating the moderator’s point-of-view. This is achievable with a ceiling-mounted camera behind the moderator pointing down the conference table.

In another common setup involving product-interaction tests such as mobile app tests where there is interest in capturing both UI interaction and handling of the mobile device, the ceiling-mounted camera is pointed directly downwards and above the test product, allowing it to zoom in and focus on the test product and user’s hands. Combine this with a specialty wall-mounted “bullet-cam” pointed at the participant to capture facial expressions and reactions for a picture-in-picture setup and you have a compact, high-quality recording of a study session for current and future reference.

In both of these examples, these sorts of recordings are difficult to achieve with common standalone cameras or webcams without being noticeably intrusive, awkwardly placed, or requiring a complicated setup.

Related to camera recording capabilities is the ability to capture software and web interactions. Depending on the product being tested and features needed during recording, this may be as simple as repurposing video-conferencing software to share and record screens, or may be more complicated and require specialty tools or “casting” / streaming devices and the knowledge of how to set up these equipment to show interactions, record sessions, and provide simultaneous feeds to stakeholders in the observation room as well as those viewing sessions remotely online.

Clearly there are many options to consider when deciding whether to run an online study or an in-person one, the location to run an in-person research study, and unique needs that may require custom solutions or additional expertise to create a presentable, professional environment. If you invest in experienced researchers, and a research staff operating out of dedicated research facilities, they will be able to provide workable solutions that fit most custom needs.

In our next installment of “Back to Basics”, we will be focusing on the thorny question of how to time and execute research activities so that they both fit within, and are relevant to, the course of development cycles within organizations.  We hope you are enjoying our look at the fundamentals of user research, and we’d love to hear about your experiences in getting the most impact from your research activities. If you’d like to revisit any of the previous topics we’ve explored in this series, pop-over to our blog. See you next time!