Back to Basics 11: Understanding User Behavior (part 2)

Welcome back for another installment of our “Back to Basics” blog series, focused on the ins and outs of conducting impactful user research. In our most recent post, we focused on how researchers can better understand user behaviors and attitudes through moderated in-person usability sessions with users.

In this article, we will discuss how researchers can better understand user behaviors and attitudes through over time and in more naturalistic context through diary studies.

Diary Studies

Research methods that observe users’ interactions with a system and gather their subjective feedback, such as diary studies or moderated in-person usability testing, can be implemented at various stages throughout the design life cycle, and for the purpose of answering different questions. Similar to usability studies, diary studies can be used in the discovery phase of product development, to evaluate the current state of a system in the middle of the development cycle, or to evaluate the final version of a system at the end of the development cycle. Here we will discuss what a diary study is, how to conduct one, and what the deliverables for such a study might look like.

What is it?

A diary study is conducted when users keep a daily log of their activities or interactions with a product over a variable period of time (a few days to several months). This type of methodology is particularly well-suited for understanding user thoughts, habits, and behaviors over time and from within a more naturalistic context than what a test laboratory can provide. The biggest advantage of a diary study is that it occurs in the real world, which may be crucial for uncovering usage issues in certain types of products or services (e.g., motivations and habits related to the use of a fitness tracker).

This method is also able to provide a richer and much more complete snapshot of user behaviors compared to shorter usability sessions. Not all task scenarios are simple enough to be condensed into an hour-long usability session. The longitudinal nature of a diary study allows the frustrations and pain points related to longer processes to be uncovered (e.g., shopping for a car, or the experience of using a music streaming app), or to evaluate the learning curve of a new product. Diary studies differ from general usability testing in terms of the scope of the problems that are uncovered. While usability testing may hone in on specific design features that cause usage problems, diary studies look at problems related to the comprehensive process of using a certain product or service. In short, usability studies are able to answer the “why” (e.g., why do users have difficulty with a certain navigation?) and “what” questions (e.g., what features do the users like the most/least?), whereas diary studies answer the “how’s” (e.g., how do people use this product?).

The length of a diary study also contributes to its more complex and time-consuming nature. The study’s length will be entirely dependent upon the purpose of the investigation and the product or system being evaluated. Due to the longitudinal nature of a diary study, there is a higher risk of user drop-out. It is often necessary to over-recruit the number of users needed, and to ensure that the research team maintains regular contact with the users throughout the entire study. Users typically also require higher incentives to keep their involvement, given the amount of work required of them. Incentives are often distributed throughout the study over time in segments, in order to maintain user commitment.

 Strengths

• Allows for the analysis of user behaviors and experiences over time, rather than just a snapshot
• Helps to contextualize the information gathered from users within social and environmental conditions
• Requires fewer participants (you get more data points per participant, and this “within-subjects” design yields more statistical power)
• Can yield more user data than other methods
• Can obtain data related to sensitive and personal experiences that may not be as easy to obtain by using other methods, such as interviews

 Weaknesses

• Recorded information is often retrospective, which is subject to degraded memory and recall bias
• Participants may become fatigued by the number of entries required of them, and may become less thorough or consistent in their reporting over time (which may decrease along with the length of the study)
• Compared to other research methods, it can be a time-consuming and cost-intensive approach
• Due to the large amount of data produced, the process of making sense of the data and analyzing it can be long and arduous
• Vetting participants can be a lengthy process, as users may belong to a limited group and must possess sufficient motivation to fully engage in the study

How to do it?

Users are given recording materials and/or devices that they will use to periodically enter feedback, along with explicit instructions on the expected length and frequency of the diary entries. The users will self-report their thoughts and activities at pre-determined intervals of time. The activity logging may also include having users take photos to help illustrate things that stood out to them, screenshots, or survey responses.

The format of the diary entry is also predetermined by the researcher, and can range from low-fidelity paper diaries to various digital entry forms (e.g., email submissions, or web/app-based tools). The choice of the entry format should match the capabilities of the user group – you don’t want to detract from the use of the product under evaluation by creating a frustrating learning experience for the data entry device! Therefore, err on the side of simplicity. However, one advantage of using digital applications is that they send notifications to users to prompt them to complete their entry so that they are less-likely to miss an entry, although some can be rather costly.

The researcher’s role is also somewhat different when conducting diary studies compared to other methods of user testing. The quality of the instructions given to participants prior to beginning the study and the contact maintained throughout the study is crucial to overall success. For this reason, the researcher should pilot diary entry materials before the study to make sure that they are understandable and easy to use. The entries also need to be monitored on a daily basis, and the researchers need to make themselves available to respond to questions from the participants. Some participants may even show the need for some mental or practical support throughout the process of completing the study, as certain reflections and experiences that emerge may cause unexpected psychological reactions.

 The main steps involved when conducting a diary study include:

1.  Clearly define your test objectives (e.g., Are you looking for opinions to stabilize after the “newness” wares off?)
2. Determine the type of data to be collected, and create diary materials
3. Qualify and recruit test participants from the target populations of interest (you will need fewer participants for longitudinal studies)
4. Ensure that users understand the expectations of the study and how to make entries
5. Check in with users periodically throughout the study
6. Users return the diaries to the research team, and the data is examined for patterns and detailed findings
7. Conduct wrap-up interviews to catch anything that was missed in the self-reported diary entries

 What is the output?

The data analysis process need not wait until all of the data has been collected. Given the large amount of data that diary studies produce, it is often a good idea to begin the analysis process as soon as entries start to arrive. All of the raw data, which may include the written diary entries, ratings, photos, and interview transcripts, are gathered and analyzed in order to form insights. This process may involve transcribing the diary entries depending on the entry format (e.g., voice-recorded) and synthesizing all of the data into a format that lends itself to various methods of data analysis. The patterns and insights derived from the data can then be used to construct a report or presentation of the findings.

Final Thoughts

Testing with real end users in a natural setting yields rich and influential insights and also reveals the ways in which people interact with a system in the real world. Diary studies are conducted longitudinally (a participant provides feedback periodically over a long period of time) to identify problems that are exposed during prolonged interaction with a product. Diary studies do an excellent job at exposing the “hows” related to the contextual use of a product.