Learning From Users in their Natural Habitat

If you like this presentation – show it...

Slide 0

Learning from Users in their Natural Habitat Sara Cambridge sara@gotoresearch.com
 2015 UXPA

Slide 1

What is the best way to learn about the experience that people are having with your product?

Slide 2

Interviews don’t capture the range of ways people do things or the rich, in-the-moment details that people quickly forget. 3

Slide 3

Digital diary studies allow you to: • Get instant feedback as people naturally use a product or service • Ask them follow-up questions throughout the study • Access everything through a digital platform

Slide 4

Are You Wearing One? 5

Slide 5

Fitness Tracker Case Study We recruited 40 people who use fitness trackers to participate in several digital diary studies over 8 months. The first diary study had them report on 5 things they do with their tracker. With every entry, they submitted a short video and answered a few questions. Over 170 entries were collected. We did 4 other diary studies with them over the course of 8 months to better understand how their fitness tracker usage evolved and the role it plays in their overall health goals. 6

Slide 6

Erica, 33 “It’s 10:53pm, and I’m only at 7388 steps….we’re about to go take the dog for a walk. Got to make sure we get to our 10,000 steps before the end of the night.” 7

Slide 7

What This Presentation Will Cover What a Diary Study is The Benefits of Using Diary Studies How Diary Studies Compare to Other Research Methods Traditional Diary Studies vs. Digital Diary Studies What the Participant Experience Is Like How to Create a Digital Diary Study Findings From Our Fitness Tracker Diary Studies 8

Slide 8

What is a Diary Study?

Slide 9

A Diary Study is a Cultural Probe Cultural probes use a collection of tools, artifacts and tasks intended to provoke the user to think about their environment in new ways….With minimal intrusion, researchers can glean insights into participants’ environments that can help to identify problem statements and uncover new opportunities. Adam Starkman & Jennifer Chow http://designresearchtechniques.com/casestudies/cultural-probes/ 10

Slide 10

Diary Studies Have 4 Key Elements Theme-based: the subject and questions are suggested by the researcher. Self-reported: the actual reporting is done by the participant, using whatever platform/prompts were provided. In-the-moment: the participant is asked to do the reporting immediately after engaging in the activity, in order to capture as much detail as possible. Longitudinal: the goal is to capture a range of uses, rather than a singular experience. This can include how the user’s experience evolves over time. 11

Slide 11

Traditional Diary Study Mozilla used paper diary studies in 2012 to learn how participants were using bookmarks in Firefox. They asked 10 people to record their experiences with bookmarks over 3 days. Each day had a different set of questions. Brian Groudan, Mozilla research, 2012 / https://blog.mozilla.org/ ux/page/24/ 12

Slide 12

Traditional Diary Study People came in for an in-person interview afterwards, where they discussed their entries and sorted them by theme. Guiding design principles were culled from the analysis that were the basis for design better bookmarking tools. 13

Slide 13

Why Use 
 Diary Studies?

Slide 14

Looking for the Details The goal of research is to understand not just what people do (and want and believe), but why. This is complex and requires a deep understanding of people’s behavior. Knowing details about their behavior is necessary to get to that understanding. 15

Slide 15

Barriers to Collecting Details However people often have a hard time remembering those details when not engaged in the activity. This is especially true for habitual behaviors. Habits are formed when our brains: Recognize that we repeatedly do the same sequence of actions (like checking email) after being triggered by a specific cue (such as picking up our phones). The brain “chunks” that sequence into one automatic routine. This allows our brain to save mental energy for more complex tasks. The Power of Habit, Duhigg 16

Slide 16

Barriers to Collecting Details A consequence of having a sequence of behaviors “chunked” together is that details of the habit become accessible only after the cue has been activated. So outside the actual ‘doing’ of the task we have difficulty remembering the details. This is why gathering information in context collects such rich details. The Power of Habit, Duhigg 17

Slide 17

The best way to collect these rich details are either observation (contextual interviews) or in-themoment reporting (diary studies). 18

Slide 18

How Diary Studies Compare to 
 Other Methods

Slide 19

why people do it Qualitative
 (subconscious & emotional) Contextual Interviews Diary Studies Usability Testing Focus Groups Quantitative 
 (conscious & logical) Surveys Big Data what people do Quantitative methods only explains what people do while qualitative methods offer insight into why people do them. 20

Slide 20

Stories Contextual Interviews Diary Studies Usability Testing Focus Groups Surveys Data Big Data evaluative generative Once you understand why, your insights can move from evaluative (improvement) to generative (new opportunities). 21

Slide 21

Yesterday’s session about Storytelling by Mary Wharmby discussed this. 22

Slide 22

Karen, 26,
 grad student “I got drunk with my friend and convinced her to download Snapchat on the contingency that I send her a [Snapchat] everyday. I use my [Jawbone Up] to remind me at 7pm to send her a picture….it vibrates on my wrist.” 23

Slide 23

How to Use Diary Studies with Other Research Methods

Slide 24

Insight Study Process An Insight Study is a series of 2-3 focused research cycles built around a specific question. Multiple research methods are used in order to capture different types of insights. Each cycle concludes with an Insight Review that shares the findings with the client team. The benefit for the client is that they: • Deliver real-time findings that can be acted upon right away • Offer an opportunity to refine the next phase of research based on the learnings. 25

Slide 25

Insight Study Methods: Contextual Interviews allow us the deepest insights by observing people’s usage firsthand. Diary Studies collect in-themoment behaviors around a variety of issues throughout the project. Remote Interviews fill in the gaps by probing deeper into the diary study entries and related areas. 26

Slide 26

The Benefits of Using Mixed Methods Using mixed methods allow us to validate our findings and take advantage of the unique strengths that the various research methods offer. • Diary studies let us collect actual behaviors around a range of issues or track ongoing usage patterns. • Contextual interviews give us the deepest insights by letting us observe their natural environment directly. • Remote interviews give us access to people who are geographically distributed and allow more frequent opportunities to engage with them. 27

Slide 27

What is a Digital Diary Study?

Slide 28

http://kotaku.com/hong-kong-residents-supposedly-cant-live-without-their-476758731 Getting people to actually do the reporting is a major obstacle for traditional diary studies. 29

Slide 29

http://kotaku.com/hong-kong-residents-supposedly-cant-live-without-their-476758731 Smartphones are ideal for digital diary studies since people use them in short spurts throughout the day. 30

Slide 30

Digital diary studies can be as simple as text or email prompts that people reply to. 31

Slide 31

Digital diary study platforms Name Platform Cost 24Tru Web/app diary study high CrowdTap Web/app diary study high UX 360 Web diary study high Dscout App diary study medium Dedoose Web diary study low Tumblr private blog low Typeform online survey low 32

Slide 32

$595 for recruiting (or you can bring in your own participants) $295 for each study 33

Slide 33

Dscout has great guides and online support to help get you started. 34

Slide 34

The Participant Experience

Slide 35

Participant reviews opportunities on app Apply for a study Fill out screener This view is only if recruiting through dscout. 36

Slide 36

Email invite to mission Inside the app Q1: Begin with media prompt (photo, video or text) Once invited to a study, this is what all participants see. 37

Slide 37

Studies can have up to 10 questions each, and can be any mix of multiple choice, rating, or openended questions. 38

Slide 38

How to Conduct a Digital Diary Study

Slide 39

The 3 Phases of a Diary Studies The goal: 
 a concise, focused study with the right participants. The goal: 
 high participation and high quality data. The goal: 
 to turn data into relevant insights. 40

Slide 40

• Clarify your research question • Refine and pretest your study • Minimize the work for your participants • Recruit the right participants

Slide 41

Slide 42

Clarify Your Research Question Our Initial Questions: How are activity trackers used by people attempting to make concrete improvements to their fitness habits? Can we discern patterns around when activity trackers are more or less effective at helping people make those changes? 43

Slide 43

Make a list of possible questions. You will have too many. Edit and sort them by theme. 44

Slide 44

Define the goals for each study. This will help maintain focus as you refine the questions. 45

Slide 45

Diary Studies are basically surveys with a twist. 46

Slide 46

That means answer options, which require a lot of refining. The more time spent building the study, the better the data will be. 47

Slide 47

Make it Easy for your Participant to Succeed A guiding principle as you build your study should be to minimize the work your participant has to do. 48

Slide 48

Consider how often you ask them to report. It should be enough to collect useful data, but not enough to cause fatigue. 49

Slide 49

Pairing themed multiple-choice questions with open-ended ones creates a logical flow for people and answers both “what” and “why”. 50

Slide 50

Another benefit of multiple-choice is you can quickly see trends as the data comes in. 51

Slide 51

I think it should send reminders on when to be more active when I'm lagging in my goal such as vibrating. Maybe if I could get some sort of notification on my phone if a specific friend or group of friends had passed me. I do wish that other activities could be tracked with the Fitbit without having to input it myself. Open-ended questions provide the richest data but are more work for both for you and the participant. Use them sparingly. 52

Slide 52

Pretest by sitting down with someone as they go through every step. Look for awkward or unclear phrasing. 53

Slide 53

Take a picture that symbolizes the role your tracker plays in your health/fitness goals Tell us what ideal health would look like for you. Media options: video adds a lot, but also adds more work for both you and the participants. 54

Slide 54

My tracker is a set of tools I'm using to build a healthier life for myself. A long road ahead. Before last year I never thought I could be THAT mom who was fit & healthy. Some questions are better suited for a photo, which makes it easier to compare (and share) responses. 55

Slide 55

Karen, 26,
 grad student “Being able to walk upstairs without being out of breath. Also, be at the pick of my thinking and not feeling tired and drained because of ill health. And I think that my tracker helps me with that especially with sleep.” When capturing real experiences or complex ideas, video is better. 56

Slide 56

Slide 57

10 x 10 x 10 = questions participants entries 1000 data points Diary studies generate a lot of data. It is good to start small. 58

Slide 58

Recruiting options include customers, friends, Craigslist, social media and dscout. 59

Slide 59

If recruiting through dscout, you’ll only pay incentives for participants that finish. dscout recommends paying $5-10 per entry. 60

Slide 60

We looked for participants who have a fitness tracker and were regularly using it as part of a fitness or weight loss goal. 61

Slide 61

40 34 Typical dropout rate is 15-25%, so over-recruit. 62

Slide 62

You can view each applicant’s responses inside the screener survey and rate them based on how good they are. 63

Slide 63

It’s easy to add participants if you do your own recruiting. They get an email link for downloading the app. 64

Slide 64

• Review & make notes from your data as it’s coming in • Be persistent with participants • Ask them questions for clarity • Do post-study interviews

Slide 65

Slide 66

What is your favorite thing about your fitness tracker? Kayla, 27,
 teacher “My favorite thing is that by a single click of a button, the display shows me the time, date, exact steps I've taken, my heart rate, steps climbed and calories burned as opposed to having to find my phone, unlock, open the app...it's all right there, easy to read. I like having it all right there on the display. . Watch the data as it comes in. This is the fun part. 67

Slide 67

Entries are reviewed individually. 68

Slide 68

Take notes of interesting patterns and questions to ask participants as you go through it. 69

Slide 69

Moment of truth: are your questions holding up? If not, can you fix it? Either way, learn for next time. 70

Slide 70

Slide 71

Stay in touch with your participants. Send them reminders if they haven’t submitted an entry. Be friendly and persistant. 72

Slide 72

Ask them questions if you want more clarity about their answers. Be appreciative and interested in what they have to say. 73

Slide 73

Do follow-up interviews with the most engaged participants. You already have rapport and can dig deeper into their actual behaviors. 74

Slide 74

• Start by answering your initial questions • Use the built-in analysis tools • Identify patterns and/or exceptions • Share your findings with real examples

Slide 75

The good news is that by simply reviewing and making notes on the data, you have already started on the analysis. 76

Slide 76

What Further Analysis Should be Done? That depends on many variables, such as: • What your initial question/s were • How surprised you are by the data that has come in • How it fits into other research methods • Your final deliverables 77

Slide 77

The internal tools dscout offers are helpful. Tagging can quickly reveal patterns. 78

Slide 78

Multiple choice questions can be filtered. Here we wanted to review the entries in which people were not as satisfied with their tracker. 79

Slide 79

For Those Who Rated an Experience with Their Tracker as “OK” or Lower: • 30% were unhappy with some aspect of the band 
 (too bulky, not water resistant or something had broken) • 25% were complaining about the charger or battery • 15% had issues logging their food 
 (either through a secondary app or via their main device) • The remaining 30% were a random mix of issues Once the responses were filtered, it was quick to manually categorize what remained. 80

Slide 80

Transcribing video is time-consuming, but makes the data much more accessible. Tagging is an ideal way to remember interesting video. 81

Slide 81

Downloading the data into a spreadsheet that can be coded, refined, sorted and easily reviewed is our preferred method if we intend to dive deep. 82

Slide 82

If the original answer options were not accurate enough, you’ll have to throw data away or recode it yourself; here we recoded it. 83

Slide 83

Fitness Tracker High-Level Findings

Slide 84

Why did you get your first fitness tracker? Larry, 31 “I got a fitness tracker because was about 300 lbs, on a diet, exercising, wasn't losing weight. I knew there was something missing. I wanted to learn how many calories I was burning so could know how little or much I should be eating. That was the missing link. I have lost 150 lbs, and the only way did it was because I started using Jawbone Up.” 85

Slide 85

People Love Trackers Because of What They Learn About Themselves Fitness trackers teach people things about their eating and exercise patterns they’ve never known before. They had made guesses, but they had guessed wrong. People usually overestimated the exercise they were getting and underestimated how much they were eating. Knowing the actual numbers gives people control so they can get to the outcome they want. 86

Slide 86

What do you like least about your tracker? Amy, 38 “Now for least favorite. Even though I loved a lot about all the trackers: Omron and Fitbit, the one thing I didn't like was that after a while I got bored. There's only so much competing you can do, so many steps you can take, but they didn't do anything with the data. It was great when I reached my 10,000 steps for the day, but then what? It doesn't help me do anything more with it.” 87

Slide 87

Once People Learn their Patterns, they get Bored (and/or Frustrated) Depending on how motivated people are to make changes to their health behavior, they usually go through a slump after they learn their patterns. Changing exercise and eating pattens require significant changes to one’s lifestyle, and knowing one’s patterns is only the first step. People want more help getting to their goals. 88

Slide 88

What does ideal health look like to you? Mary, 42 “My ideal health would be to feel better, not drag, and eating better foods. It would help me feel motivated to follow my exercises I need a tracker that would also [suggest what] types of foods to eat. It knows that I am running and this is the food you should eat. That would be so wonderful! That would help me a lot even more with my runs, my exercise, with getting in shape. 89

Slide 89

Trackers Only Address a Fraction of People’s Health Goals When asked what their ideal health looks like, people talked about both complex ideas (such as mood, energy level, endurance and eating habits), and hard numbers (like weight and body fat). Trackers only help with some of the aspects of ideal health. People want their trackers to know their goals, learn their patterns and offer suggestions to help them achieve their goals. 90

Slide 90

Digital Diary Study 
 Key Takeaways

Slide 91

Main Benefits of Using Diary Studies • To collect rich stories about how and why people do things • Get intimate glimpses of how a product fits into people’s lives • Allows for tracking ongoing use of a product • Cost-effective way to collect contextual insights 92

Slide 92

Tips for Building Your Own Diary Study Before you can use it, you may have to sell it. Here are a few things to keep in mind: • It’s an extremely flexible tool; make it work for your needs • Start small, let it prove it’s value to your organization and yourself • Evangelize the insights and stories you collect to build more support. • You become an expert by doing. So dive in! 93

Slide 93

What Will You Do with Diary Studies?

Slide 94

For a 4-min video about diary studies: 
 www.gotoresearch.com/speaking/sara/ Questions about diary studies? Ask! sara@gotoresearch.com @ maybanks

Slide 95

Thank you! sara@gotoresearch.com @ maybanks

Slide 96