Learning From Users in their Natural Habitat

Понравилась презентация – покажи это...

Слайд 0

Learning from Users in their Natural Habitat Sara Cambridge sara@gotoresearch.com
 2015 UXPA

Слайд 1

What is the best way to learn about the experience that people are having with your product?

Слайд 2

Interviews don’t capture the range of ways people do things or the rich, in-the-moment details that people quickly forget. 3

Слайд 3

Digital diary studies allow you to: • Get instant feedback as people naturally use a product or service • Ask them follow-up questions throughout the study • Access everything through a digital platform

Слайд 4

Are You Wearing One? 5

Слайд 5

Fitness Tracker Case Study We recruited 40 people who use fitness trackers to participate in several digital diary studies over 8 months. The first diary study had them report on 5 things they do with their tracker. With every entry, they submitted a short video and answered a few questions. Over 170 entries were collected. We did 4 other diary studies with them over the course of 8 months to better understand how their fitness tracker usage evolved and the role it plays in their overall health goals. 6

Слайд 6

Erica, 33 “It’s 10:53pm, and I’m only at 7388 steps….we’re about to go take the dog for a walk. Got to make sure we get to our 10,000 steps before the end of the night.” 7

Слайд 7

What This Presentation Will Cover What a Diary Study is The Benefits of Using Diary Studies How Diary Studies Compare to Other Research Methods Traditional Diary Studies vs. Digital Diary Studies What the Participant Experience Is Like How to Create a Digital Diary Study Findings From Our Fitness Tracker Diary Studies 8

Слайд 8

What is a Diary Study?

Слайд 9

A Diary Study is a Cultural Probe Cultural probes use a collection of tools, artifacts and tasks intended to provoke the user to think about their environment in new ways….With minimal intrusion, researchers can glean insights into participants’ environments that can help to identify problem statements and uncover new opportunities. Adam Starkman & Jennifer Chow http://designresearchtechniques.com/casestudies/cultural-probes/ 10

Слайд 10

Diary Studies Have 4 Key Elements Theme-based: the subject and questions are suggested by the researcher. Self-reported: the actual reporting is done by the participant, using whatever platform/prompts were provided. In-the-moment: the participant is asked to do the reporting immediately after engaging in the activity, in order to capture as much detail as possible. Longitudinal: the goal is to capture a range of uses, rather than a singular experience. This can include how the user’s experience evolves over time. 11

Слайд 11

Traditional Diary Study Mozilla used paper diary studies in 2012 to learn how participants were using bookmarks in Firefox. They asked 10 people to record their experiences with bookmarks over 3 days. Each day had a different set of questions. Brian Groudan, Mozilla research, 2012 / https://blog.mozilla.org/ ux/page/24/ 12

Слайд 12

Traditional Diary Study People came in for an in-person interview afterwards, where they discussed their entries and sorted them by theme. Guiding design principles were culled from the analysis that were the basis for design better bookmarking tools. 13

Слайд 13

Why Use 
 Diary Studies?

Слайд 14

Looking for the Details The goal of research is to understand not just what people do (and want and believe), but why. This is complex and requires a deep understanding of people’s behavior. Knowing details about their behavior is necessary to get to that understanding. 15

Слайд 15

Barriers to Collecting Details However people often have a hard time remembering those details when not engaged in the activity. This is especially true for habitual behaviors. Habits are formed when our brains: Recognize that we repeatedly do the same sequence of actions (like checking email) after being triggered by a specific cue (such as picking up our phones). The brain “chunks” that sequence into one automatic routine. This allows our brain to save mental energy for more complex tasks. The Power of Habit, Duhigg 16

Слайд 16

Barriers to Collecting Details A consequence of having a sequence of behaviors “chunked” together is that details of the habit become accessible only after the cue has been activated. So outside the actual ‘doing’ of the task we have difficulty remembering the details. This is why gathering information in context collects such rich details. The Power of Habit, Duhigg 17

Слайд 17

The best way to collect these rich details are either observation (contextual interviews) or in-themoment reporting (diary studies). 18

Слайд 18

How Diary Studies Compare to 
 Other Methods

Слайд 19

why people do it Qualitative
 (subconscious & emotional) Contextual Interviews Diary Studies Usability Testing Focus Groups Quantitative 
 (conscious & logical) Surveys Big Data what people do Quantitative methods only explains what people do while qualitative methods offer insight into why people do them. 20

Слайд 20

Stories Contextual Interviews Diary Studies Usability Testing Focus Groups Surveys Data Big Data evaluative generative Once you understand why, your insights can move from evaluative (improvement) to generative (new opportunities). 21

Слайд 21

Yesterday’s session about Storytelling by Mary Wharmby discussed this. 22

Слайд 22

Karen, 26,
 grad student “I got drunk with my friend and convinced her to download Snapchat on the contingency that I send her a [Snapchat] everyday. I use my [Jawbone Up] to remind me at 7pm to send her a picture….it vibrates on my wrist.” 23

Слайд 23

How to Use Diary Studies with Other Research Methods

Слайд 24

Insight Study Process An Insight Study is a series of 2-3 focused research cycles built around a specific question. Multiple research methods are used in order to capture different types of insights. Each cycle concludes with an Insight Review that shares the findings with the client team. The benefit for the client is that they: • Deliver real-time findings that can be acted upon right away • Offer an opportunity to refine the next phase of research based on the learnings. 25

Слайд 25

Insight Study Methods: Contextual Interviews allow us the deepest insights by observing people’s usage firsthand. Diary Studies collect in-themoment behaviors around a variety of issues throughout the project. Remote Interviews fill in the gaps by probing deeper into the diary study entries and related areas. 26

Слайд 26

The Benefits of Using Mixed Methods Using mixed methods allow us to validate our findings and take advantage of the unique strengths that the various research methods offer. • Diary studies let us collect actual behaviors around a range of issues or track ongoing usage patterns. • Contextual interviews give us the deepest insights by letting us observe their natural environment directly. • Remote interviews give us access to people who are geographically distributed and allow more frequent opportunities to engage with them. 27

Слайд 27

What is a Digital Diary Study?

Слайд 28

http://kotaku.com/hong-kong-residents-supposedly-cant-live-without-their-476758731 Getting people to actually do the reporting is a major obstacle for traditional diary studies. 29

Слайд 29

http://kotaku.com/hong-kong-residents-supposedly-cant-live-without-their-476758731 Smartphones are ideal for digital diary studies since people use them in short spurts throughout the day. 30

Слайд 30

Digital diary studies can be as simple as text or email prompts that people reply to. 31

Слайд 31

Digital diary study platforms Name Platform Cost 24Tru Web/app diary study high CrowdTap Web/app diary study high UX 360 Web diary study high Dscout App diary study medium Dedoose Web diary study low Tumblr private blog low Typeform online survey low 32

Слайд 32

$595 for recruiting (or you can bring in your own participants) $295 for each study 33

Слайд 33

Dscout has great guides and online support to help get you started. 34

Слайд 34

The Participant Experience

Слайд 35

Participant reviews opportunities on app Apply for a study Fill out screener This view is only if recruiting through dscout. 36

Слайд 36

Email invite to mission Inside the app Q1: Begin with media prompt (photo, video or text) Once invited to a study, this is what all participants see. 37

Слайд 37

Studies can have up to 10 questions each, and can be any mix of multiple choice, rating, or openended questions. 38

Слайд 38

How to Conduct a Digital Diary Study

Слайд 39

The 3 Phases of a Diary Studies The goal: 
 a concise, focused study with the right participants. The goal: 
 high participation and high quality data. The goal: 
 to turn data into relevant insights. 40

Слайд 40

• Clarify your research question • Refine and pretest your study • Minimize the work for your participants • Recruit the right participants

Слайд 41

Слайд 42

Clarify Your Research Question Our Initial Questions: How are activity trackers used by people attempting to make concrete improvements to their fitness habits? Can we discern patterns around when activity trackers are more or less effective at helping people make those changes? 43

Слайд 43

Make a list of possible questions. You will have too many. Edit and sort them by theme. 44

Слайд 44

Define the goals for each study. This will help maintain focus as you refine the questions. 45

Слайд 45

Diary Studies are basically surveys with a twist. 46

Слайд 46

That means answer options, which require a lot of refining. The more time spent building the study, the better the data will be. 47

Слайд 47

Make it Easy for your Participant to Succeed A guiding principle as you build your study should be to minimize the work your participant has to do. 48

Слайд 48

Consider how often you ask them to report. It should be enough to collect useful data, but not enough to cause fatigue. 49

Слайд 49

Pairing themed multiple-choice questions with open-ended ones creates a logical flow for people and answers both “what” and “why”. 50

Слайд 50

Another benefit of multiple-choice is you can quickly see trends as the data comes in. 51

Слайд 51

I think it should send reminders on when to be more active when I'm lagging in my goal such as vibrating. Maybe if I could get some sort of notification on my phone if a specific friend or group of friends had passed me. I do wish that other activities could be tracked with the Fitbit without having to input it myself. Open-ended questions provide the richest data but are more work for both for you and the participant. Use them sparingly. 52

Слайд 52

Pretest by sitting down with someone as they go through every step. Look for awkward or unclear phrasing. 53

Слайд 53

Take a picture that symbolizes the role your tracker plays in your health/fitness goals Tell us what ideal health would look like for you. Media options: video adds a lot, but also adds more work for both you and the participants. 54

Слайд 54

My tracker is a set of tools I'm using to build a healthier life for myself. A long road ahead. Before last year I never thought I could be THAT mom who was fit & healthy. Some questions are better suited for a photo, which makes it easier to compare (and share) responses. 55

Слайд 55

Karen, 26,
 grad student “Being able to walk upstairs without being out of breath. Also, be at the pick of my thinking and not feeling tired and drained because of ill health. And I think that my tracker helps me with that especially with sleep.” When capturing real experiences or complex ideas, video is better. 56

Слайд 56

Слайд 57

10 x 10 x 10 = questions participants entries 1000 data points Diary studies generate a lot of data. It is good to start small. 58

Слайд 58

Recruiting options include customers, friends, Craigslist, social media and dscout. 59

Слайд 59

If recruiting through dscout, you’ll only pay incentives for participants that finish. dscout recommends paying $5-10 per entry. 60

Слайд 60

We looked for participants who have a fitness tracker and were regularly using it as part of a fitness or weight loss goal. 61

Слайд 61

40 34 Typical dropout rate is 15-25%, so over-recruit. 62

Слайд 62

You can view each applicant’s responses inside the screener survey and rate them based on how good they are. 63

Слайд 63

It’s easy to add participants if you do your own recruiting. They get an email link for downloading the app. 64

Слайд 64

• Review & make notes from your data as it’s coming in • Be persistent with participants • Ask them questions for clarity • Do post-study interviews

Слайд 65

Слайд 66

What is your favorite thing about your fitness tracker? Kayla, 27,
 teacher “My favorite thing is that by a single click of a button, the display shows me the time, date, exact steps I've taken, my heart rate, steps climbed and calories burned as opposed to having to find my phone, unlock, open the app...it's all right there, easy to read. I like having it all right there on the display. . Watch the data as it comes in. This is the fun part. 67

Слайд 67

Entries are reviewed individually. 68

Слайд 68

Take notes of interesting patterns and questions to ask participants as you go through it. 69

Слайд 69

Moment of truth: are your questions holding up? If not, can you fix it? Either way, learn for next time. 70

Слайд 70

Слайд 71

Stay in touch with your participants. Send them reminders if they haven’t submitted an entry. Be friendly and persistant. 72

Слайд 72

Ask them questions if you want more clarity about their answers. Be appreciative and interested in what they have to say. 73

Слайд 73

Do follow-up interviews with the most engaged participants. You already have rapport and can dig deeper into their actual behaviors. 74

Слайд 74

• Start by answering your initial questions • Use the built-in analysis tools • Identify patterns and/or exceptions • Share your findings with real examples

Слайд 75

The good news is that by simply reviewing and making notes on the data, you have already started on the analysis. 76

Слайд 76

What Further Analysis Should be Done? That depends on many variables, such as: • What your initial question/s were • How surprised you are by the data that has come in • How it fits into other research methods • Your final deliverables 77

Слайд 77

The internal tools dscout offers are helpful. Tagging can quickly reveal patterns. 78

Слайд 78

Multiple choice questions can be filtered. Here we wanted to review the entries in which people were not as satisfied with their tracker. 79

Слайд 79

For Those Who Rated an Experience with Their Tracker as “OK” or Lower: • 30% were unhappy with some aspect of the band 
 (too bulky, not water resistant or something had broken) • 25% were complaining about the charger or battery • 15% had issues logging their food 
 (either through a secondary app or via their main device) • The remaining 30% were a random mix of issues Once the responses were filtered, it was quick to manually categorize what remained. 80

Слайд 80

Transcribing video is time-consuming, but makes the data much more accessible. Tagging is an ideal way to remember interesting video. 81

Слайд 81

Downloading the data into a spreadsheet that can be coded, refined, sorted and easily reviewed is our preferred method if we intend to dive deep. 82

Слайд 82

If the original answer options were not accurate enough, you’ll have to throw data away or recode it yourself; here we recoded it. 83

Слайд 83

Fitness Tracker High-Level Findings

Слайд 84

Why did you get your first fitness tracker? Larry, 31 “I got a fitness tracker because was about 300 lbs, on a diet, exercising, wasn't losing weight. I knew there was something missing. I wanted to learn how many calories I was burning so could know how little or much I should be eating. That was the missing link. I have lost 150 lbs, and the only way did it was because I started using Jawbone Up.” 85

Слайд 85

People Love Trackers Because of What They Learn About Themselves Fitness trackers teach people things about their eating and exercise patterns they’ve never known before. They had made guesses, but they had guessed wrong. People usually overestimated the exercise they were getting and underestimated how much they were eating. Knowing the actual numbers gives people control so they can get to the outcome they want. 86

Слайд 86

What do you like least about your tracker? Amy, 38 “Now for least favorite. Even though I loved a lot about all the trackers: Omron and Fitbit, the one thing I didn't like was that after a while I got bored. There's only so much competing you can do, so many steps you can take, but they didn't do anything with the data. It was great when I reached my 10,000 steps for the day, but then what? It doesn't help me do anything more with it.” 87

Слайд 87

Once People Learn their Patterns, they get Bored (and/or Frustrated) Depending on how motivated people are to make changes to their health behavior, they usually go through a slump after they learn their patterns. Changing exercise and eating pattens require significant changes to one’s lifestyle, and knowing one’s patterns is only the first step. People want more help getting to their goals. 88

Слайд 88

What does ideal health look like to you? Mary, 42 “My ideal health would be to feel better, not drag, and eating better foods. It would help me feel motivated to follow my exercises I need a tracker that would also [suggest what] types of foods to eat. It knows that I am running and this is the food you should eat. That would be so wonderful! That would help me a lot even more with my runs, my exercise, with getting in shape. 89

Слайд 89

Trackers Only Address a Fraction of People’s Health Goals When asked what their ideal health looks like, people talked about both complex ideas (such as mood, energy level, endurance and eating habits), and hard numbers (like weight and body fat). Trackers only help with some of the aspects of ideal health. People want their trackers to know their goals, learn their patterns and offer suggestions to help them achieve their goals. 90

Слайд 90

Digital Diary Study 
 Key Takeaways

Слайд 91

Main Benefits of Using Diary Studies • To collect rich stories about how and why people do things • Get intimate glimpses of how a product fits into people’s lives • Allows for tracking ongoing use of a product • Cost-effective way to collect contextual insights 92

Слайд 92

Tips for Building Your Own Diary Study Before you can use it, you may have to sell it. Here are a few things to keep in mind: • It’s an extremely flexible tool; make it work for your needs • Start small, let it prove it’s value to your organization and yourself • Evangelize the insights and stories you collect to build more support. • You become an expert by doing. So dive in! 93

Слайд 93

What Will You Do with Diary Studies?

Слайд 94

For a 4-min video about diary studies: 
 www.gotoresearch.com/speaking/sara/ Questions about diary studies? Ask! sara@gotoresearch.com @ maybanks

Слайд 95

Thank you! sara@gotoresearch.com @ maybanks

Слайд 96