#diary study

As a group of 2, we chose a method - Diary study, and focused on the use of this method in longitudinal HCI research. We established three criteria to evaluate the use of diary study: continuity, efficiency and triangulation. Then we dived deep into two typical research papers to apply these criteria for evaluation. 

Diary study in longitudinal study is of special interest to me because it is trying to utilize this time-consuming and labor-intensive method into the agile design world. How to acquire informative insight as much as possible while keep a balance with time is the key to success. With this in mind, researchers should not only keep the field research going (continuity), but also   keep it going well (efficiency & triangulation). Beyond evaluating the method, we were surprised about how likely a method would be introduced to a new field with only surface understanding. The “longitudinal” study in HCI focuses on the average of variables rather than the changes over time, which, to some extent, violates the original definition of longitudinal study.

Aan de hand van de interviews en verder onderzoek heb ik een focus group samengesteld. Deze bestaat uit zes mensen en heb ik voor komende week een diary study uitgezet. Ik heb de focus group gevraagd om vijf dagen lang foto's te maken wanneer ze last hebben van cognitieve dissonantie en deze met een korte uitleg in een appje (blogger) op te slaan.

Deze omschrijving hebben de respondenten ontvangen: 1. Post een foto van een moment waarop er iets gebeurd waar jij je niet prettig bij voelt. Dat kan iets heel kleins zijn (dat de metro bijvoorbeeld vertraging heeft) of iets groots (een functionering gesprek wat niet lekker liep op je werk o.i.d.) 

2. Deze foto mag je vervolgens uploaden in een app (inloggegevens en download instructie hieronder).

A user research method I've used many times, and talked and written about several times too, is the cultural probe (also known as a 'diary study' or simply 'user research diary').

Briefly, the purpose of a cultural probe is to conduct user research from a distance. So rather than having to literally follow the user around for two weeks, they contribute to the probe, either explicitly by writing 'diary' entries, or implicitly by leaving 'digital footprints' of their online activity. One might label the former as a reflective diary probe and the latter as a 'lifestream' log probe. Both types are useful, the lifestream log as evidence akin to analytics of what they actually do, and the reflective diaries in terms of exploring the motivations behind what they think and do.

In years gone by, diary studies have ridden both peaks and troughs in popularity with social and market research practitioners, but these days there are now many ways to conduct a probe online, for little or no cost (aside from recruiting users and compensating them for their time). For example, you can easily setup a blog to act as an online diary—I've done this using Wordpress on a number of occasions—but if you're less technically inclined or want the convenience of using an "off the shelf" tool, then there are things like Tumblr and Posterous.

My colleague, Chris Khalil, touched on using a Tumblr blog in his UX Australia presentation last year. More recently I've been using Posterous for a similar purpose and thought I'd share some hints and tips.

Think about whether a probe is suitable

Are cultural probes for everyone? No, not all research objectives are suitable for probes, nor are they necessary for all research projects.

Similarly, if you're investigating an audience that isn't big on 'public' discussion or extroverted self-promotion as some call it, or if the subject matter is not something people want to discuss openly, then online probes are probably not suitable.

Having said that, I've used probes and diaries on several different research projects, including those where it didn't work that well. There's no better way to learn.

Are online probes for everyone? Likewise, it's not always suitable or effective to use an online probe. If you're studying offline behaviour or if your audience is not big on online, then an online probe is probably going to be counter productive. Using the method might actually skew your results because to find people willing to participate in the probe, you're automatically selecting people comfortable with digital media (either that or they just won't do it).

Offline probes can work very well (for example, an A5 booklet that participants fill in over the course of the study). In fact I believe they work better than online diaries because they:

cause a moment of pause, you have to stop, think then write (separating the medium from the message)

can easily be 'multimedia' with little effort (ie write, draw, cut+paste),

are easily transported anywhere,

exercises and 'puzzles' are easily integrated.

The main argument against these old school probes are that they do not share the medium we are (usually) studying; if someone is using the web to do XYZ, then reporting what they're doing online would be easier. Perhaps for some, but you at least need to have an alternative for those who don't find that to be convenient.

Ultimately, you need to decide if a cultural probe is going to be suitable for your research project, and whether an online or offline probe will work best for your participants.

Is Posterous right for my probe?

The main benefit of Posterous is the fact that you interface with it using email, primarily. So the participant simply sends an email to a unique email address and it creates a blog post for them automatically (in our case it becomes a diary entry). There are other interfaces, which are described below, but the email interface is a boon for those investigating an audience who is adept in the use of email, but possible little else in the web world.

Posterous can be useful for both the shorter, 'lifestream log' style probes and the longer, 'reflective diary' style probes. But that's not to say it has to be the only tool you use, your probe could actually be made up of several elements: Posterous (via email), iPhone app, Twitter updates and hand-drawn 'mud maps'.

Timing is crucial

The timing of a probe is possibly the biggest factor that determines its success as a research method. You need to ensure your probe takes place during the activity that you wish to investigate is occurring.

Recently we were researching people undertaking renovations to their house and how they interact with tradespeople, a diary that spanned the period of their renovation would be extremely useful. Again, the recruitment is such a crucial part of this method, you would need to find participants that would be willing to keep a diary over this time. One way of selling or "incentivising" the idea would be to talk about the fact that keeping a diary of such a big thing in their lives can be very rewarding to look back on later. Mention documentaries and TV shows like Grand Designs.

(Talking about benefits, I've found that participants usually really enjoy the diary experience. They tell me it's fascinating to actually see what and when they do certain things. This is particularly the case with the reflective diaries; they find the reflection quite enlightening.

So the timing (or more accurately the duration) of a probe can be a big deal in terms of both recruiting appropriate participants and encouraging their participation.

Carefully recruit participants

One thing that academic researchers (design PhD students etc) who use probes, stress is the importance of, is recruitment. They even go to the extent of interviewing candidates beforehand to select the best people to participate in a probe. This is obviously time consuming and costly, but it goes to show how important it is to get the right people. It's a case of "garbage in, garbage out", I suppose. This can start to lead or influence the process somewhat, rather than it being completely open and exploratory, but the amount of time and investment in a probe justifies the risk. I've learned this the hard way!

I usually make do with a phone call, as described below, but it's also a good idea to build as many fail-safes into your recruitment processes as possible. My main point of concern in this regard is what exactly the participant has been asked and told; using a third party recruiter often means you relinquish control of this information flow to a certain degree.

Tell your recruitment firm to recruit suitable candidates; those that meet your specs and are willing to spend two weeks keeping an online diary. I usually suggest telling potential participants that it's like blogging but private (only they can see it), and they will be required to write a few entries a day. Usually recruiters want you to quantify the effort required, in terms of minutes per day etc. This can be difficult but you need to make sure you're going to get people who are going to use the diary.

Get the recruiter to send you the participants' phone number and email address, if nothing else. That way you can call them and send them important details in writing, both of which I recommend below.

By the way, if you're not using a professional recruiter, the above still applies, just perform the actions yourself. Then ask yourself why you're not using a professional recruiter :)

Send each participant a "welcome pack"

Via the recruiter, I almost always send each participant a document (a "primer", if you will) with an overview of the research and general instructions on what they have to do. This allows me to ensure that each participant gets the information I want them to have before they start the probe, as well as setting focus questions or pre-tasks as per a normal primer.

In terms of guiding the participant in the use of their diary, I usually include the following list of tips in the welcome pack:

Write as much as you can about your experiences, both positive and negative, big and small.

We are interested in whatever you have to say, no matter how minor it might seem to you.

We are especially interested in things that are surprising or unexpected.

If you’re not sure whether to include something or not, please put it in.

If you make a mistake, you can correct it, or you can just keep on going.

Setup a Posterous account for each diary

You can create an account with Posterous for free, and it's possible to create up to three blogs per account. You can "add a contributor" to each blog such that you have one blog per participant, creating more accounts if you need more than three blogs. Making each of the blogs private means each participant can only see their own blog, but you as the researcher, can see them all. This works fine, but since each participant is only a contributor, they don't have a Posterous login and thus cannot use things like the "bookmarklet" (described below).

A better solution is to create a Posterous account for each research participant, with one private blog for each account. I create these accounts as generic research accounts (eg myresearchdiary1, myresearchdiary2, myresearchdiary3) that are re-used for each project, I simply clear out each blog and reset its security when a project is finished.

Add one participant to each blog using the email address they provided during recruitment, if they have multiple email addresses they might want to use (eg work and home) then them all. When you add a contributor you have the opportunity to send a welcome email to that contributor. This is when you send them the details of how the diary works.

Make sure you include the unique details for each blog in the welcome email:

The "post" email address (eg [email protected])

Blog URL (eg www.myresearchdiary1.posterous.com )

Password (one idea is to set the password to the participant's surname)

Make yourself a contributor for each blog, so you can monitor what's happening and also ask questions or make comments if need be.

Call participants on the phone

Before the probe begins, call each participant and introduce yourself. It goes a long way to making the whole process much smoother if there is a real person they're talking to via the diary. Of course you could meet them in person too, but there are many reasons why this isn't practical (eg travel time and cost).

Other than building rapport, you want to use the phone call to check that they…

Have answered the recruiter's screener questions correctly

Have received the research "welcome pack" including the consent form

Have received the Posterous welcome email (see above)

Are able to send a test email to Posterous

Are willing and able to participate in the diary for two weeks

(Get the recruiter to arrange the phone call for a specific time when they will be in front of their computer and able to access their email.)

You should also reiterate the purpose of the research and what you'd like them to put in the diary. They should have already received the welcome pack and the welcome email—if you use them—but it doesn't hurt to run through it all again and answer any questions they may have.

Remind and encourage your participants

Inevitably you will have some (possibly all) participants who are really not sure what they are supposed to be doing. Even if you use a welcome pack, you may still need to prompt them into action.

The beauty of an online diary is that you, as the researcher, can monitor the submissions as time progresses. If necessary you can prompt participants to write diary entries by

sending them a reminder email

posing a question (or task/exercise) in the blog itself

commenting on one of their posts (asking for clarification etc)

I find that positive feedback and examples of what you're after, can go a long way. I usually don't like to be too perspective, preferring to "let nature take its course" but sometimes you do need to step in and guide or moderate.

Posterous has a nice feature in terms of email notifications. If I write a post in a diary asking a specific question or setting a task for the participant, the system will send them an email. They can simply hit Reply and starting typing their answer, hit Send and their response will be added to the diary as a comment on my original post. Neat and simple.

But be careful not to spam them.

Introduce the Posterous "bookmarklet"

If the participant is comfortable with Posterous via email, and if you feel they would benefit from a more powerful interface with their diary, you can suggest two additional tools that might be more convenient for them. The first is the Posterous bookmarklet which can be added to their browser toolbar and allows quicker and easier posts to the diary (especially good for video etc). But not everyone will "get it" or be able to "install" it (eg corporate IE6 users).

You can also do the lifestreaming using the Posterous "autopost" feature, allowing the participant to bring in their activity from social networking services like Facebook. This is where Posterous is on par with Tumblr.

Suggest the Posterous iPhone app

Going beyond the bookmarklet, the Posterous iPhone app is much more convenient for posting to the diary when using an iPhone, especially good for photos taken with the device. Again they need to be fairly savvy in order for this to be worth the time it might take to explain it to them (and they obviously need an iPhone).

This app is great if you're doing research that expands wider than just the online environment, and into the 'real' world. For example if you're reaching the process of buying a new car, you might have research participants document that process using photos and Twitter-esque snippets of text via their iPhone.

You can tell them about these more advanced features in the "welcome pack" or during the phone call, or you can wait and see how they handle Posterous in the first place and then make the suggestion to those who you think are up to it.

Follow-up with an interview

It's pretty important to conclude the probe with a face-to-face interview (or 'contextual inquiry' if you like). This is where you can talk through the diary and any other artefacts created, and get the all important explanation from the participant: why did they do this? what did they mean by that? who did they talk to then?

It makes sense to have access to the Posterous blog and go through it on-screen, but if need be you can print it out and go through it on paper, adding scribbles and notes as you go.

Collect the right data

As with any research method, you need to use a probe in the right way. Namely, the type or style of diary determines the data you are typically able to get. For example, a reflective diary is self-reported and thus trying to capture an accurate picture of their activity is difficult, but you can legitimately explore their motivations and reasoning. Similarly, a lifestyle log is like a record of their activities, but will probably tell you very little about what they were thinking whilst doing it.

This decision regarding the style of probe also impacts on how you encourage participation. Asking participants to post entries 'as and when' doesn't work so well if you want reflection, and alternatively if you want to view their lifestream then one entry a day isn't going to work.

Be aware of confidentiality risks

There is, of course, the issue of security and confidentiality risk. Any online service like Posterous is not going to be 100% secure, so there's a chance that the contents of the blogs could be leaked. In reality though it's a small chance; who is going to bother trying to break into a research diary?

Besides the system itself, your participants might be the problem. My standard practice is to have all research participants sign a Non Disclosure Agreement (NDA) along with the consent form, but that still won't stop information from getting out if someone wanted it to. Your company's strategy or potential new website features might be leaked to your competition. But that's the case with any user research or usability testing, and the risk is the price of being user-centred!

Typically though, the contents of a diary and the questions asked of participants, are not that commercially confidential. I certainly believe the benefits of this type of research far outweigh the risks.

So there you have it, my top 13 tips for using Posterous for cultural probes. I stand by probes as a useful research method, though some of my peers question my continued fascination with them. What's important is to be aware of the strengths and weaknesses of probes, as with any research method, and ideally use them in combination with several other methods.