Virtual Floorspace

The discussion guide consists of the interview script and notes to the interviewer. The high level objectives of the interview should be reflected in the questions and exercises that comprise the script. The script is divided into modules, each with a specific research focus, and intended to very directly and purposefully capture data that meets the predefined research objectives. The types of modules that are included in the script, and the way that they are executed in terms of specific questions and exercises, vary for every project according to the subject matter, the project objectives, and the participant sample. There is no one-size fits all for customer interviews. Research objectives and the interview scripts that achieve them should be uniquely tailored to each project.

In the remainder of this chapter, I present example interview modules that I have used for different e-commerce design research projects, which seem to have a generalized application. As I stated above each design research project has its own unique goals and requirements; so these example modules are provided as guidance for creating your own interview modules, not as a template or boilerplate. The interview modules I typically include in customer interviews are:

• Welcome
• Participant characterization
• Context of use
• Motivations
• Formation of the consideration set
• Case history of relevant online experiences
• Detailed task analysis
• Card sorting
• Description of ideal experience
• Participatory design of future system
• Evaluation of existing design work

I’ll describe each of these interview modules in a separate posting. 

The discussion guide consists of the interview script and notes to the interviewer. The high level objectives of the interview should be reflected in the questions and exercises that comprise the script. The script is divided into modules, each with a specific research focus, and intended to very directly and purposefully capture data that meets the predefined research objectives. The types of modules that are included in the script, and the way that they are executed in terms of specific questions and exercises, vary for every project according to the subject matter, the project objectives, and the participant sample. There is no one-size fits all for customer interviews. Research objectives and the interview scripts that achieve them should be uniquely tailored to each project.

I’m going to describe some example interview modules that I have used for different e-commerce design research projects, which seem to have a generalized application. As I stated above each design research project has its own unique goals and requirements; so these example modules are provided as guidance for creating your own interview modules, not as a template or boilerplate. The interview modules I typically include in customer interviews are:

• Welcome
• Participant characterization
• Context of use
• Motivations
• Formation of the consideration set
• Case history of relevant online experiences
• Detailed task analysis
• Card sorting
• Description of ideal experience
• Participatory design of future system
• Evaluation of existing design work

I’ll describe these interview modules in subsequent posts.

 Copyright 2009, Paul Bryan, Usography Corporation (www.usography.com)

Linked In: http://www.linkedin.com/in/uxexperts

Uncategorized customer interviews, in-depth interviews, interview questions, research methodology, research methods, user interviews, user research

I was on a design strategy project recently for a top-tier retailer for which an agency had developed dozens of personas. The personas weren’t differentiated by behavioral attributes or measurable characteristics, only by their ages, personal stories, and features they were (purportedly) likely to use. The persona printouts were plastered all over the walls, but with time the ink had faded. The team told me that these personas had taken a serious amount of time (and budget) to develop, but were never used to guide design. This exercise obviously gave personas a bad name in that company.

Another national retailer I consulted with also had had personas created for them. I was tasked with creating profiles for the most valuable online customers, and create a design strategy with these profiles as a guide. One rule stated plainly at the outset: Don’t mention personas and don’t put that word in the deliverable. The senior management team wouldn’t endure another project with personas.

How did this perception of uselessness, and even outright hostility toward personas develop? Nearly a decade earlier I worked on a personas project for a leading company in the photography products and services industry, the team loved the personas. They brought them up in every major conversation where design issues were debated. They remembered their names for months during a massive new product effort. I still remember their names. What was different?

I think it was a matter of the foundation on which the personas rested. In the first two examples I used, the personas were simply personal, plausible stories that were invented on the basis of what was known about customers from anecdotal evidence. They were constructs that could not be measured or disagreed with. They were very logical and well-organized, and were used by their creators to propose numerous features and design changes. But they were hollow. The photography company, on the other hand, had decades of data that they were able to attribute to the personas. They knew these people from a numbers perspective, how many pictures they took, what they did with the pictures, how likely they were to produce new picture artifacts, etc. The personas were clearly segments of their target audience and they knew how many people those segments represented.

Despite my initial positive experience with personas back in the 1990’s, I promote a different, but related, construct to my clients these days, that of user archetypes. Some agencies use the terms personas and user archetypes interchangeably, but I think of them rather differently. The distinguishing aspect of all personas that I have read about in the popular literature is the individual identity and the matching personal story that involves the interactive domain in question. What distinguishes user archetypes is that they are archetypical, meaning they are the quintessential or most frequently observed example or universally understood type.

I have been in quite a few persona-creation sessions. The methods vary widely, from the most whimsical to others that match specific data points of interest. User archetypes, on the other hand, by the nature of being archetypal must rely on data of some sort to achieve that status. Reading through the popular design resources, I haven’t found a common understanding of how user archetypes should be developed, so I will describe the steps in the Usography process for creating user archetypes. Note that the steps described below are for the largest e-commerce projects with the biggest budgets for research. For smaller projects, we combine many steps and use the best available data. So each of the following steps is contingent upon the scope, goals, and budget of the project.

• Consult existing research
• Conduct ethnographic study or in-context interviews
• Determine needs, motivations, activities, resources, artifacts, etc. that are involved leading up to and following use of the web site in question.
• Determine the variables and higher-order concepts that impact behavior
• Use scales to characterize users more precisely
• Determine the range of values for all key variables in the population of users studied
• Look for qualitative evidence of correlations and dependencies between variables
• Create profiles that summarize values of these variables for actual participants
• Group participants based on shared characteristics and behaviors
• Create a user archetype profile that represents each distinct group.
• Measure the variables that define and differentiate the user archetypes in the population of interest more precisely using quantitative methods
• Finalize and prioritize user archetypes based on anticipated value to the company
• Develop a design strategy to engage and support each of the top priority archetypes.
• Track results, and refine the model

 Copyright 2009, Paul Bryan, Usography Corporation (www.usography.com)

Linked In: http://www.linkedin.com/in/uxexperts

Uncategorized design strategy, e-commerce, personas, user archetypes, user segmentation

The daily schedule should be included in the interview protocol document, and also be posted outside the room where the interviews will take place. Research sessions are typically scheduled at 2 hour intervals. The interviews last between 60 and 90 minutes, followed by a break of 30 to 60 minutes. The team arrives approximately one hour before the start of the first interview to get set up, and stays until the interview room is in order.

A typical Usography daily schedule runs for about 12 to 14 hours, and might include some or all of the following activities:

-       Prepare participant testing station and observation room

-       Make copies of script

-       Replenish materials for exercises as necessary

-       Prepare water and/or other refreshments

-       Conduct Interview 1

-       Debrief

-       Replenish materials and prepare rooms, as needed

-       Conduct Interview 2

-       Debrief

-       Replenish materials and prepare rooms, as needed

-       Conduct Interview 3

-       Debrief

-       Replenish materials and prepare rooms, as needed

-       Conduct Interview 4

-       Debrief

-       Assemble and label materials completed during the day

-       Conduct Interview 5

-       Debrief

-       Assemble and label materials completed during the day

-       Copy data to portable hard drives or DVD

-       Create MP3’s of audio, as applicable, and upload to team site

Copyright 2009, Paul Bryan, Usography Corporation (www.usography.com)
Linked In: http://www.linkedin.com/in/uxexperts

Uncategorized customer interviews, daily schedule, timeline for research

The sections of the interview protocol that are supplemental to the discussion guide are included to explain and coordinate all aspects of the data collection process. The overview sections should explain everything that will happen before, during, and afer the research sessions. It is surprising how details left out of this section can cause such havoc and confusion. Therfore, my team and I think through all aspects of the research days, from refreshments, to materials needed for exercises, to notes for security, to phone numbers of every conceivable person who we may need to call. To some extent this information can be standardized as a checklist, but there seem to be variations for every project that, if left out, come back to bite you, and some of them bite very hard!
In addition to practical preparations for the research team, the overview sections often provide a bullet point list of objectives of each interview section or module. These objectives should directly reflect the central research questions decided on in the planning phase of the project.

The topics that I include in the research overview portion of the interview protocol include:

• Materials
• Daily schedule
• Data capture procedure
• Summary of the interview sections or modules

The materials section specifies the physical items required for each user research session. Include every conceivable object, presentation device, document, list of web sites, etc., that will be required, since this will serve as a checklist the day before the session. Also specify the person(s) responsible for each type of material. Example materials that could be included in this list are:

1. Copies of research guide for research team and observers
2. Pad and pens for participant sketches 
3. Dry erase markers
4. Non-disclosure agreement
5. Agreement and release to be represented in image/video/audio
6. Receipt of payment

And so on. I’ll describe the other research overview sections in another post.

Copyright 2009, Paul Bryan, Usography Corporation (www.usography.com)
Linked In: http://www.linkedin.com/in/uxexperts

Uncategorized customer interviews, customer research planning, depth interview, materials needed for interviews, research overview, user research

Adobe to acquire Omniture in a transaction valued at approximately $1.8 billion. (http://tinyurl.com/oce2xr)

What does that mean for UX designers, marketing managers, and e-commerce strategists? Probably a broader range of tools to measure engagement and conversion in rich media. But Adobe has some pretty thick garden walls. I wonder if the other major analytics players and rich media platforms will experience a strange discontinuity in tracking and reporting capabilities?

The tagging scenarios could be a source of serious headaches. You get a standard set of analytics behavior tracking widgets when you have the right combination of front end and analytics technologies. But when you want to change either, you lose the widgets and integration with historical data.

Then again, it might be fine.

Copyright 2009, Paul Bryan, Usography Corporation (www.usography.com)

Linked In: http://www.linkedin.com/in/uxexperts

Uncategorized adobe, analytics, omniture, tags, web strategy

The most important component of planning in-depth interviews is the Interview Protocol. I’m going to describe the kinds of interview sections I include in the interview protocol to capture data that is directly applicable to innovative web design. Today, I’m going to write a brief overview of the interview protocol, in case you’re not familiar with what it is. And the next entry will start getting into the actual contents of an interview protocol that is targeted for design innovation.

The interview protocol describes in as much detail as possible everything that will occur on the day of the interview sessions. Interview protocols take many forms in different organizations and even vary within organizations for different projects or teams. Therefore, I will describe what is typical in my Usography experience and professional practice, and readers can adapt this to their particular environment as necessary. Although the format and contents of interview protocols can vary widely, their purpose is consistent: to elicit the responses and data from customer interviews that will provide the insights and understanding needed to guide design of a web site.

Interview protocols have two main components, which may or may not be physically separate documents. The first component provides overview information for the project team and stakeholders about objectives, procedures, data capture and recording, pre-determined thematic codes for note taking, interview materials, maps, security, refreshments, and any other topic that people need to know regarding the reception of participants and the interview sessions. The second component of the interview protocol is the discussion guide, consisting of the interview script as well as notes to the interviewer to ensure that topics are fully covered and that necessary data is captured. 

Copyright 2009, Paul Bryan, Usography Corporation (www.usography.com)

Linked In: http://www.linkedin.com/in/uxexperts

Uncategorized in-depth interviews, interview protocol, research protocol, user research

The concept of virtual floorspace is that attention is valuable, and that e-commerce web sites need to evaluate how successful their designs are by the return they experience on each design component. I have a lot to say about analytics scorecards for tracking e-commerce design components after first covering research methods, but wanted to take an opportunity to introduce the topic by posting  my response to George Papazian’s question in Linked In: “What are the best ways you have found to show Return on Design?”

The way Usography shows return on design is to create a metrics scorecard with very specific conversion goals, and then compare the scorecard results pre-launch and post-launch.

For e-commerce sites, the scorecard is simply based on revenue growth in categories relevant to the redesign. For example, did we sell more appliances with the new design, and if so, how many more? If the usability of a specific section was the topic of the redesign, e.g. checkout tunnel, then we’ll look at pathing and dropoff funnel pre-launch and post-launch.

In other types of sites, conversion is based on stakeholder goals that we capture in initial planning workshops, such as document downloads, time in a rich media piece, newsletter sign-ups, etc. A monetary value is placed on each conversion event, and then the scorecard results are compared with the cost of the redesign.

Previously we’ve used a six sigma assessment, but the assessment took a lot more time and energy to construct than analytics tools. Using Omniture, Coremetrics, Google Analytics, Webtrends, the data is easy to capture and compare apples to apples. However, one thing the six sigma study did better was measure the value of employee time, which may be completely missing from assessments based on agency design expenditures.

There are sometimes platform issues with tracking the same variables if the architecture has shifted dramatically from one release to the next. Another factor that complicates matters is when several aspects of the design change and you want to specifically attach a value or result to some design elements but not to others. Multivariate statistics are required to estimate the impact attributed to specific design changes.

Copyright 2009, Paul Bryan, Usography Corporation (www.usography.com)

Linked In: http://www.linkedin.com/in/uxexperts

Uncategorized analytics, design roi, e-commerce, metrics, return on investment, scorecard, user research

It is unlikely that a company conducting customer interviews is going to spend time cold-calling the general public to recruit participants. Typically they contact a market research recruiting company. These companies have a database of people in the general public who are willing to participate in research studies, such as a customer interview.

If an outside agency is doing the recruiting, then I usually write the screener in the form of a script. The recruiter calls potential participants and walks them through a series of questions. People are dismissed if they don’t match the criteria. If they do meet the criteria, then they are tallied so that the minimum and maximum number of people specified for each criteria is adhered to.

I have experienced a couple of problems when interviewing participants recruited by outside agencies. The reason that these problems arise is straightforward, but the solution is not. Recruiting companies have a vested interest in finding participants who match the screening questionnaire, while at the same time spending the least amount of time and money on the search. That means that their database of known participants is the first stop for your research sample. This database includes people who they’ve used before for similar projects, who in some cases “do testing” for extra cash. Such people are usually determined to take on whatever persona the researcher seems to want them to have, so that in the end, the data from these sessions is worse than useless, because it is misleading.

The second problem I’ve experienced is related to the first. The research recruiter’s database may include people who are unemployed and who desperately need the incentive money. They give very brief answers, waiting for the session to end so they can collect the money or gift card and disappear.

Of course, there are recruiting companies who are very adept at what they do. They try hard to eliminate the two types of problem participants mentioned above. I don’t hesitate to call them when I’m not able to recruit more directly and precisely from a customer database. But I add a fudge factor to the number of participants, anticipating that at least 10% of the participants will not contribute useful information to the study, and so adjust the total number of recruited participants accordingly.

Copyright 2009, Paul Bryan, Usography Corporation (www.usography.com)

Linked In: http://www.linkedin.com/in/uxexperts

Uncategorized customer research, Depth interviews, e-commerce, e-retail, recruiting agencies, recruiting participants, user research

One way to ensure a relevant sample of participants is to select them from your customer database. You can establish precise criteria and then query the database for matches. This is my preferred method for recruiting participants for customer interviews, because the sessions tend to be very productive. This isn’t always possible or desirable. In some companies, privacy restrictions make it difficult to contact customers based on purchase history or other interactive behaviors. These companies don’t want their customers to feel that they are being observed while they shop on the web site. Of course, it is questionable whether there are still many customers out there who do not realize their every move on web sites is being tallied and reported.

One case where the customer database is only a partial solution is when you want to understand both customers and non-customers. In this case you can use the customer database to recruit customers, but must rely on other means to obtain the non-customers. One resource for the non-customer participants is discussed in the next section.

Copyright 2009, Paul Bryan, Usography Corporation (www.usography.com)

Linked In: http://www.linkedin.com/in/uxexperts

Uncategorized customer research, Depth interviews, finding participants, recruiting, screening criteria, screening participants, user research

When you have a completed and approved screener, then you can start the recruiting process. Depending on your organization, budget, and project context, your team may do the recruiting and scheduling, or you may have an outside agency handle part or all of the process. In either case, someone will contact prospective participants by phone or email, and then use the script described above to determine if they fit the screening criteria or not.

The advantage of recruiting by phone rather than by email is that you may be able to contact customers, screen them according to the criteria, and schedule them for a research session all in one sitting. The problem with this approach, on the other hand, is that you often don’t reach people the first time, and so leave a message on a machine or with another person who answers the phone, and this can get messy. Using email to make the first contact is simpler in terms of logistics, and can be done much faster. The problem with using an email as the first contact is that it is hard to distinguish your recruiting email from the onslaught of other emails from similar companies, with wording that appears similar to yours, but whose message customers are avoiding like the plague.

If you are conducting the recruiting in house, and a prospective participant has passed through all of your screening criteria (i.e. they weren’t screened out), then you will need to discuss the time and location of the interview. Scheduling gets tricky because your participants probably have a few different time slots available during the research period, you want to nail down the time and date of their sessions, but then you may find other participants who can only come during time slots that you’ve already scheduled. There may be some opportunity for back and forth maneuvering, but in general you are stuck with scheduling decisions you’ve made. You can deal with this by asking participants for several time slots, tell them you are scheduling them at a particular time on a particular date, but that you may need to contact them again to move if they are still available at the alternate times. Behind the scenes, the people making these calls have the screener script, a table of screener criteria with maximum and minimum numbers of participants who match each of the criteria, a calendar, a map of the research site and surrounding area, and a scheduling sheet with all the dates and timeslots (and lots of scratched out writing).

Once people agree to participate and are placed into the schedule, you should follow up with an email to confirm the details. The follow-up email should include:

• A detailed map to the research location
• Phone numbers of people to contact if there are challenges getting to the site
• Phone numbers of people to contact if they have any questions prior to the sessions
• Any special instructions required to find the specific room or to get through security
• Reminders about any homework that needs to be completed prior to the session
• A passionate plea to be on time, perhaps with some added threats about reduced compensation if they are late
• Notes about parking, if relevant (in big cities it is very relevant!)

Send another email the day before the session that duplicates this information in a slightly different format to get their attention. If something has occurred between the time they agreed to participate and the day of the session and they are not planning to attend, you really need to know this. They may be embarrassed, so you’ve got to craft your message to get their attention. You may want to call each participant the day before their session to confirm that they will be attending.

Uncategorized