Virtual Floorspace

Depth interviews typically result in a mountain of data, including researcher notes, results of exercises and other research artifacts, video or audio recordings, and transcripts. Reviewing and analyzing this data is very time-consuming. You need a plan before the interviews start for what will be captured and how it will be processed prior to presenting results. As a rule of thumb, we allocate at least two hours for every hour of interviews for data review and analysis: 1 hour to review, 1 hour to analyze and formulate findings. It is easy to underestimate the time this will take. Watching a one hour video typically takes more than an hour, because you not only have to watch it through to make sure you catch all the information, but you will probably start and stop it to highlight the most significant sections, write notes, get the timecode precise for the clips you want, etc. Then you need time to formulate higher level findings, produce clips, and trim down the highlighted data for final presentation.

An important aspect of the data analysis plan is data reduction, i.e. how you will go from raw data to the first step of processed data. Usography standardizes this procedure so that all researchers use the same approach to data analysis, for comparison and corroboration purposes. Each researcher goes through the detailed transcript and notes at least once, recording notes about themes and high-level findings as we go. We then note recurring topics and place them into a table. Example data elements that we look for consistently across all participants are:

• HHI
• Internet IQ
• Shopping IQ
• Goals
• Drivers
• Categories
• Business value
• Typical purchase process
• Barriers
• Dropoff
• Requested content/features
• Competitive sites
• Quotes
• Method for tracking this user type with analytics

If we have previously conducted research in the area, or if we have done extensive secondary research prior to conducting primary research, we may have a preliminary coding system going into the study.  In this case, research assistants are labeling notes with these codes as they record or transcribe them. Those codes are then used to form the data summary table for each participant. For example, in cell phone research we may designate “communication enabler” as a predefined code. Research assistants type “ce” everytime a participant makes a comment that matches this theme. This is part of the analysis planning that takes place before we conduct the research, and is a big time saver. 

Written by Paul Bryan, Usography Corporation (www.usography.com)

Linked In: http://www.linkedin.com/in/uxexperts

Uncategorized coding, data analysis, Depth interviews, design research, themes

As augmented reality and virtual reality systems gain prominence and are adopted, a new set of usability requirements are needed to assess ease of use.

My three children, currently 7, 10, and 12 years old, are very similar in their usage patterns of web sites. Ok, Club Penguin doesn’t present a lot of different types of interactions, but still, my kids share certain demographic and psychographic characteristics, as well as group experiences, that have homogenized their cognitive processes and expectations. However, as they interact with spatial systems, those requiring 3-dimensional perception and system/reality visual transisions, they vary widely in their interaction styles and abilities, and a system optimized for one would not be optimal for the other two.

A couple of situations arose today in a big box store that made me think about this.  The first was in the grocery line. I always have my kids do the self-checkout and payment, up to the point where I need to sign the pad. They do not perform these tasks equally quickly or easily. In part there are age effects, but there are other factors that do not seem to be age or gender dependent.

The second example was the video kiosk. They need to use a screen to select a movie, and then need to do the credit card payment, and finally need to eject or insert videos. Again, their 3-d perceptions differentiate their abilities. I also saw this when they play sports on the wii.

Of course, these examples have nothing to do with augmented reality, which has more to do with the underlying digital data associated with a physical 3-dimensional space or context. But they made me think about aspects of cognition that will determine ease of use of these near-future systems, and how inadequate today’s heuristic evaluation criteria are going to be for assessing them.

Written by Paul Bryan, Usography Corporation (www.usography.com)

Linked In: www.linkedin.com/in/uxexperts

Uncategorized

A critical step in any qualitative research study is selecting participants. In quantitative research, the selection is random from a given pool of participants, and the number of participants is high. But in qualitative research, you will probably work with a small sample of 10 to 100 participants, so who they are and their responses will have a much larger impact on the findings than they would in a quantitative study.  

Participants for qualitative research are usually selected by using a questionnaire called a participant screener. The participant screener contains a series of questions based on pre-determined criteria that, when asked in sequence, results in a set of research participants that have the characteristics you’re most interested in, and has the proportions of those characteristics that are representative of your target user population.

The screening criteria can be very simple for small projects with a clearly delineated, homogeneous user base. Research for more complex web sites that have widely varying audiences performing complex or high-value tasks typically call for more precise screening criteria. An example of simple screening criteria would be :”Five males and five females who work in reception at a SeaSide Resort facility in Florida.” An example of more specific selection criteria would be: “Females between the ages of 18 and 25 who have purchased 3 or more apparel items within a 3 month period from a web site, spending a minimum of $40 per item.” Neither approach is right or wrong; their correctness depends on the scope and objectives of the research. The screener criteria should result in participants who vary widely in the characteristics that are most likely to impact the way that people will interact with the final design.

There isn’t a single format for participant screeners. They are structured to meet the needs of the project. If your team is doing the screening, then the document might be a simple table. The first column lists the screening criteria, e.g. Age. The second column has the quota for that criteria, which means the number of participants needed for the study who have particular values of that criteria, e.g. 5 people 20 – 29 years old, 5 people 30 – 39 years old, etc. The third column is a tally of people recruited so far who have the specified values for the criteria. The fourth column may have comments about the criteria, e.g. how the ranges and values can vary and yet still be acceptable.

If an outside agency is doing the recruiting, then the screener is written in the form of a script. The recruiter calls potential participants and walks them through a series of questions. People are dismissed if they don’t match the criteria. If they do meet the criteria, then they are tallied so that the minimum and maximum number of people specified for each criteria is adhered to.

If the criteria are multi-tiered, meaning more than one factor in each criteria, then a branching script is necessary. For example, if you want to find 5 women who purchased 3 or more pairs of shoes who have a household income of 75k or more, and a maximum of 2 women who purchased no more than one pair of shoes in the past year but who have a household income of 200k or more, then the screener needs to have a series of related questions that branch from one to the other. For every answer that the participant gives, there is a corresponding script answer, even when they are being excused from the study. So for example, when a question asks potential participants their age, the person conducting the screening interview needs to have a response for whatever the person says. This can be more difficult than it sounds. In addition to having a reply for anything the potential participant says, there is also a branching instruction that tells the screener what to read next, and what to tally, and the maximum or minimum number of participants who answered in the same category who are needed for the study.

Written by Paul Bryan, Usography Corporation (www.usography.com)

Linked In: www.linkedin.com/in/uxexperts

Uncategorized Depth interviews, screener, screening criteria, screening participants, user research

Jennifer Wolfgang posted an interesting question to ixda about communities that are started by companies who sell products. Should companies differentiate their community site from their primary site? In terms of e-commerce floorspace, should the community space be tightly linked to the retail section?

I think the question is the degree to which you want the community to appear to be a separate entity from the primary offering. The examples below illustrate a range of options from completely integrated to almost completely differentiated.

Amazon completely integrates community into into its product catalog.

http://www.amazon.com/gp/product/B000EZYKTS/ref=s9_al_gw_tr02/179-0449618-1343656?pf_rd_m=ATVPDKIKX0DER&pf_rd_s=center-3&pf_rd_r=1FG9MJMZR5FS1MFW2C5A&pf_rd_t=101&pf_rd_p=488826231&pf_rd_i=507846

HP keeps the header on the landing page of communities…

http://welcome.hp.com/country/us/en/welcome.html#Connect

…but then uses a streamlined header once you’ve selected a community

http://www.communities.hp.com/online/

Dell uses a market segment approach to global navigation, similar to TrendMicro’s global nav. Once you go to the community piece, you see a streamlined header that makes it seem like you are on slightly more neutral territory.

http://en.community.dell.com/forums/

Best Buy has a community site that is clearly distinct from its primary e-retail site, but which does not sub-branded to the extent that Sears is.

http://www.forums.bestbuy.com/t5/Computers/bd-p/Computers_New

Pampers does the inverse of Amazon, integrating its product offering into the community piece.

http://www.pampers.com/en_US/Shop

Sears uses a completely different visual treatment for its community, giving you the impression that you are closer to the other customers and a bit removed from the commercial entity Sears.

http://www.mysears.com/

See original post:

http://www.ixda.org/discuss.php?post=45003

Written by Paul Bryan, Usography Corporation (www.usography.com)

Linked In: http://www.linkedin.com/in/uxexperts

Uncategorized community, e-commerce, social media

Depth interviews are a useful research technique when user needs, motivations, usage context, and personal success factors need to be understood in a greater degree of detail than is possible using other research methods, such as surveys. Depth interviews usually last between 60 and 90 minutes. The research team conducting depth interviews usually consists of an experienced moderator or senior researcher who facilitates the interview, and a research assistant who captures data.

The volume of data in depth interviews is high, and participants tend to wander around various topics unless strongly facilitated to stay on track. Therefore, having a senior researcher on hand to conduct the interviews is highly desirable, preferably one who is skilled in both interviewing techniques and web design (wink wink). The research assistant captures notes about what the participant says and does, as well as observations. Video or audio recording equipment is typically used to supplement the research notes with a visual and/or audio transcript.

Additional components of a successful depth interview include:

• Participants who accurately represent the audience that the research is targeting
• Detailed research protocol
• Materials for completing research exercises

Interviewing people in the place where they work, or purchase goods, or use the information tool you are designing is called an in-context interview. Conducting the research in context provides many clues about the kind of design that will be successful, in terms of how users expect the web site to communicate with them, and the kinds of additional support they may need beyond basic transactions. When interviews are conducted in context, participants may refer to artifacts present in the activity context that affect their ability to use the information system. They may also point out offline or peripheral factors that significantly impact their ability to successfully use an online system, which may not be discussed if the interview were to take place in a meeting room or lab.

Because of the logistics and expense involved, depth interviews involve a lot of planning to run smoothly. I’ll write about the depth interview process as a set of manageable steps and provide examples to help you plan for successful depth interviews in subsequent blog posts.

Written by Paul Bryan, Usography Corporation (www.usography.com)

Uncategorized Depth interviews, design research, in-context, methodology, qualitative research, user research, web site design

Yesterday I wrote about the importance of selecting the most appropriate research method for the problem or opportunity at hand. A significant step in that process is deciding whether the research questions are best answered by qualitative research or by quantitative research.

Some people assume that there are two categories of research methods, some of them qualitative and the others quantitative. However, it isn’t the methods themselves that are qualitative or quantitative. It is the way in which the data is captured and analyzed that is either qualitative or quantitative. For example, if a survey’s question structure or sample designation is not statistically sound, then it is actually qualitative research, not quantitative, despite the table of percentages that is reported at the end of the project. Interviews, on the other hand, could be conducted in a way that would be quantitative and statistically relevant; I haven’t found a client yet who was willing to pay for quantitative interview data, though…

Quantitative research is different from qualitative research because it has a known confidence interval, and it is relatively small. The confidence interval is typically reported as +/- percentage points of error, so, for example, a result with a +/- 5% margin of error has a confidence interval of 10 percentage points. In qualitative research, the confidence interval is either unknown or it is relatively large. This is usually due to the reliance on rich data, or non-numeric data, which leads to less precision in the resulting data. Another reason that the confidence level is either unknown or very large in qualitative research is because the sample size is typically smaller than in quantitative research. Increasing the sample size would lead to a smaller confidence interval, indicating greater precision in the results. Sample size in qualitative research is usually limited by budget considerations or because of the time it would take to capture and analyze rich data from a large number of participants.

Researchers whom I’ve met in corporate settings seem to favor quantitative research over qualitative research, because they can express their results with confidence. However, qualitative research, if conducted properly, can provide insights that quantitative research will not provide, or cannot provide in a cost effective manner. Qualitative research can help the design team:

- Discover unseen factors that impact whether users will find a given system easy or effective.

- Understand unfamiliar phenomena, or phenomena that cannot easily be measured.

- Model interactive behavior, as opposed to determining the statistical prevalence of a given attitude or preference.

Examples of research methods that are typically associated with qualitative data collection include:

- Depth Interviews

- Intercept interviews

- Ethnographic research

- Focus groups

- User testing (usability testing)

- Paper prototyping

Uncategorized

Many customer or user research efforts allocate the lion’s share of hours to data collection, but the validity of results is dependent upon activities that occur much earlier in the process, in the preparation phase. The most important preparation step is pinpointing exactly what the research will measure, and how that data will be used to shape design work.

The problems that lead to customer research are typically design issues that either have led to lower adoption (people won’t use the system or device because it is too difficult or clunky) or lower than expected conversion rates (the percentage of people that complete desired transactions or events). optimizing the user interface.

Many research projects quickly, or even instantly, gravitate to a specific methodology. A list of research topics is often created to guide the design of the data collection instrument (a survey, a customer interview protocol, etc.) But there’s a critical step missing. Before selecting a research method and designing the instruments and materials, the research lead should zero in on precise research questions that need to be answered, and select valid methods that will answer those questions with an acceptable level of reliability.  Instead, what often happens is someone says, “We have a survey tool, so let’s use that to conduct the research.” Unfortunately, a survey may not obtain the answers needed to guide the design. This is especially true when the data required for design innovation is not something that customers can state about themselves, but which must be observed in context by experienced researchers. It’s analogous to using a heart rate monitor to measure glucose levels, or a ruler to measure blood pressure. To ensure a successful customer research project, the data collection method has to be selected to answer specific research questions that are formulated to address a specific set of design issues.

Uncategorized research methods, research questions, surveys, user research

Optimizing your virtual floorspace requires an in-depth understanding of customers. To gain this understanding, companies undertake the expensive process of collecting customer data. For customer data to impact UX design, it must be collected in a manner that answers specific design questions. As a design strategy consultant, I am continually presented stacks of data about customers. Unfortunately, much of it is irrelevant or tangential to the understanding that is needed to optimize design work. Some of the research methods and data collection techniques that have been useful in Usography’s design strategy practice are listed below, along with a brief explanation.

Depth Interviews

Depth interviews are helpful to get a better understanding of why customers behave the way they do, and for developing mental models that serve as a basis for interaction design. Depth interviews are most useful at the beginning of a project, before the product has taken shape.

Web analytics

Web usage data can be collected, aggregated, analysed and reported by a number of web analytics packages. Every action taken by a site visitor can be tracked by most of these packages. They vary in their ability to analyze and visualize usage data. Some packages allow you to zoom into individual sessions and observe customers who meet specific criteria.

Surveys

Surveys are easy to conduct using tools like Survey Monkey. However, they are not easy to construct so that their results are meaningful to designers. The construction of a reliable, valid survey requires the support of people who have a significant understanding of statistics.

User testing

Observing customers while they are using the tools you (or others) have designed can be invaluable, but drawing conclusions that are useful to designers depends on framing the exercises to evaluate specific design issues, rather than randomly walking through a series of site tasks. User testing should begin with the first prototype, and be repeated with each major revision, including the final release.

Ethnography

Ethnographic studies involve the capture of rich data related to the full context of an activity for which a web site or information system is being developed. Originally an anthropological method for studying little known cultures in undeveloped nations, ethnographic studies have been gaining favor across a wide range of commercial design situations. Ethnography is most often used as a data gathering tool when the cost of research is significantly outweighed by the potential revenue of creating a superior product. Ethnographic methods may include one or more of the following: observing customers in a natural setting (participant observation), asking them to keep journals, webcam diaries, video observation, artifact collection, or in-context interviews.

A/B and multivariate testing

Design options can be empirically tested by introducing one variation on a design element, sending some users to the old design and some users to the new design. Statistics about the relative conversion rates reveal which design, A or B, is likely to be more successful. The duration of the test and the number of users sent to the experimental design, option B, need to be sufficient to obtain statistically valid results. Tests I’m aware of have sent up to 90% of the site traffic to the current site( option A) during the test, and 10% of the site traffic to option B. The specific test parameters are closely related to the non-test state analytics of the system being tested.

A/B testing usually requires approvals from a variety of stakeholders, and is expensive because of the resources involved and potential negative impact. Therefore, companies often want to test several design variations at once. When more than one design variation is tested, the math becomes more complicated, because it is difficult to tell which change has resulted in the conversion rate differences. Multivariate analysis is required to reliably sort out the results.

Help desk / customer service

This is one of my favorite streams of customer data, and one that is often overlooked in the design process. Customer service reps and logs are excellent sources of customer feedback related to design issues, although they are not necessarily the people who should be involved in developing design solutions.

Social media / discussion forums

Many customers who would never volunteer for a depth interview or user testing express themselves freely online in discussion forums. While this is great fodder for strategy meetings with executives, the issue of prevalence is one that suggests that quantitative validation is needed before any serious conclusions can be drawn.

Secondary research reports

Companies like Forrester, comScore and Nielsen produce research reports with customer data and trends that are very relevant to web design. Data about the needs of specific segments of the population is helpful for designing solutions for those segments of the population.

Market data

I use market data to understand the size of the market, the potential of the market, the demographics of customers, and trends. However, I don’t usually draw many conclusions about specific user experience design issues from such data. It is usually too highly aggregated for this purpose, and does not provide the kind of behavioral data I need to make design strategy decisions.

Uncategorized

Floorspace is a concept that is highly developed in retail stores, with big box retailers leading the way. Before a new display or product can take up costly space in the store, its probable return is analyzed using available data. Once implemented, its performance is tracked to determine how profitable it is compared with other store areas. Top performing displays are replicated and refined. Poor performing areas are improved through changing promotional tactics or changing products.

Web sites also have specifically delineated areas devoted to different features, functionality, and content. Similar to a retail environment, these areas comprise the web site’s offering. For commercial web sites, this offering is expected to bring some kind of financial return.

Many companies I consult with do not apply the kind of rigorous analysis to the space used on a web site that retailers apply to space used in their stores. The reason is simple: space on a web site costs very little money. E-retailers call the internet the “endless aisle.” What many site owners and sponsors fail to realize is that while space is virtually free and unlimited, customer attention is not. In test after test, I have witnessed the extremely selective attention that customers exhibit when using web sites, in comparison with the browsing behavior that is seen in physical retail outlets. Attention engineering is necessary to realize the full potential of the web site user experience design. Attention engineering can be accomplished using best practices, but a more reliable approach is to base it on customer data that is carefully collected and analyzed. Basing user experience design decisions on customer data is the premise of Virtual Floorspace.

Uncategorized

Virtual Floorspace is a blog about web site design strategy. It describes approaches to optimizing design, primarily through the collection and application of customer data. “Customer” in this case can be any person whose usage of the web site results in a financial return to the company or organization that sponsors the web site. Virtual floor space focuses on ways to maximize the financial return associated with web design.

Uncategorized