Thoughts @

integrated mobile research & testing

Traveling through a remote area in the south of Spain has caused me to rethink the importance of context in the mobile user experience. Without an active internet connection (only available through a much-used public hotel lobby) I have started to rely on my mobile as more than just a business tool or communication device – it is a lifeline. I have been able to recieve urgent SMS messages and email through my P910a and it has offered much comfort knowing I can reach my colleagues, friends and family at any time. Context is the missing piece of most testing and research methods, because during a normal usability test or a short interview, it is impossible to gain insight on the true emotional impact and lifestyle needs the mobile device has on it’s owner.

Surrounding the entire mobile user experience, and setting it apart from the experience of using other web-based or electronic devices, is the all-encompassing factor of context. Since the product or service is expected to be accessible at all times and in such a variety of situations, insights into the end-user’s lifestyle is far more critical in the mobile industry than it is during the development of more traditional products. How, when, why and most importantly – where interaction takes place – are major considerations that are often overlooked or disregarded during the authoring process. Accurate testing should always consider context.

Integrated Research and Testing

For best results, the user-centered approach employs not just traditional marketing research techniques, but also contextual inquiry, or methods that involve asking questions and observing within the subject’s own context. Traditional market research and focus group testing are commonplace in the product and service development category, and yield quantitative information and general information that informs a product team and allows them to target their audience and approach. Contextual inquiry yields a significantly different type of information than traditional research techniques. The two types of contextual inquiry employed in an effective approach are ethnographic-based research and usability testing.

Ethnographic-based Research

Ethnography as applied to market research involves a “A descriptive, qualitative research methodology for studying the customer in relation to his or her environment. Researchers spend time in the field observing customers and their environment to acquire a deep understanding of the lifestyles or cultures as a basis for better understanding their needs and problems (Product Development and Management Association, 2004).”

Ethnographic-based research is immersive, one-on-one observation in a subject’s own environment. Also called ‘naturalistic’ research, this method pinpoints emerging global trends by studying firsthand how users incorporate products and services into their daily lives. It is a powerful alternative to traditional marketing analysis, which studies only patterns and percentages rather than identifying the attitudes and motivators behind the data. While this type of study once took months to years to complete, the pace of the mobile industry has influenced ‘rapid ethnography’ studies. These studies compress the period to a matter of weeks by employing multiple teams concurrently conducting observation and inquiry in multiple countries or locations.

Rapid ethnography is an effective method of contextual inquiry as it results in a balance of both rigor and relevance in data capture and analysis. It is characterized by small teams of multi-disciplinary experts, working collaboratively to observe, collect insights and data and determine relevance to the product or service they are working on. The research is usually conducted in a matter of weeks, instead of months. Studies such as this yield insights not possible without immersion into the attitudes and behaviors of the participants’ lives. Rapid ethnography is one method of contextual research that can be applied quickly and at a fairly low cost to help inform the conceptual development of new products and services.

Usability Testing

Usability testing is a method by which users of a product are asked to perform certain tasks in an effort to measure the product’s ease-of-use, task time, and the user’s perception of the experience. Usability testing can be done formally, in a usability lab with video cameras, or informally, with paper mock-ups of an application or Web site. Changes are made to the application or site based on the findings of the usability tests.

Complimentary to ethnographic-based research is usability testing, based on human factors engineering to test the literal ‘ease-of-use’ of a product based on task-oriented testing and observation. Under this method, one participant at a time is asked to perform certain tasks in an effort to measure the product’s ease-of-use, task time, and the user’s perception of the experience. Usability testing can be done formally, in a usability lab with video cameras, or informally, with paper mock-ups of an application or Web site. Developers make changes to the application or site based on the findings of the usability tests. Traditional usability testing draws from the field of human factors, an engineering-based evaluation system set in a laboratory setting.

Characteristics of usability tests are that they are designed to be one-on-one (one participant and one moderator/evaluator) and to be taskoriented. One variation of testing involves heuristic analysis, also known as expert reviews, in which specialists in the field of human factors engineering or cognitive psychology conduct product walk-throughs using their own systems of system analysis and feedback. When conducting expert reviews, it is often helpful to have two to three experts walk through the product, discuss the issues and come up with mutual recommendations for improvement.

There are always more reasons to test not than to test. Many companies think that by conducting focus groups or gathering marketing data, they are conducting usability tests. This is not the case. Usability testing involves one-on-one task oriented tests that measure ease-of-use. This type of testing provides insight on how minor changes on an interface, outside of the engineering mindset and environment, can be made simpler, thus raising the odds of user adoption.

Mobile Testing Considerations

Testing for usability on mobile devices is a layered proposition. Besides testing for ease-of-use, other factors such as learnability, reliability, efficiency and likeability are also measured. For mobile devices specificity, predictability and expectations also set parameters for testing. Creating the proper context for testing is challenging, as mobile devices are generally used while on the move, and often while doing other things, such as walking or driving a car. Other factors such as culture, aptitude and personal mobile history (how many mobile phones have you owned, how long have you been with one carrier, etc.) also affect the interaction. There are a number of issues surrounding usability testing for mobile devices. Usability tests rely on establishing real-world scenarios for participants to mimic in formal lab settings, informal office settings and sometimes in the participant’s own environment. Testing for mobile devices is challenging due to the number of scenarios surrounding the use of the device, the number of phone types available and the demographics and psychographics of the participants themselves.

Screening for Participants

Due to greatly varying audience segments, and the number of handheld devices and types of tasks being performed on a regular basis, it is important to screen and recruit test participants in as targeted a manner as possible for both user type and handset type. Participant screening for mobile devices rely on a lot more than general demographics such as age and occupation. Understanding lifestyle habits, mobile phone history and current phone usage helps to establish the characteristics of the targeted audience. Most mobile devices are second or third generation, meaning the user has replaced his or her handset once or twice already, and is generally aware of the device type category and general functionality. When moving into a new phone category (for instance, moving from a SonyEricsson T16 series to a Sony Ericsson P910a smartphone), novices or new users are the target audience. Clearly defining the needs of the end users and their goals is part of the test plan.

Test Plan Creation

Each testing situation is different, along with the goals and demographics. With usability testing, it is important to include at least five individuals fitting a specific demographic or psychographic profile. This said, there are some specifics when writing a test plan for a mobile device. The development of a test plan should view the goal from three perspectives. The first is user motivation – forced or voluntary, internal or external. The second is use-context, making sure testing takes place in as natural a situation as possible. The third perspective notes user actions, with an understanding of what the user is trying to accomplish and how.

Conducting the Test

Replicating actual context and situation is important when conducting a mobile usability test. The moderator needs to understand that generally mobile phones will be used while multitasking — which means while walking, driving, and even conducting another task on the same device concurrently. Replicating this experience in a natural setting is moderately difficult. First, the participant is asked to stand, since sitting and completing a single function on a device is very rarely done.

Iterative Cycles and Ongoing Testing

As with most development cycles, ongoing testing and an iterative approach to testing and feedback should be incorporated into the process. Creating a toolbox of testing methods starting with usability testing, contextual interviews and ethnographic-based observation gives insight and feedback necessary to refine early in the process. At later stages of development and release, ‘rapid usability’ and ‘rapid ethnographic’ methods of research are used to facilitate refinements or validate the product or service is meeting the end user’s needs. With the fast-paced and frenzied nature of the mobile industry, time-to-market and outpacing the competition is often a driver for development and deployment of new products and services. While the pressure is authentic, creating a mobile experience that is positive and integrates into the individual’s actual lifestyle is an even greater measurement of success.

Posted on March 3rd, 2006 in Thoughts
Tagged as , , , , ,
Written by Kelly Goto

4 Responses to “integrated mobile research & testing”

Comments

  1. fabien says:
    March 3rd, 2006

    Field testing might be a third valuable methodology to analyze mobile activity. It can be setup with rapid prototyping techniques and generates a mix of qualitative and quantitative data. For example, it might be advantageous to support interviews by self-confronting the users to a replay of what they did (based on the quantitative data).

  2. Hemanshu Narsana says:
    March 3rd, 2006

    Well written, though a long read nonetheless. A mix of qualitative and quantitative analysis is the need of the hour. There are enough fights going on about which technique is better, or which yields better results.

    People obviously don’t seem to be getting that idea that they can get the best of both worlds, and can have the cake and eat it too by combining both qualitative and quantitative methods.

    Great work, and good writing. I’ll be back!:)

  3. Kelly says:
    March 3rd, 2006

    Fabien, would like to get more details about your thoughts on ‘field research’ techniques. We tend to lump together our concepts of ‘field’ and ‘ethnographic-based’ research, and would like to help define these techniques in a more comprehensive and clear manner for the non-academic crowd.

  4. Kelly says:
    March 3rd, 2006

    Hemanshu, thanks for your comments. It IS a long read and for that I apologize. Some entries will be shorter, this was actually an essay format and I hope to put all of the information/writing together into a downloadable PDF for a more comprehensive read. Any other feedback is very welcome. :-)

Related Stuff