I’ve outlined several of the most common user experience research methods which are most prevalent and relevant today. I’ve weighed pros and cons, and included high-level cost breakdowns.

Heuristic Evaluation

Heuristic evaluation is an informal method of discovering and identifying usability issues in which one or more “expert” individuals inspect and evaluate the product. Evaluation can be completed within a couple of hours (more or less, depending upon the scope and degree of complexity) and may or may not follow heuristic guidelines established by industry experts (such as Nielsen’s heuristics [standard], Gerhardt-Powals’ cognitive engineering principles, Weinshenk & Barker classification).

PROS

This can easily be performed at a high level and oftentimes is already occurring. Stakeholders and product experts frequently weigh-in on the design, functionality, and usability from start to completion of most projects. Fortunately this seldom requires any additional cost to the company — hooray!

CONS

Not every individual will uncover the same usability problems. The term “expert” is subjective and findings often vary between individuals. It is up to the product team to create a systematic approach in order to uncover all usability problems.

Customer Feedback

Establishing a foundation for good customer service is vital for the developing and maintaining a user-centered product. Listening to your customers will provide direct insight into determining the user’s needs.

PROS

Most users are eager to give feedback regarding pain points to Customer Service. Most companies already have the tools in place to measure and determine these needs — at no additional cost!

CONS

Customer feedback can be extremely subjective, what’s good for the goose isn’t always good for the gander. Allowing designer(s) accessibility to customer feedback and incorporating it into the team’s workflow may be a challenge in an environment that is not agile.

In-Person, Email & Intercept Surveys

Surveys can be collected in-person, via e-mail, or through portals within the platform itself (intercept). I’ve found that surveys yield the best results when they are short, granular and clearly stated. This often results in ideal objective answers surveys which are especially fast to complete, review and analyze.

PROS

Surveys are typically free or inexpensive, quick to produce, and enable access to remote users. The results are received asynchronously (sit back & relax!), and can be quantified.

CONS

What people say versus what people do are very different things and can often yield inaccurate results. Poorly worded questions can also alter a user’s response (aka interviewer effects). Most people dislike taking surveys which can create a skewed demographic pool. This is why it’s important to create a short survey so that users will be more inclined to participate in the future. Incentives should also be avoided if possible, to prevent users from feeling responsible for a particular outcome.

(User) Personas

A user persona is essentially a snapshot of what the standard user of a product looks like. We will take the following information and create a user’s back story to add a more human element to the product.

Marketing

“Our target demographic is female, American, between the ages of 36–48, and makes between $55–75k/yr.”

Persona

“This is our user, Becky. Becky is 41, Caucasian, and lives in Los Angeles with her husband, 10 year old son and their dog. Becky loves technology, uses all the latest apps and always upgrades to the newest iPhone. Despite loving technology, she maintains balance by being frugal in all other aspects of her life. She earns $75k/yr with a household income of $115k/yr and holds a B.A. in Communications.”

The user persona is based on actual data from the marketing and/or analytics teams, though some information can be improvised for sake of a 360º view. Using a persona will help answer design-related questions by keeping the user in mind at all times. As designers, we constantly ask ourselves and our stakeholders about how the user will interact with the product. “Would Becky use this feature? Could Becky understand this without a tutorial?” Oftentimes, multiple user personas are created to represent the majority 2–3 demographics, as well as an outlier. For example, multiple user personas could be represented by making this assumption: “Becky, Juan and Brenda would be able to understand this easily, but Henry may have some difficulty.”

PROS

Oftentimes, this requires no additional cost. Creating a user persona can be quick and easy because when using data that already exists within a company’s market/client research. I recommend printing out and posting the user persona(s) so that it’s easily accessible, and when questions arise, the conversations are always user-focused.

CONS

The user persona is not a real person, it is purely hypothetical. You can’t gather any additional information about them, nor ask specific user questions.

Individual Interviews

User interviews are exactly how they sound and more complex than they may appear.

PROS

Useful for determining high-level ideas/general & overall attitudes. The more granular the questions the more fallible the responses.

CONS

Referencing the past and accidentally leading the interviewee are common mistakes in interviewer questions/techniques. Specs/mocks don’t work, since they may inevitably change in development, rendering interview results useless. In combination with other research methods, interviewing can provide useful input. As a singular method, agile development and rapid prototyping (among others) will provide much more useful feedback.

Parallel Design

This is when multiple designers all attempt to create alternative designs (can be done by individuals or by multiple teams) simultaneously without input from each other. This is to see where consistencies and design concepts are aligned and what areas deviate and require rework. The following iterations thereafter will contain the majority-ruled design concepts. Focus can then shift to creating alternative/creative/hybrid solutions in the areas that weren’t aligned.

PROS

Uniting and validating design concepts and solutions. Time, energy and focus devoted to the areas that need the most work; oftentimes resulting in a team’s most creative solutions and work. Also, free!

CONS

Can be time-consuming and not always efficient or effective with smaller teams/departments.

Wireframes

A wireframe is a visual representation of what a webpage or app will look like before it is built by developers. Wireframes can be represented by varying levels of fidelity; from rough hand drawn sketches, to computer drawn with grayscale block image representations, to high-fidelity, pixel-perfect images that look exactly like the site itself. There are a vast number of wireframing tools available with varying degrees of sophistication (Balsamiq, Axure, and Sketch are examples of some popular tools).

PROS

Wireframes allow us to see a visual representation of how all of the content will fit on the page and how these pages will connect to each other. Because tech development is more time consuming than wireframing, it’s much more efficient to get the layout designed first as it often take multiple iterations before determining the best placement. Wireframing is also a helpful visual tool to assist communication with developers and stakeholders, especially as the designs become increasingly complex.

CONS

At minimum, lo-fi wireframe sketches should be an established part of any workflow. As it is crucial in determining the layout and flow of any project and communicating this visually to the development teams and stakeholders.

Clickstream/Click Path Analysis

Essential in e-commerce and ad-click business models, click paths can be monitored using analytics programs such as Google Analytics. This allows us to understand how the user is navigating the site by tracking where and what they click on. We can then determine what features are popular as well was what they’re not using or are overlooking within the site.

PROS

The data retrieved is quantifiable and allows us to establish KPIs. We can analyze the path of the majority, including what features and functions are being used most as well as which are being underutilized or ignored.

CONS

Analytical data doesn’t explain the reasons behind user behavior, and the more complex clickstream analysis tools will require additional costs (for example, CrazyEgg.com = $100/mo for their Pro tool).

A/B Testing (Multivariate Testing)

For A/B Testing (also known as “multivariate testing”), we create and release 2+ versions of the same design and measure which one performs best. This is mostly used for short-term tracking and conversion rates (i.e. e-commerce, email newsletters, subscriptions, downloading whitepapers, etc).

PROS

It is free, or cheap, and easy to use. A/B Testing helps determine an answer when there are multiple possible solutions.

CONS

This method can only be applied if a project has a clear goal or KPI(s). Example: The copy in Email A generated 20x more newsletter subscriptions than the copy in Email B. This information offers no behavioral insights and requires the assistance of a web developer (to write the variant 2+ codes).

Card Sorting (Folksonomy)

We use card sorting to determine both a current site and a new site’s information architecture and navigation design. This is very useful for determining an FAQ structure, creating a site map, and determining product categorization. How it works:

2–5 participants (5 is best, actual users preferred — 3x yields best quantitative data) are given cards ( Post-It or index) representing a page on the site (no more than 40) and grouping them first into categories, then organize those categories into hierarchical branches.

PROS

Quick, easy, cheap and effective! This is a fun, easy way to get user input while organizing the, often complex, structure of the site.

CONS

Recruiting participants is really the only drawback.

Moderated Usability Test

The difference between moderated and unmoderated lies within whether or not the participant is being made to perform tasks and answer interview questions given by the test moderator. A Moderated Usability Test assigns tasks and applies follow-up questions. (Complex interfaces/interactions require a moderated approach.)

PROS

Can be conducted remotely with online tools (UserTesting.com, for example). Doesn’t require a large sample size (3–5 users is typically adequate). Online tools allow ability to easily filter participant sample demographic and asynchronously receive responses (set it & forget it), and there are video recorded screen shares that can be downloaded and referred back to throughout the design process.

CONS

At a typical $50/test (unsure of subscription pricing), costs can add up quickly if not careful. Poorly worded questions/tasks can easily derail a test, and remote testers don’t have access to moderator to remain on track.

Unmoderated Usability Test

As you may have guessed, an unmoderated usability testing approach is a freeform style of testing the participant to observe their natural behavior when using the site. A hybrid of moderated and unmoderated testing styles is often applied.

PROS

Less costly/time-consuming. Another method of analyzing metrics (i.e. time spent on task, clickstream paths, time spent on page, task-completion rate, etc) free of moderator intrusion. The same PROS as the Moderated Usability Testing can be applied here as well.

CONS

Minimal or no follow up questions/feedback/open-ended conversations. Requires additional time to ensure test quality prior to launch.

Rapid & Paper Prototyping

Ranging from paper sketches to high fidelity interactive mock-ups that look and behave exactly like the end-product, rapid prototyping is a useful method to stealthily determining UX design solutions within an iterative process.

PROS

The ability to actually show how an interaction will take place is invaluable in ensuring continuity for the final product. Paper prototyping is extremely useful when designing for mobile and determining UI placement.

CONS

Complex interfaces and interactions can be time consuming to create in some instances. Sketches can sometimes leave the user unclear/confused. Limited task flow prototypes can “trap” users who aren’t given proper instruction.

Competitive Comparative Analysis

Often part of the initial research phase of the design process, and often with the assistance of the Marketing team. This phase of research answers the questions — “Who is doing this best and how can I improve on that?” “What are our competitors doing wrong/right?” “What do their customers hate/love?” “What are the best practices?” “What are common mistakes?”

PROS

Free (when utilizing existing market research)! Solves taxonomy problems by recognizing organizational standards. Provides clarity on design principles within a particular industry. High-level inventory charts allow for side by side comparison of features and interactions.

CONS

Time consuming to create and requires in-depth research to define. However, there is really no good reason why this should not be executed should the research already exist.

Journalistic (Diary) Study

This user research method is performed by requiring participants to keep a living list or diary of thoughts they have as they use the site. Ideally, we want to know everything; the good, the bad and the ugly.

PROS

We gain direct insight into the user’s personal experience and thought-processes as they interact with the product

CONS

This insight can be highly subjective and would require multiple participants, making it difficult to recruit and monitor everyone involved.

Contextual Inquiry (Field Studies)

Contextual inquiry is a subset of the ethnographic research design utilized in true user-centered design. It is a semi-structured, one-on-one interaction with the user in their actual workplace. Which combines asking structured questions and observing the user using the product in their own environment, within context. It can be done in combination with other research techniques (i.e. individual interviews).

PROS

Fast & easy to produce. The BEST way of determining almost exactly how the user interacts with the product, free of influence. Vital tacit knowledge obtained, the best way to determine how a user uses a product — observe them using it! Extremely convenient for the user.

CONS

User’s workplace security measures can sometimes be an obstacle. Results are qualitative, not quantitative. Travel time and costs the designer incurs are a noteworthy variable.

Focus Groups

Focus groups work best when trying to determine the user’s needs either prior to interface design or long after implementation. This should not be used as the sole source of user feedback and should serve as supportive information.

PROS

We gain direct access to the users themselves. These are usually conversational and can encourage feedback not always gained through other methods.

CONS

Focus groups are expensive! Not only is it difficult to recruit participants, time and travel to a specified, neutral location is also required. At times, monetary incentives must also be used to attract participants. In addition to renting a space (for neutrality), we must hire a professional moderator at an hourly rate. When conducting focus groups, we must keep in mind that whatever the users say may differ from what they actually do.

RECOMMENDATIONS

Submitting questions to for the Marketing team to include (if they are already planning to conduct a focus group for market research) could provide UX Designers with user feedback without incurring additional costs.

Lean UX vs. Agile UX

Credit to: uxpin.com

You often hear people use buzzwords such as “lean” and “agile” when discussing UX and how it relates to businesses, specifically startups. People mistakenly use the terms interchangeably quite often, causing a general confusion in the industry about what it means, its level of importance and whether or not to adopt these practices.

LEAN UX

Lean UX integrates with Agile UX and the overall Agile Scrum framework. It emphasizes the creation and shipment of a core minimal viable product (MVP) as quickly as possible and utilizing “learning loops” (build — measure — learn). Constantly gathering data on the target user (as opposed to the entire demographic) and uses this information to fuel the design process and future iterations of the product. Some argue that even the term “Lean UX” is unnecessary as it is just simply good UX.

AGILE UX

The Agile methodology was created to go up against the classic Waterfall development process, which is heavy in documentation and leaves little to no room for development (and ultimately) design changes as new information is discovered through the process. Agile emphasizes a dynamic process as opposed to a static one, allowing for such changes to occur mid-project as user information/data is uncovered. We currently subscribe to the Agile method and as we push to include more lean UX methodologies, the two will continue to integrate seamlessly as we push to design an (even more) user-centered product.

By: Jackie Tanner, UI/UX Designer