Research and testing are great things — they give design teams the ability to inform their designs with reality. Nowadays, research and testing are pretty much a requirement for web and mobile projects. There are plenty of methods for conducting UX testing, but many of these methods are resource-intensive and time-consuming, and this often stops teams from testing in the first place.
Luckily, an easy technique for refining the user experience exists. It helps product team validate (and invalidate) critical assumptions at cheap cost and with rapid speed. It’s called guerrilla testing.
In this article, I’ll show you how to conduct guerrilla usability testing to get the most out of it. You’ll learn how to avoid or minimize the technique’s weaknesses, and improve planning for all research and testing. But before we dive into details, let’s first define what guerrilla testing is all about.
What is guerrilla testing?
Perhaps the best definition of guerrilla testing was coined by Martin Belam. He defined guerilla testing as “the art of pouncing on lone people in cafes and public spaces, and quickly filming them whilst they use a website for a couple of minutes.”
Basically, guerrilla testing means going into a coffee shop or another public place to ask people about your prototype. It’s low cost and relatively simple testing that enables real user feedback.
This type of testing has following characteristics:
- Participants are not recruited but are approached by persons conducting testing sessions.
- The sessions themselves are short (typically 10–15 minutes) and are structured around particular key research objectives.
- The output is typically qualitative rather than quantitative. Testing helps to quickly validate how efficient design is on its intended audience or whether specific functionality works in the way it is supposed to.
Each testing session can be represented as a following number of steps:
- Approach a person
- Introduce yourself and ask if they would like to participate in product testing
- If they agree, get basic information about them
- Give them a few scenarios to do
- Observe their interaction
- Ask about their experience
- Thank and reward them for participation
The beauty of guerilla testing:
- Guerrilla testing is a fast method that provides sufficient enough insights to make informed strategic design decisions. Guerrilla research can be squeezed into nearly every timetable or deadline.
- Since it doesn’t require a lot of money, most product teams can afford to do guerrilla testing on a regular basis.
- Doesn’t require specific research skills. Anyone on the product team can conduct guerrilla testing.
- Can be used as a demonstration the value of user testing/research for stakeholders, especially for those who struggle to acknowledge the value of usability testing.
Guerrilla testing is great for:
- Identifying critical usability issues early in a product design lifecycle.
- Testing hypotheses/assumptions during design sprints. Guerrilla testing can be an easy way to validate those hypotheses and create validation checkpoints.
- Validating tasks that don’t require specific knowledge (e.g., completing a signup form, ordering a product in e-commerce store).
- Getting quick baseline measures of an existing product experience (your key competitors).
Guerrilla testing isn’t so great when:
- Domain-specific knowledge is required to use a product (e.g., completing specific use-cases in financial or medical apps). You can’t expect all the people you talk to will have all required skills.
- A specific environment is required to conduct testing (e.g., testing can be done only in a certain location).
Now when you know what guerrilla testing is, let’s walk through how to conduct it.
Step 1: Prepare for testing
Many UX professionals consider guerrilla testing as bad practice. They say it often doesn’t reflect the real picture. This opinion comes from a flawed approach to testing.
The danger of guerrilla testing comes from poorly planned and executed test sessions. Such test sessions don’t provide any reliable insights. While guerrilla testing is a definitely less formal way of testing (in comparison with lab usability testing), that doesn’t mean that it can be successfully done in a planned fashion.
Know your objective
The number one rule of research/testing is that before collecting data, you must know why you’re collecting data. No matter what testing technique you employ, it’s always worth being certain of your objectives before conducting testing sessions.
If you don’t know what you expect to get from your research, you won’t gain sufficient insights after the test session. Having an objective doesn’t mean having a big-detailed plan (which would probably be overkill for this flexible, “on the fly” technique) but it does mean having a clear definition of what you’re looking for when you start.
For example, if you’re testing an app for ordering food you might want to know answers to questions like:
- Can users easily search a particular type of food they would like to order?
- Can users submit the order without too much effort?
Have an interactive prototype
Some UX experts suggest guerrilla testing can be done with almost anything, including concepts drawn on paper. As a result, many researchers print out designs and ask test participant to test paper sheets. This isn’t right way to do guerrilla testing because that’s not how someone experiences a product in real life. You can’t expect people to understand how your product works by flipping through paper pages.
While you don’t need to have a finished product to conduct guerrilla testing, you still need a prototype testers can experience. The more it resembles a real product, the more valuable feedback you’ll receive.
Tip: If you’re at the early stage of your design process and don’t have a prototype yet (even a semi-functioning one), you can test your competitor’s solution. This way you’ll be able to understand how real users interact with a product from the category and what areas of UI/UX you should focus your attention during development.
A prototype of mobile app created in Adobe XD
Pick the right location
A lot of UX professionals think guerrilla testing can be conducted in the nearest public place (cafe, sports venue, shop, etc). But that’s not always true. You need to pick a location where your target audience spends their time. For instance, if you’re testing a new mobile app for a retail chain, you might go to one of the stores and test there.
Tips:
- Ensure the environment is relaxed. You won’t be able to conduct a proper guerrilla testing if all your test participants are rush or stressed.
- Always ensure staff at the venue are OK with you doing some user testing.
- If your prototype requires internet connection, make sure to pick a location with a stable Wi-Fi.
Image credit: johnferrigan
Create smart scenarios
The tasks you select for your testing session play a critical role in whether findings will be useful or not. Since it’s impossible to test everything at once (not in regular usability testing, and not in guerrilla testing), you need to select carefully.
Tips:
- Think about all the important things people need to be able to do using your product and write down a short list of tasks. For example, if your product is a mobile app to order food, you probably want to test how people find a product, compare products, order a product, add a product to a wishlist, etc. Write these tasks down.
- Now that you have a list of tasks, it’s time to prioritize them and decide what to test. Give each task points from 1 to 3 based on how frequently the tasks are performed. 3 points are for tasks a majority of users will do most of the time, 2 points if they do it occasionally, 1 point if they only perform this task rarely (like complaining about an order).
- Choose the top 3 tasks. You’ll use them to create scenarios users can understand and grasp easily.
- Create a scenario based on each task. A good scenario has following characteristics: it describes a problem for participants to solve; it’s easy for them to relate to (by providing context and a realistic level of detail); and it doesn’t hint to the participant how to achieve the goal. An example of a scenario:
“Imagine that you’re looking for ordering a pizza for your office party. You found this app and you want to try it. Go ahead and give it a try.”
- Before testing your scenarios with test participants, pre-test them with friends and colleagues. Make sure that scenarios are easy to understand and people can follow them without any confusion. Not doing a proper run through of the test in advance is a common guerrilla testing mistake.
Take someone else with you
While you definitely can conduct guerilla testing alone, it is worth taking someone with you for two reasons:
- You won’t be worried about your stuff — your colleague can look after your things while you approach potential subjects.
- It will be easier to discuss the results of test sessions.
At the same time, it’s not recommended to have more than two people in research group because it will make test participant feel uncomfortable. Two is the perfect number.
Conduct five test sessions
Jakob Nielsen did an extensive research and found that testing with five users will help you find up to 85% of the core usability problems in your product. You learn a lot from the first person you talk to, a little less from the next, and so forth. After the fifth user, you’ll observe the same findings repeatedly, but won’t necessarily learn anything new.
You only need to talk to five users to find 85% of the core usability problems.
Be ready to adapt to context
The lack of a controlled environment is one of the most significant differences between guerilla testing and regular user testing. Guerrilla testing is about adapting to the situation. Even when you carefully picked out time, location, there’s always a possibility that things may not go according to plan. Consider all potential risks and have a plan of action for it.
Tips:
- Have a backup internet connection in case the Wi-Fi connection isn’t fast enough. Also, don’t forget your charging cables or additional power packs.
- Be ready to trim the number of questions if a person decides to leave a testing session earlier.
- If the environment around you is too loud, focus on tasks that require less verbal clarification.
Step 2: Approaching people
Proper introduction is essential
Think in advance about how you are going to introduce yourself. You’ll only have a couple of minutes to get across what you’re there to do and what you want from the participant, so you better get your intro right!
For my initial approach I use a 5-step formula:
- Question: Do they have a time? (1)
- Describe: Who are you and why are you there? (2)
- Describe: What do you want from them? (3)
- Questions: Can they stop for 10 minutes? (4)
- Describe: How you reward them. (5)
Hello, do you have a minute? (1) My name is Amy and I work for company ‘Awesome.’ (2) I’m here asking people to take a look at our product (3) and let me know what they think of it. If you have 10 minutes for me (4), I’ll ask you a few questions and record the answers. In return, I’ll buy you a coffee or a muffin to say thank you. (5)
Before starting a test session, it’s important to learn some basic facts about your test participant. Spend a minute or two to find out if they are the type of person who may use your product. If you have specific criteria for people who you want to talk to, a few simple questions will help clarify. For example, if you’re testing mobile app for ordering food, you can focus on finding the following information:
- Do they order food online?
- What is their level of technology proficiency (whether it’s a tech-savvy or regular user)?
Step 3: Running the testing session
Explain the purpose of testing, and mention you’re testing the app, not them
Before you start testing, make sure your participants have a good idea of what you’re doing and why you’re doing it. Since most people don’t know what guerrilla testing is all about, provide some context. This helps develop trust between the researcher and test participant, and leads to more honest responses.
Another important moment that should be taken into account is that people in testing situations often can feel as though they are being tested (as opposed to the product itself), and sometimes start to apologize or shut down. To prevent that, say something like:
“I’m testing the product, not you. No need to worry about making any mistakes. And please don’t worry about our feelings. We want to improve our product and need to hear your honest reactions.”
Follow ”Think Aloud Protocol”
The think aloud method is critical for getting inside the user’s head. It means asking the user to speak out loud everything they are thinking, so you can gain insight into the thought process behind the user’s actions.
As the participant uses the product, you should encourage them to think out loud and share their thoughts and ideas with you. Since talking while doing isn’t typical for people, you should ask them to do it:
“While you’re using the product, I would like you to think out loud. Just say what you’re thinking, what you’re trying to accomplish, what you expect to happen after an interaction, and so on.”
For example, if you’re testing a mobile app for ordering food, you should expect people to say things like, “Hm, this looks like an app for ordering food. I wonder how to find a product category. Maybe if I tap here, I’ll see it.” Give them an example like that one to help them understand what you’re looking for.
Tips:
- Don’t hesitate to ask test participants questions like, “What are you currently thinking?” “What do you think will happen next?” or “Is that what you expected to happen?” during the testing session to stimulate them to verbalize their thoughts and feelings.
- Even when people are thinking aloud, sometimes they experience problems with verbalizing their thoughts. That’s why you should ask clarifying questions if something seems unclear or you think there’s more information a test participant can share.
Keep testing consistent
A common problem with guerrilla testing is the temptation to go with the flow when querying the user. Often a particular question can trigger interesting insights, and a person who conducts the testing wants to get the bottom of it. This can cause the test to go in an entirely different direction and, as a result, it’ll be impossible to compare results.
It’s always better to focus on your primary questions while conducting sessions. If you found something unexpected during a test session, note it and circle back to a particular topic to dig deeper at the end of the session.
Encourage creativity in user feedback
Making testing consistent doesn’t mean you can’t foster creativity. Help test participants get creative by asking them to sketch their ideas or feelings. For example, if they think there’s something wrong with UI, you can ask them to draw a simple schematic solution for the problem. These ideas might help you to improve things in the next iteration.
Speak less, observe more
As soon as you give test participant your scenario, you should stop talking, lean back and watch how they use your prototype. Don’t provide any extra explanation. Why? Because you want the testing session to be as real as possible, and you want to see how test participants figure out things for themselves — and the best way to see the natural behavior is to remain silent during testing sessions.
Just give test participants your scenario, lean back and observe how they use your website, app or prototype.
Your participants will undoubtedly have questions, and since you’re sitting next to them, they will ask you. To avoid any misunderstanding or bad feelings whenever they ask you something, just say something like this:
“That’s a good question, but I can’t tell you the answer right now because we’re interested in natural user behavior — how people use the product without any hints. I’ll give you the answer after the testing session, but for now I would like to hear what would you do if you were using this product on your own.”
Be a timecop
Remember, guerrilla testing isn’t a usability lab with paid users. You’re asking people to take a break from their busy lives and spend a couple of minutes with you and your research. Be mindful of how much time you spend with test subjects. If you ask someone for 10 minutes, make it 10 minutes—not 30. If you don’t respect their time, they’re not going to very happy, and that will impact their feedback.
Minimize note-taking
During the test, make sure you don’t continuously keep taking notes. When you’re sitting next to test participant and spend most of your time writing, it’ll definitely make test participant stressed and wonder what’s going on. There’s absolutely no need to write down everything you find — write down only the most critical things and observations, your key findings.
Tip: Record the session. Video or audio recording a session can be a good way to collect all important information. Obviously, recording should only be done with the interviewee’s permission. Be prepared to abandon recording if your interviewee is uncomfortable or reluctant. In the context of guerrilla testing, you can utilize specific software for this purpose: apps like Silverback or UX Recorder collect screen activity along with a test subject’s facial reaction.
Image credits: Sean Melchionda
Step 4: After testing
Conducting the tests is only half the journey to perfect UX. The feedback from testing sessions should be converted into improvements or (and) requirements for your product.
Analyse task completion ratio
Completion ratio is perhaps the most obvious finding from guerilla testing. You should capture the task completion for each of your participants and each of the critical tasks you identified earlier. The number of users who face problems on certain screens (calculated as the drop-off rate) will help you decide which parts of your app should be reworked.
Fix the biggest problems first
After guerilla testing, you’ll probably have a list of usability issues faced by test participants. How to know which problem to start first? You should focus on fixing the most important problems that affect a majority of your users.
Tips:
- Don’t try to fix the problem in code — instead, create a prototype. Fixing problems can be expensive, especially if the issues affect an already-released app. To prevent any potential reworks, it’s better to create a prototype and test it again to be sure the fix works for your users.
- Good is better than perfect. Find easy-to-implement solutions. Remember that time is money — don’t seek a perfect solution. When fixing usability issues, it’s essential to find solutions that can be implemented quickly.
- Fix only the problems you find. When working on a fix, it’s tempting to make changes that go beyond the problems you actually observed. Avoid that temptation. Remember that you have a lot of work to do, and it’s better to move according to the plan.
Combine guerrilla research findings with other research data
It’s important to understand that guerrilla testing can’t be a replacement for other types of UX testing. Thus, you can’t focus entirely on guerrilla testing to understand how people perceive/interact with your product. Nearly all projects would benefit from multiple research methods such as:
- User interviews. User interviews can deliver a lot of insights about a target audience, such as their behavior, expectations, etc.
- Moderated in-person usability testing. Moderated testing is a great way to understand how users complete certain user flows. In most cases, it’s possible to record interaction on a video and use it as a reference in a future.
- Remote usability testing. Remote testing removes many of the challenges related to session setup. This inexpensive way of testing allows you to quickly verify a design with an unbiased audience.
- A/B testing. A/B testing, also known as a split test, is a perfect type of testing when you want to compare two versions of the design and select the one that performs better.
When you combine results of guerrilla testing with other research techniques, you can get valuable results more quickly. To achieve really great results, make testing a habit.
How different companies conduct guerrilla testing
Here are a few case studies for your inspiration:
Guerilla Usability Test: Yelp
Improving Yelp’s web experiencemedium.com
Wrapping up
Whether you’re exploring a particular research problem or testing a design solution that you already have, guerrilla testing can be a huge time-saver. Now when you know a lot more about this great testing tool, it’s time to try it in the real world!