But how many users did you test that with?

When discussing and delivering research findings it can be tricky to successfully convey the value that can be achieved from a small sample size.

But, being able to explain that value goes a long way in establishing credibility for user research.

Is this scenario familiar to you?

You’re presenting the findings of customer research on the latest version of your designs. Some areas of the design need improving, but some performed really well and you are confident that they should be implemented in the next iteration of the product.

Someone interrupts: “I’m sorry… how many people did you test that with?”

The answer is often followed-up by: “Only 5 people? Is that enough?”

And then finally: “Should we really be making these changes to the product just based on the opinion of these 5 people?”

Maybe this is your first-time presenting research findings to your team. You feel pretty sure that your results are valid, but you aren’t sure of how to respond. Or maybe this is your hundredth-time presenting research findings, but you keep coming across this challenge.

Whether you are a beginner or an experienced design researcher you’re likely to come across this common question at some point in your career, if you haven’t already. We’re trying to provide a foundation of clarity and certainty, failing to do so may undermine our whole creative effort, resulting in a waste of time, resource, and money. I’ve had to answer this question hundreds of times, and want to share some tips on how to handle this question/situation when it happens.

Let’s spend a minute exploring the context of this question, and why it’s being asked, to understand what’s going on and how best to respond.

Think back to the last time that you were in a room where this happened:

• When did it happen?

• Why do you think they were asking you?

In my experience, you will typically notice that this challenge comes from someone:

• Quite senior in the organisation

• Disconnected from the day to day activities of your programme

• From outside the design team or from another department

Why are they asking?

Often, if you take a closer look at who’s asking, you will see that there are three main characteristics that all of your sample-size doubters are likely to have in common. 

1. They are busy people: You might have already told them what you’re doing and why, but they are likely managing multiple workstreams and have probably forgotten what’s happened before and what’s coming next in your specific programme.

2. They are ‘numbers’ people: They are strategic thinkers, more used to working in the initial strategy phase and late launch phases of product development that rely on quantitative data and are focused on establishing risk, reward and levels of confidence in an idea or solution.

3. They aren’t design folk: they don’t know much about the tools and activities of the design phases of a programme because they are not UX experts.

So how do you persuade them that there is value in what you are doing?

Remember, it's our duty as UX designers and design researchers to help other stakeholders around the business understand what we do and the value that it generates. So, take a deep breath and recognise that this question is actually presenting you with a brilliant opportunity to grow the knowledge of UX design tools and techniques in your organisation and can help to pave the way towards greater UX maturity.

Here are some tips on how to prepare for this type of meeting and respond to the question when it comes up: 

• First of all, be confident that there is a value in conducting research with ‘just’ 5 customers. Conducting some research to understand your audience and their reactions to your ideas is better than conducting none. And 5 may just be the perfect number - depending on the complexity and riskiness of your project and what phase of design you are in. If you are usability testing during detailed design, then you can probably safely rely on Nielsen’s often quoted advice that five participants will discover over 80% of the critical problems with a system.

• Make sure you are using the right sample size for your research: it sounds obvious, but it’s the key to being able to defend your findings! Choosing the correct sample size is quite a tricky topic deserving of its own blog post, and I can’t do it full justice here. It does depend on where you are in your programme (discovery versus evaluation) and how much confidence you need. As a quick rule of thumb, go for a larger sample (over 10 users) when you are working on something complex or risky, or are looking at an uncommon behaviour or characteristic and need greater certainty about your findings. You can go with a smaller sample size (3-10 users) when you are working on something simple or low-risk and looking at a common behaviour or characteristic and just need some detail rather than certainty.

• Before presenting findings, think about the audience and identify who might come up with questions or challenges on the sample size. Remember, they are likely to be senior folk or those that are not involved in the day to day activities of the project. Knowing who is coming and what objections they might have will help you feel prepared and to be more confident.

• At the start of your presentation give a quick recap of the project history: remind stakeholders of the research that’s happened so far and the findings that led you to your current design. Remind them of what research is going to happen in the future too. This helps reinforce that, although today you are reporting what the latest 5 users said, there is a bigger body of work and research to support the direction you are going in, and there will be other opportunities to validate the thinking later on.

• State clearly what your research is supposed to achieve and what it didn’t address: was your intention to gain certainty about the problem, or was it to get depth and detail about a few real people to inspire creative thinking? Were you looking to get some feedback to help you iterate the design, or were you looking to prove that it has resolved the business problem it was designed to? This will help to reinforce why using a small sample size was right in your particular context.

• Be prepared to point to other research studies you have done, data from other departments, or published academic research that can help corroborate your findings. This can help reassure your stakeholder that although your specific study is small, there is further evidence that points to a similar conclusion.

• When challenged, help your stakeholder understand how qualitative insight is particularly useful to other members of the project, like interaction designers, visual designers, front-end developers and copywriters. Show them how the detail and texture from qualitative research gives them valuable steers on detailed design decisions. Get others to support you by explaining how they can use the findings to progress their thinking.

Now you should be better equipped to face your stakeholders confidently, but tactfully, when discussing and delivering research findings. Remember, not everyone understands the value of qualitative research findings and it can be tricky to convey results successfully, but answering challenges head-on will help you to communicate the value of your work and educate others about UX practices across your business.