How to use quantitative data in design research

An exceptional Design Researcher recently announced to me ‘I don’t know anything about quantitative data’.

She actually knows more than she thinks. She’s just fallen victim to what I refer to as the mystification of quantitative research. And that is acting as a blocker to using quantitative approaches to beneficial effect in design research.

A quick Google search for ‘quantitative research’ will yield terms such as; statistical models, empirical investigation of observable phenomena, correlation, multiple linear regression, causality, dependent and independent variables, computational techniques and prioritisation frameworks.

Those terms are enough to send many a researcher or designer running for the nearest testing lab, to the familiar territory of qualitative testing. But in doing so, you’re missing a trick.

Quantitative data is your friend, just a misunderstood one.

I’m a qualitative researcher, I don’t know how to get quantitative data

Yes, you do.

You just told me that three of your eight test participants closed the log-in pop-up and that six of them said they’d never use the ‘search’ functionality. That’s quantitative data.

You could also tell me that there are four white cars parked outside right now, that each one of those cars can comfortably carry four passengers, and that it’d take them 35 minutes to get to my house from the office. Yet more quantitative data…

Quantitative data = numbers and counting

And the bigger those numbers, the more you can make them do for you.

What can bigger numbers do?

As Design Researchers, we regularly encounter the question ‘is that enough people?’ when we do research with 8-16 people.

From the point of view of testing the usability and user experience of an offering, the often-cited but contentious Nielsen explanation tells us why eight people can capture the majority of user issues and challenges with a system. These are often the issues which need the most design attention.

But testing with eight people isn’t going to be enough to tell us, with confidence, about what might be the top priority to invest time in to potentially boost satisfaction among the wider customer base. That’s where the bigger numbers of quantitative research come in.

How big are we talking?

The short answer to this … the bigger the better, within reason.

There are three key things to consider when choosing how big the number should be, and how many people you should talk to (your sample size): 

Margin of error

A Google search for ‘margin of error’ will yield results which include mathematical equations and curve diagrams.

In plain English, these searches tell you that the more people you talk to, the smaller the margin of error is. Meaning the more confident you can be that the results of your research are replicable you’re targeting (assuming you’ve taken a representative sample of that population). An example:

• If you talk to 1,000 people, the margin of error at 95% confidence interval is +/-3%. This means that you can be 95% sure that if 50% of your sample prefer the colour black, then in the total target population, between 47%-53% prefer the colour black

• If you just talk to 200 people, the margin of error is +/-7%, in the total target population, the proportion preferring the colour black is somewhere between 43%-57%

The smaller the percentage range in which our findings can be placed, the better. And the bigger the sample size, the smaller that percentage range will be.

Is there a connection between this then and the number of subgroups you mentioned?

Absolutely.

If you want to examine each subgroup of customers in your population separately, then the margin of error rules still apply.

If you have 3 particular types of customer, and you want to have findings for each of those with a +/-3% margin of error, then you’re going to need to talk to 3000 people in total. You could do less – 500 each, for example, would come with a +/-4.4% margin of error – it depends on...

…budget?

While in an ideal world we’d like to talk to the maximum number of people available and have the lowest margin of error possible on both a total and subgroup level, we have to be practical as well and work within the budget available for the research to inform the design.

That said, it’s not as expensive as you might think. Remember, this isn’t recruitment for qualitative research that we’re talking about. Quantitative research, especially using online data collection methods, is much cheaper.

Yes, it becomes expensive when you’re looking for a ‘needle in a haystack’, i.e. a niche audience, but less niche audiences can be contacted relatively cheaply using online panel sources.

What can I use quantitative research for in the design research process and practice?

The short answer, anything you want to attach a number to.

The long answer, here are some examples:

• Examining the wider customer base to assess which elements of your digital offering they rate less highly, either as a whole or as customer subgroups, and identify areas most in need of development or design

• A/B test designs or content, either within market or across markets, to confirm qualitative assessment

• Conduct key driver analysis to determine what elements are most likely to drive customers’ satisfaction with and affinity to the brand.

• Validate and quantify pain-points and challenges which users experience during the process of using a service, adding quantitative support to the experience-mapping process

• Using an approach such as Kano analysis, identify basic, performance and excitement needs for a service

• Test prototypes with larger sample sizes to add weight to qualitative findings

These are just some examples based on work we’ve done. There are many other scenarios where quantitative research can help.

These are also just examples of where we’ve spoken to users or potential users via survey methods. There are other forms of quantitative data which can be used either in conjunction with, or in addition to survey data, such as analytics, sales data, etc.

I want to do quantitative design research - what’s my next step?

There are just three questions you need to answer:

1. Who do I want to talk to?

2. How much budget can I spend? (We can tell you how many people you’ll be able to talk to for your budget)

3. What do I want to talk to them about?

Once you have answers to those questions, let’s talk quant.