Skip to main content

Experimenting with multivariate testing

Tom explores the pros and cons of multivariate testing.

Multivariate testing (MVT) is a great way to monetise customer insight, but it can be an extremely expensive way to generate customer insight.

Recently we’ve enjoyed some pretty stellar successes using multivariate testing platforms (likeOmniture’s Test & Target) to tune online sales processes for increased yield. I remember, fondly, how when these tools first came to market a couple of years back their vendors confidently predicted the end of the world for the likes of Foolproof.

Now that everything in a digital sales process can be empirically tested in combination with everything else why would anyone need the anecdotal opinion of customers, creatives or so-called user experience experts to inform design decisions? Let the computer figure out what your site should look like. There are a couple of problems with MVT though:

1. When you can test everything, what should you test first?

In theory, MVT allows you to test anything and everything on your site, but practically speaking this takes a long time. When companies go to the time, effort and expense of implementing an MVT platform they expect results. Expectations of the early MVT campaigns are high.

The standard starting point in this situation is to look at site metrics from the sales process and look for the points at which customers are bailing out. OK, you’ve found your problem page, but there are a hundred thousand possible variants of copy, layout, field format, images. What components should we test first?

Oh well, every journey starts with the first step. Let’s see if making the ‘Buy Now’ button bigger (or blue, or flashing) does the trick. That didn’t crack it? Oh well, let’s test something else next time around.

Meanwhile, corporate patience starts to wear thin. Especially because;

2. The monkeys have to pay to use the typewriter

Apart from Google’s free (but rather limited)Optimizer tool, MVT costs quite a lot of money. The heart of the charging model is a tiny cost each time a page variant is served to your site. Multiply this by hundreds of variables across tens of pages over 12 months in a year and these tiny costs add up – it’s like buying media space on your own site.

If you think about this charging model for a couple of seconds, you can see that the more clueless clients are about where to start looking for efficiency gains, the better MVT providers like it. You want to spend six months (and thousands of pounds) figuring out what colour your ‘Buy Now!’ button should be? Go for your life! Like the proverbial monkeys locked in a room waiting to type the Complete Works of Shakespeare, you may be a long time finding perfection. However, if you’re paying a penny every time a monkey hits a key you’ll probably rethink your approach.

Here’s how we use MVT as part of a sales optimisation programme:
  • Use quantitative data (sales and conversion figures, site stats etc) to understand the environment you are trying to tune

  • Conduct user research to pin-point where you have pressure points or problems in your sales process – and to understand what’s causing these. (Hint: The place where the customer is having a problem isn’t always the same place where your site stats show they are dropping out of your sales pipe)

  • Make changes to the site which immediately remove big, obvious problems

  • Use your new understanding of customer needs and expectations to devise focused MVT tests in areas where the design decisions are more subtle. Often this comes from where user research showed you the nature of the problem, but didn’t categorically show you how to fix it

MVT is a terrific tool when it’s targeted deliberately on points in a sales process where you know customers are having difficulty, or need to be persuaded to commit to purchase. But if it’s used without clear objectives developed out of customer insight, well, someone may be making a monkey out of you.

Multivariate Testing - FAQs

Multivariate testing is an optimization method used to evaluate multiple combinations of design elements simultaneously. Unlike A/B testing, which compares one variation against another, multivariate testing examines how multiple changes interact with each other to influence user behavior and business outcomes. For example, a team might test different headlines, images, buttons, and calls to action at the same time. By analyzing how these variations perform in combination, researchers can identify which elements contribute most to conversions, engagement, and overall user experience. Multivariate testing provides deeper insights into optimization opportunities and user preferences.

A/B testing focuses on comparing two or more complete versions of a page, feature, or experience to determine which performs best. Multivariate testing goes a step further by evaluating multiple variables within the same experience and analyzing how those variables interact with one another. While A/B testing is generally simpler and requires less traffic, multivariate testing provides more detailed information about the effectiveness of specific design components. Organizations often use A/B testing for broader comparisons and multivariate testing when they want to optimize individual elements and understand complex relationships between design decisions.

Organizations should consider multivariate testing when they want to optimize high-traffic digital experiences and better understand how different design elements influence user behavior. It is particularly useful for websites, landing pages, checkout processes, signup flows, and other conversion-focused experiences. Because multivariate testing requires sufficient traffic volumes to achieve statistically meaningful results, it is often best suited for organizations with large audiences. When implemented effectively, it helps businesses make evidence-based decisions and refine experiences based on actual user behavior rather than assumptions.

Multivariate testing helps UX teams identify which design elements have the greatest impact on user engagement, task completion, and conversion rates. By examining interactions between variables, teams gain a more detailed understanding of what drives successful experiences. This approach reduces guesswork and supports data-driven optimization efforts. It can reveal unexpected relationships between content, layout, visuals, and calls to action, leading to more effective design decisions. Ultimately, multivariate testing helps organizations improve customer experiences while achieving measurable business outcomes.

Although multivariate testing offers valuable insights, it can be more complex than other testing methods. Testing multiple variables simultaneously requires careful planning, sufficient website traffic, robust analytics capabilities, and statistical expertise. Without adequate sample sizes, results may be unreliable or difficult to interpret. Additionally, managing numerous combinations can increase implementation complexity and analysis effort. Organizations must ensure that testing objectives are clearly defined and that experiments are designed to generate meaningful conclusions. Proper planning is essential for maximizing the value of multivariate testing initiatives.

Multivariate testing provides organizations with evidence about how users respond to different design choices. Rather than relying on personal opinions or assumptions, teams can evaluate performance using measurable data and statistically validated outcomes. This evidence-based approach improves confidence in decision-making and helps prioritize changes that generate the greatest impact. By continuously testing and refining digital experiences, organizations can enhance usability, increase conversions, improve customer satisfaction, and create more effective products and services. Multivariate testing therefore plays an important role in ongoing optimization and digital experience improvement.

Related articles