Multivariate Testing – Beyond the A/B Test

Are you looking to experiment with different versions of your website or app? You may have heard of A/B testing, or split testing, to carry out these experiments. A/B testing delivers two different versions of your website or app to your audience to determine which is the better version, with one half of your audience receiving version A and the other half receiving version B. But let’s break away from this model and start looking at multivariate testing.

Leading experimentation platform provider Optimizely, define multivariate testing as “a technique for testing a hypothesis in which multiple variables are modified”. Think of this as A/B/n testing. Where A and B stand for version A and version B in your standard A/B test, the n stands for the number of these versions being tested.

The goal of multivariate testing

The outcome of multivariate testing is determining which is the ideal combination of all the possible combinations tested.

A visual example of multivariate testing can be seen below.

Multivariate Testing ExampleIn the example above you can see the experiment is testing different combinations of the headline and image to determine which is more successful. Having more variations allows you to see how they interact with each other, giving you more information on what factors are positive or negative to your visitors and ultimate conversion goal. In fact by testing 5 or more variations, you can improve your win rate by 50-60%.


This advanced way of testing means you don’t need to carry out multiple A/B tests on the same page to reach your desired conversion goal. You can instead run a number of variations simultaneously in a reduced time frame.


As you can imagine, the number of variations that you can test can multiply quickly! This results in more visitor traffic being needed to achieve results of statistical significance as a lower amount of traffic is being split and sent to each variation.

What to test to improve conversions

Experimenting with more variables allows you to see how they interact with each other, giving you more information on what factors will help you reach your conversion goal.Experimentation Ideas Tag Cloud

So which test should you choose?

When trying to choose if you should run a multivariate test, decide if the variations being tested are going to have a quantitative effect on the conversion goal. If you think it’s possible only one factor may contribute to the conversion goal, opt for running this as an A/B test. If you think it’s possible multiple factors may contribute to the conversion goal, and you have the necessary visitor traffic, opt for a multivariate test.

What more testing ideas?

Test beyond button colours and headlines. Experiment with any feature whether it be different pricing models, personalised content or even speed performance impacts.

Best in class CRO software SDK’s allows us to test across any device and platform in multiple development languages.

Testing price variations

Try different pricing strategies for improved conversions.

Price Testing Version 1

Personalised landing pages

Deliver different landing pages based on browsing history.

Personalised Landing Pages

Deploy mobile app features in controlled tests

Stage the release of any new feature by delivering it through an experiment to a small portion of users first. This is especially powerful with mobile app features where we can roll out your features in experiments without an app store release.

Mobile Experiment

New feature delivered to 30% of users vs. current app delivered to 70% of users.

Interested in a CRO program?

Interested in a CRO program?

Learn how a CRO Agency can create winning versions of your website.

Find out how

Read some more interesting stuff.