Multivariate Testing – Beyond the A/B Test

Are you looking to experiment with different versions of your website or app? You may have heard of A/B testing, or split testing, to carry out these experiments. A/B testing delivers two different versions of your website or app to your audience to determine which is the better version, with one half of your audience receiving version A and the other half receiving version B. But let’s break away from this model and start looking at multivariate testing.

Leading experimentation platform provider Optimizely, define multivariate testing as “a technique for testing a hypothesis in which multiple variables are modified”. Think of this as A/B/n testing. Where A and B stand for version A and version B in your standard A/B test, the n stands for the number of these versions being tested.

The goal of multivariate testing

The outcome of multivariate testing is determining which is the ideal combination of all the possible combinations tested.

An example of multivariate testing can be seen below.

Multivariate Testing ExampleIn the example above you can see the experiment is testing different combinations of the headline and image to determine which is more successful. Having more variations allows you to see how they interact with each other, giving you more information on what factors are positive or negative to your visitors and ultimate conversion goal. In fact by testing 5 or more variations, you can improve your win rate by 50-60%.


This advanced way of testing means you don’t need to carry out multiple A/B tests on the same page to reach your desired conversion goal. You can instead run a number of variations simultaneously in a reduced time frame.


As you can imagine, the number of variations that you can test can multiply quickly! This results in more visitor traffic being needed to achieve results of statistical significance as a lower amount of traffic is being split and sent to each variation.

So which test should you choose?

When trying to choose if you should run a multivariate test, decide if the variations being tested are going to have a quantitative effect on the conversion goal. If you think it’s possible only one factor may contribute to the conversion goal, opt for running this as an A/B test. If you think it’s possible multiple factors may contribute to the conversion goal, and you have the necessary visitor traffic, opt for a multivariate test.

Interested in experimenting on your website?

Interested in experimenting on your website?

Find the winning version of your website. Test landing pages, call-to-action buttons, images, forms, content and more for ultimate conversion optimisation.

Find out how

Read some more interesting stuff.