Multivariate Testing: What it is and How to Run the Best Tests

What is Multivariate Testing?

Multivariate Testing, also known as MVT is the process used by optimizers to test multiple variations of multiple elements, then measuring how those elements interact with each other. The goal is finding the highest performer and increasing conversions. When using MVT testing you can test different variations of any element on your page like headlines, images, and buttons.

What is A/B Testing?

A/B testing, which is also known as split-testing allows you to compare two different versions of a web page to determine which converts visitors better.

The two pages can be:

Slightly Different

If you have a high-converting design and you’d like to figure out how to improve a single element, you can test two different pages with only one difference. If you want to find out if a curiosity or news-focused headline performs better, you’d be able to test headline vs. headline. Also, if you wanted to discover if a video or GIF impacts conversions more, you could test one on each page.

Once you’ve driven equal traffic to each one, you’ll know the one that generates the most conversions is the winner. This way, the reason that one page generated more conversions than the other will be clear because there was only one difference between the two pages.

Drastically Different

You also have the option of testing two pages that are different in multiple ways. Your original page can have a different headline, featured image, and call-to-action button than your variation.

After driving equal traffic to both the original page and the variation page, the one that coverts better is the winner. Unlike when you change only one element per test, when you test drastically different pages there isn’t a way to determine the reason a specific page won. That’s where multivariate testing comes into play, it will allow you to determine how changes to multiple elements interact with each other.

The Difference Between A/B and Multivariate Testing

Running a multivariate test is similar to A/B split testing, but different because A/B testing only tests one variable. A minimum of one variable is tested to determine the effect of a change to one variable. In a multivariate test, multiple variables are tested together to uncover the fitting combination that is effective in improving the goal.

The best use for A/B testing is for testing two or more entirely different pages. In testing terms it’s best to A/B tests to find what’s called the “global maximum” and multivariate tests to clarify the “local maximum.”

Conducting a Multivariate Test

The software available for multivariate testing allows marketers to create easy tests in just a few hours. Testing is an important part of any conversion optimization project, but it shouldn’t be the only one. Testing should only take place after the conclusion of the other equally critical optimization stages like persona development, voice of customer research, heuristic evaluation, usability testing, site analysis, and design and copy creation. Each of these elements will create a building block towards a highly optimized website that converts visitors into customers.

To create a successful test, you should follow these steps:

• Identify a problem

Before you start attempting to improve your webpage, it’s imperative to gather data on how your visitors are interacting with it.

• Formulate a hypothesis

After conducting a thorough examination, then you should create a hypothesis. For example, if you realized a download link on a webpage wasn’t noticeable, an obvious solution is to make it the most prominent part of the page. By making the word “Download” more noticeable it should boost conversions.

• Create variations

Next, you should create variation pages for the test. A full factorial test should be conducted because it tests all combinations instead of fractional factorial, which only tests the most impactful.

Even though there a several methods of multivariate analysis including full factorial, fractional factorial, and Taguchi, most optimizers recommend running a full factorial because of its accuracy.

• Determine sample size

Prior to driving traffic to your pages, you should determine your sample size. Which is the number of visitors each page needs to generate before making conclusions about the results of your test.

You must decide how many visitors, and how long you should run tests based on website traffic, how many variations you have, and your statistical significance.

• Test your tools

You should test everything before you start running traffic. Make sure your landing page looks the same in every browser, make sure your CTA button is working, make sure all the links in your ads are correct, and so on.

Prior to running anything, it’s very important to QA every part of your campaign to make sure that nothing messes up your results.

• Start driving traffic

Once you’ve created your variations and know how much traffic you’ll need to generate to each one, it’s time to begin driving traffic to them. Note that the biggest downside to multivariate tests is the large amount of traffic you’ll need before you can conclude the tests, so you’ll have to be patient.

When conducting an A/B test, there are only two pages you’ll need to drive significant traffic to. On the other hand, in a multivariate test, there can be so many different pages that you’ll need to collect a large sample size before you can call the test.

• Analyze your results

Once you’ve completed all the steps above you can finally analyze the results of your test to determine the correctness of your hypothesis.

• Learn from your results

Every one of your tests should be used to learn something about your web pages and its visitors so that you can use it to inform future tests.

The Results of Running Multivariate Testing

Multivariate testing is very useful to help online businesses increase conversion rates, but the results from running the test may vary.

You can use different approaches to design and create a multivariate test:

• Element level testing: Testing different variations of an element on the page. The goal is to measure the impact of that element on your conversion rate. Element level testing is known as the simplest kind of testing. It requires the least amount of effort and most of the time, element level testing has minimum impact on your website conversion rates.
• Page level testing: When doing this type of testing, you can test multiple page elements simultaneously. This type of testing requires more effort from the development team to implement and it generates a higher impact on conversion rates compared to element level testing.
• Visitor flow testing: Doing this type of testing you would test several navigation paths for visitors within your website. For example, an ecommerce website might test a single step checkout vs. a multi-step checkout. Visitor flow testing can become complicated fast. It usually requires a lot more effort from a development team to implement. When done properly, this type of testing will have a greater impact on conversion rates compared to the other testing options.

Potential Setbacks of Using Multivariate Testing

You must be strategic when planning multivariate tests because you’ll end up with below par quality tests that take way too long to implement and don’t produce results or insights. Always remember that testing is only one component of conversion rate optimization work.

When creating a multivariate test be mindful of:

• Creating tests without paying close attention to the hypothesis behind it.
• The number of variables you’re testing and their dependency on one another.
• The amount of time it will take to complete the test to a statistical significance.