Over the past decade, data has taken over. While inspiration, creativity, and intuitive design sense are still critical in getting a customer’s attention, most companies rely heavily on data collected through testing to best gauge how consumers are impacted by a site or campaign and what changes would yield better results. Today, the number of vendors available that perform these tests combined with the huge volume of data they produce can be overwhelming. It can become very easy to lose sight of your larger marketing goals.

To help sort through all these options and keep your marketing goals in sight, it is important to remember that, at its core, A/B and multivariate testing is all about asking questions – especially at the beginning. If you are not asking the right initial questions, you may suffer from data-overload, not actually address the specific issues you were hoping to address, and miss out on optimization opportunities. One of the most important questions to initially answer is: “What tool should I use to set up and run testing on my site?” As luck would have it, ROI·DNA has built a testing vendor comparison tool which walks you through some of these initial questions to help select the best tool for your testing program.

To start the process of selecting a testing vendor, let’s make sure we’ve answered one of the most basic, often overlooked questions: “What are A/B and multivariate testing and what is the difference between the two?”  In short, A/B and multivariate testing both allow you to test two or more versions of a webpage or creative (like a display ad for example) to determine which version better produces the desired consumer behavior (usually in the form of some sort of conversion).

Specifically, A/B testing allows you to simultaneously serve site visitors two versions of your site or creative: version A, often the current version, and version B, the new and potentially improved version. The purpose of the test is to see which one performs better.  A/B testing is best for testing either a drastic change or testing one element.

If you plan to test a combination of changes – that is, separate the variations between the items you want to test into distinct components – multivariate testing is the better option.  For example, if your page contains multiple items that you want to test different versions of (i.e. a call to action, headline, and an image), you would create different versions of each and test each of those combinations simultaneously. In the example above, if you had 2 versions for each of those items (2 calls to action, 2 headlines, 2 images) you would need to test eight (or 2³ ) combinations.

To answer the big question of which A/B or multivariate testing tool will work best for you, there are a handful of additional questions you’ll want to answer. Below is our A/B & Multivariate testing tool finder which is designed to help you ask and answer the essential questions that will guide you to the best choice.

How much are you willing to pay for the tool?

There are options ranging from free tools to pricey, enterprise tools with deep features. Determine your budget and what makes the most sense based on your project.

Which testing type are you going to use?

You’ll choose A/B testing if you want to test two (or more) different versions of your website or landing page to analyze which version gets maximum conversion rate or sales. You’ll want to use multivariate testing if you are testing different versions of elements on your website or landing page (such as headlines, images, buttons, etc.) to analyze which produces the most successful outcome.

It’s also important to note that some tools, like Google Content Experiments, have a maximum number of pages that can be A/B tested, while others allow for an unlimited number of pages. In addition, the number of active tests can be significant for websites with a lot of traffic that could benefit greatly from regular testing on different parts of the site; therefore, requiring several active tests.

Do you want to manipulate the traffic you’re testing?

Traffic throttling is the practice of controlling the percentage of traffic that is directed to each variation – this allows you to test some variations more heavily than others; for example: variation A gets 10% of traffic and variation B gets 25%. There’s also the option to run on a percentage of total traffic, which enables users to limit testing to only a subset of their visitors, avoiding running the test on everyone.

Which page editor do you want to use?

A WYSIWYG page editor provides a simple interface for creating variations, for which knowledge of HTML is not required. HTML and/or Javascript editors are a more technical interface for creating the variations. ‘Ability to edit variations during test’ in our tool finder is referring to tools that allow you to edit the variations while the test is still running, before the experiment’s end date/time.

How do you want to see your results?

A/B and multivariate testing results can be viewed as quickly as real-time or take as long as two weeks. Some analytic options include: heatmaps and clickmaps, which allow you to see where visitors are clicking on a webpage; revenue tracking & goals, which allow you to track various revenue metrics; and segmentation of results, which analyzes separately for different segments.

What display options are required for your testing?

A Mobile API enables testing within mobile apps. Testing tools that feature support for dynamic websites allow you to create variations of dynamic content, including lightboxes or mouseover pop-ups.

What is your method of testing?

Based on how you distribute traffic to your combinations, there are several types of multivariate tests. Full factorial testing is a method where website traffic is distributed equally among all combinations. Taguchi testing reduces the number of combinations that are tested in order to receive feedback more quickly; however, this method can be risky since it is not inclusive of every combination. HP IDOL (Intelligent Data Operating Layer) is a database that allows for disparate types of data.  It’s effective for testing, especially multivariate tests with large amounts of data, because it allows the user to run queries that asks the database to match certain patterns.

Campaign preview allows you to see previews of your different variations before you start the test. In addition, some tools can integrate third party data including customer databases and surveys.

How do you want to serve content to your users?

Targeting or pre-test segmentation enables you to target content to different groups of visitors based on marketer-defined segments. Predictive learning  enables you to store each visitor and click individually, so targeted offers can be served based on a visitor’s unique interests. Lastly, content rotation allows you to pass data into the platform through tags and further enhance the visitor profile stored by the platform, so you can target variations of content to the visitor.

Once you’ve answered these questions, you’ll be able to find an A/B or multivariate testing tool that is best suited to answer your specific marketing questions. If you just want a complete overview of all the available A/B and multivariate testing tools, check out our tool comparison matrix.

Still have questions on how to find the right A/B testing tool for your needs? Reach out to us – we’re happy to help.