With the onset of digital world, everyday a number of new websites is being created trying to sell wide range of products and services. And sustaining in this highly competitive market is itself an art. Companies are hiring tech geeks to customize their website and services according to the response rate of customers. A wide range of tools and services has come to existence specifically tackling this problem of the market.

Two of the most used techniques are A/B testing and Multivariate testing. Both of the techniques have their strength and drawbacks.

A/B testing, also referred to as split testing, is a method of website optimization in which the conversion rates of two versions of a page — version A and version B — are compared to one another using live traffic. The traffic is split into two parts and by tracking the response it can be decided which page is better.

A/B testing use statistical hypothesis testing to compare two or more version of the website. The good knowledge of statistical test is must to properly interpret the answer. Often the statistical significant remark can be misleading. A p-value alone does not mean the difference we are getting is significant. A sample size must be predetermined before the study to ensure that the difference is actually significant or effect size must be reported with the p-value.

There are mainly four things that need to be considered while running a statistical test:

• the effect size
• the sample size
• the alpha significance criteria
• statistical power

Effect size is basically how big the magnitude of effect is. Sample size is the how many number of visitors you are using for the test. Statistical power is the probability of seeing an effect where it does actually exist. For practical purpose often the power is predefined at 80%. Sample size is calculated using the standard deviation, margin of error and confidence level.

A/B testing is a powerful and widely used tools that can be integrated with website to check the response of customer to the new headline or to the new mickey mouse cursor for acceptance or to the new layout of website or any other change that you want to compare.  A/B testing is so speedy and easy to interpret that some large sites use it as their primary testing method, running cycles of tests one after another rather than more complex multivariate tests.

However, A/B testing has some limitations. A/B testing is best used to measure the impact of two to four variables on interactions with the page. Tests with more variables take longer to run, and in and of itself, A/B testing will not reveal any information about interaction between variables on a single page.

Multivariate testing uses the same core mechanism as A/B testing, but compares a higher number of variables, and reveals more information about how these variables interact with one another. Multivariate Testing can also be used as a follow up optimization test on the winner from an A/B test, once you have narrowed the field.

Using multivariate testing can be helpful when multiple elements on the same page can be changed in tandem to improve a single conversion goal: sign ups, clicks, form completions, or shares. If conducted properly, a multivariate test can eliminate the need to run several sequential A/B tests on the same page with the same goal. Instead, the tests are run concurrently with a greater number of variations in a shorter period of time.

But there are downsides of multivariate testing also. In A/B testing, the traffic is split in half to test the two versions of website but in multivariate testing to study the interaction of multiple variables the sample size is split in quarters, one-sixth, one-twelfth or even smaller segments. So to reach a statistical significant mark a sample size must be calculated before running a multivariate test and in case if the traffic is low then the A/B testing should be done.

Example: Flipkart wanted to decide the best format to send email to customer to market the products. They had some ideas but they wanted to find the best combination.

Ideas:

• 3x subject lines with and without some clever personalization
• 3x different salutations
• 3x different click to open product button
• 3x different layout of products

So, to do multivariate test analysis Flipkart wrote down a multivariate test design.

However, a problem arose while writing down the test design for full factorial design they required 3*3*3*3 = 81 permutations. So they would need to set up 81 emails to run this test. But 81 test cells were far too much to do an analysis to yield statistical significant result.

So they decided to do a fractional factorial design. This type of design does not include all possible combinations but the important ones. So they used small orthogonal arrays as a compression method. The design still would be pairwise complete. They ended up with 9 combinations that would be enough to estimate all 4 main effects with 3 level each.

So with these combinations they sent the email to customer and two days later they got the response. They were able to find the best combination out of those 9 combinations but they wanted to know how the other 72 combinations would have performed. So they did the response modeling. They used a data mining tool in order to train a statistical model based on the 9 cases. They obtained each of the variables in the binary form and using this binary form and response they trained a logistic regression model to predict the output of all possible combinations.

Take Away

Multivariate testing and A/B testing can be used together in market analytics to increase the sales of the products.
Interested in working on such niche domains? Come be a part of Busigence Technologies. Check out what we do