The Importance of A/B Testing

A/B Testing in Product Design

A/B Testing, also known as split testing, is when you compare two versions of a design to see which one performs better. In essence, it is an experiment where you send different versions to different users to see which version meets the expected goal. A/B testing is also popular in marketing but we are going to focus the application in product design.

Google and Facebook have popularized A/B testing with Google taking it to the extreme by testing 40 different shades of blue to determine which shade received the highest conversion rate.

A/B testing allows companies to make controlled changes to pinpoint which designs outperformed others. It’s one part of a larger iterative process to improve usability and conversion rates over time. A/B testing thrives as a testing method when you make small changes to optimize a current solution.

Deciding What To Test

Let’s say you are building a mediation app and your goal is to increase conversion from the free tier to the paid tier. In order to increase conversion rates there are many things you can test. You decide to start with the call to action button.

You decide that your first test is a slight color variation of the call to action button. However, if you change the placement or the size of the button when running the color variation experiment, you are introducing more than one variable at a time, and it stops being an A/B test and it becomes a multivariate test. So you can keep testing different variations until you get your desired outcome.

Prior to running the A/B test on your meditation app, you want to start by gathering some baseline data on how your conversion rates are currently performing. This will give you a benchmark. After that, you want to identify your goals. For this test, your goal is to increase the conversion from a free account to a paid account. You can also run other A/B tests to increase the number of downloads, but that should be a different project.

Using Software to Create Variations of the Designs

You can use many different software tools to automatically send your users to the different versions of your test. Here are a few of the best software tools to automate the process.

Once it’s live, website visitors will be randomly assigned to one of the two groups: the control or the experimental. Whichever software you use to conduct the A/B test will measure and review the analytics.

Analyze the results

Synthesize the results from the test to see if there is a statistically significant difference between the two designs. It is important to iterate on your results, and keep testing to find the highest performing design. Now that you have tested the different button colors, you can play around with the text!

Advantages of A/B Testing

A/B tests are high value and low cost. Some common goals that companies have when conducting A/B tests are:

Increased web traffic
Higher Conversion rates
Lower bounce rate
Lower cart abandonment
Testing copy, such as Add to cart versus Buy Now
Advertisement copy
Images
Landing pages
Promotional emails

A success rate of 25% is really good, and you should expect an average of around 10%. So don’t expect to completely transform your conversion rate on the first test.

Limitations of A/B Testing

A downside to A/B testing is that you won’t understand why users prefer one version over the other. In order to understand the “why”, you should perform moderated or unmoderated sessions to supplement your A/B tests. This way, you can uncover your customer’s thoughts and feelings towards the designs. At PlaybookUX, we automate the process of speaking to your users through unmoderated and moderated sessions. These insights are powerful, because you’ll understand the context of your customer’s preferences.