Banner design

Mobile Marketing - The Ultimate Guide To A/B Testing

The Ultimate Guide To A/B Testing

A/B testing has never been more popular. But too many organizations only use it in a superficial way - and fail to get the best out of it as a result.


That isn’t good enough, and here’s why. Across the myriad of emerging channels that characterise today’s consumer/brand interface, there is a war going on for the eyeballs, hearts and minds of the person in the street.

That war will not be won using old methods. The ‘spray and pray’ of yesteryear is - literally - worse than useless. It just alienates and irritates the customer. Instead, we need to listen and learn, and be there for the customer ‘in the moment’. In other words, be ready with the right interaction at the right time, and win enough of those moments to drive greater customer loyalty and revenue.

And one way to make that happen is to bring A/B testing into the heart of your development and marketing process. A/B testing enables you to know what works, put it into practice, and win the moment. Every time. As a result, it delivers on the business objectives that omni-channel businesses care about: retention, engagement, revenue.

This guide will help you get the most from A/B testing. We’ll go into a little further detail on what A/B testing is. look at building an A/B testing culture, and conclude with some ideas for A/B tests that will really make a difference.

What Is A/B Testing

In the simplest possible terms, A/B testing is the process of serving two or more alternative variants in user experience, to otherwise identical audiences, and then establishing which ‘works’ most effectively according to agreed criteria.

To look at little closer at the three key aspects of A/B testing covered in that short definition:

  • Alternative variants can relate to almost any aspect of user experience or the campaigns we send to customers:. price, visual design, information architecture, language and so on. They can also relate to the high-level experience itself, for example whether a bespoke onboarding experience should exist at all. Of course to test multiple variants, multiple variants need to be created.
  • Otherwise identical audiences are created in order to ensure results are statistically meaningful. We create these audiences by first building one audience that meets our criteria, and then splitting this audience at random. One ‘side’ gets experience A, the other experience B.
  • Establishing which works most effectively is the final step: measurement. Without measurement, and more specifically accurate and meaningful measurement, A/B testing is a pointless activity. It is vital to define success before beginning any experiment and ensure results are statistically significant.

It may be apparent already that delivering an A/B testing program incorporating all these elements would be a painstaking process if done manually. It is possible, of course, but greatly limits the number of tests it is possible to conduct. That in turn makes it hard to put in place the rapid test/optimise culture that marks out businesses able to respond quickly to real user data.

Instead, before embarking on an A/B testing program consider investing in a platform that supports the automatic creation, launch and reporting of A/B tests: across all the user experiences and campaigns that you deliver throughout the omni-channel environment. That includes the ‘core’ experiences delivered within apps and online - not just campaigns. A platform like this will greatly improve the effectiveness of your A/B testing efforts.

Building An A/B Testing Culture

A/B testing is an incredibly powerful and popular technique. It employs real user data to establish what truly works. Organizations that adopt A/B testing are able to iterate and optimise fast - and enjoy improved performance across the full range of metrics as a result.

But to make that happen it’s important to build a testing culture. Without one, you will struggle to get the best out of any A/B testing program - and could be wasting your time completely. Here’s three simple steps to help when building that culture.

  • Establish buy in. It sounds obvious, but the first and single most important step in building an A/B testing culture is ensuring that at all relevant levels there is a commitment to conduct testing and abide by the results. To make that happen, make clear what you are hoping to learn from each test and which decisions testing will help drive. It then becomes easier to show progress and avoid opinions over-riding user data (always fatal to any A/B testing effort!)
  • Put structure in place. Don’t test at random: that’s a recipe for seeing no clear results and undermining confidence in the A/B testing process. Instead develop a clear testing program, consider carefully what to test and when (some ideas are included below) and make sure to communicate successes.
  • Build a team. It isn’t necessary to go out and hire new people, but make an effort to build an internal team that is responsible for an A/B testing program and feels a sense of ownership. Bring individuals from every level into that team and make sure it encompasses both the data science and creative side of the organization at the very least.

Hopefully by following the advice above you’ll soon have an A/B testing culture in place. But nothing succeeds like success. You’ll want to ensure you deliver great tests that drive the metrics that really count for the business. It’s imperative to be able to point to real success early in the process.

Where To Start A/B Testing

With that consideration front of mind, here are 5 areas of the omni-channel experience that will really benefit from an A/B testing program:

1. Onboarding - Or ‘The First Five Minutes’

A/B testing can be used to improve the onboarding experience for users, and deliver stronger retention rates and revenues as a result. Nothing matters more than the first five minutes after all.

A typical onboarding process guide the user through their first interactions with the brand (in many cases before any purchase has been made). It’s vital to get it right, and A/B testing can rapidly help your company optimise that process by providing data relating to a number of key decisions, including:

  • How many screens should be shown in a mobile onboarding process, and which content should be shown within them. Whether onboarding customized to acquisition source will help improve retention and revenue.  
  • For multi-channel businesses, when is the right time to introduce new channels and encourage migration? Which channels should be suggested first? Which add most value when adopted?
  • Is it best to wait until later in the onboarding process to show certain product and service features? Is there a channel on which it makes more sense to promote certain features?
  • Registration. When should it take place, and how much information do we need to ask for?

In fact, you should also be testing whether you need a dedicated onboarding process at all.

2. Navigation And Menu Structures

There are a wealth of options when it comes to navigation and menus: A/B testing can help you choose the right one for your content and audience.

How a menu system works, the way in which content is displayed, and the way users move and browse through the experience and across multiple channels: all these things are important to get right. Do so and you’ll see improved retention and engagement rates across the board.

Some potential A/B tests that can be run in this areas include:

  • Whether to use hamburger type navigation, traditional menus or some other approach
  • What order menu items should be displayed in for maximum engagement and retention
  • Display rules within various channels and across various devices, such as whether to include images with each story on a media or news app
  • Where to include specific content and features, and at which level in the navigation

All of these questions can have a significant impact on the performance of the app, and by A/B testing alternative approaches it is possible to optimize and iterate rapidly for success. We’ve seen Swrve customers double engagement (total time spent in app) simply by testing and changing the navigation design.

3. The Checkout Process

For retail businesses - or indeed any service built around tasks like booking a taxi or making a hotel reservation, it is absolutely vital that the checkout process is optimised to perfection. A/B testing is the way to make that happen.

There are a myriad alternative approaches to these common interactions, and more so again now that we interact with these services on so many devices. Many of these are new, innovative and make clever use of ‘native’ attributes to each device (think swiping the smartphone screen on Tinder).

This emerging landscape of possibilities makes it tricky enough to predict what approach will work most effectively when it comes to building flows and interactions designed to win that most important moment of all - the moment the user converts to become a paying customer. Or a more valuable repeat customer.

That in turn means A/B testing.

If it feels too overwhelming to build multiple versions of an entire process, consider first performing a funnel analysis on the various steps involved. This will help determine where exactly users have difficulties, and enable you to zero in on areas where it might be worth trying something different. That can be simple steps like changing the position or language used on a button, or more radical changes like removing or moving key steps of the process itself - or checking to see whether the process might work better in another channel.

Either way, a clear and focused A/B testing program will help make transacting with your business as smooth as possible - and that in turn means a whole lot more success.

4. Messages And Alerts

A huge percentage of the brand/consumer interaction now takes place within messages and alerts. Whether push notifications on mobile or desktop, traditional email, in-app messages on smart TV - as businesses we have a whole range of ways to interact with consumers directly.

A/B testing can be used to optimise these campaigns, improve customer experience, and as a result drive loyalty and revenue.

Amongst an almost infinite variety of options, a messaging A/B testing program can help establish:

  • Whether to message at all. Is it effective or not to deliver messages at certain moments? Are we improving customer metrics with specific campaigns - or the opposite?
  • The right content and creative to send. There are many ways to get a message across. Which works best? A/B testing will tell you. It will also help you personalize content for specific audiences.
  • The right channel to interact in. If you have a particular type of message to deliver, what’s the smartest way to deliver impact? Email? Push? Something else? A/B testing can help establish which approach works best for you.

The results can be impressive. We worked with one client on a range of tests that delivered a 30% increase in purchase completions when compared with standard untested messages.

5. Default Configurations

In most digital services, default values are everywhere. To give just three examples:

  • How many free articles on a media app can be read before a paywall kicks in?
  • How much virtual currency does the player of a mobile game start with?
  • What is the default top up for a pre-pay mobile when I visit the site or app of the service provider?

All these are questions for which a business needs answers, and the answers can have surprisingly dramatic effects on the success or failure of a service. A/B testing is the best way to deliver those answers. A testing program can help find the ‘right’ values quickly - particularly when compared to the alternative: guessing.

Remember that it is possible to personalize experience for different groups of users. Defaults don’t need to be the same for everybody - so play around with customizing the experience for specific users based on their previous behavior or even the acquisition source they have just arrived from.


If you want to see how Swrve can manage both push notifications and other customer interactions on multiple channels, arrange a demo today