One question we encourage our customers to ask is: “is it worth actually running this offer at all?”. Marketers sometimes don’t take the time to stop and ask themselves questions like this. In some cases it may be that they don’t really want to know the answer(!), but in others it is simply because all activity feels like good activity. Can it really be damaging to send a million push notifications?

The answer, of course, is ‘yes’. And it is even more ‘yes’ when there is some value being given away as part of the campaign.

To put it another way, there are essentially two types of A/B tests an organization should be running. The first establishes whether a campaign is worth running at all. The second then establishes what the right way of running it is. What is interesting is that although there are plenty of solutions out there for the latter, it’s surprising how infrequently products help their user answer the former.

We set out to change all that. We wanted to create a very simple and easy to understand process within the Swrve user experience that both prompted the user to consider this question, and then enabled them to answer it with the minimum of fuss.



The former challenge is to be addressed with what I hope is admirable simplicity, within the prompt shown. This simply puts in front of the user the option to add a ‘hold-out group’ to any test or campaign, with a default group size of 5%. We wanted to ensure that at all times Swrve users are aware that a group of users for whom nothing changes can be helpful in determining whether it was worthwhile running the campaign at all.

It’s also important to note that a hold-out group of this type is in all other ways identical to the campaign target group. It is a case of initially segmenting out that target group and then enabling the user to hold back a certain group. Attempting to reverse those steps won’t work, or rather will give some meaningless results!

Of course, if that has already been determined, it’s simple to ignore the requirement and get down to the second type of test we discussed above.

Showing The Results

I wrote a couple of weeks ago about the way in which we’ve begun to show results in Swrve. In this context there is an additional layer of complexity, but it is in fact relatively straightforward to show uplift against the ‘hold-out group’, which is ultimately what we care about.


The screengrab above shows the approach we are following. The performance of the hold-out group against the metric that we’ve decided we care about is displayed clearly (and the white color-coding remains consistent) at the bottom of the ‘actual’ results. In addition, a simple uplift number against each of those results demonstrates exactly how they compare against doing nothing.

From there ROI on any activity is easily calculated - and you can finally answer that awkward “why are you doing this?” question with confidence!