Creative Testing for User Acquisition. Part 2: Variant Testing

by Nilay Patel, Product Manager (Creative) at Aarki

In our previous article, we covered how to brainstorm, design and test ad creative concepts. We learnt that concepts describe large changes that equate to different value propositions of an app. The best way to find a concept that works is taking a hypothesis-based approach, testing different options and honing in on a creative design that resonates with your audience. 

Today we will cover variant testing - a systematic approach to refining creatives that will help make sure you get the most out of concepts that you design. This will help increase your ROI and let you reach your app growth goals. 

What is variant testing

Picture yourself in your app growth engine. You have just activated a new ad concept and it’s like a high-friction part of the engine. At first, it worked like a charm. Your app reaches top speed. It’s faster than ever. Over time, however, you realize something is changing. Your new, shiny creative is experiencing wear and tear each time it's used and this means it’s losing its edge and becoming less impactful. 

Variant testing, in its simplest form, is giving ad concepts a tune up. It’s the small changes that keep an existing concept working in good order, without drastically changing its fundamentals. 

Going back to the idea of an app growth engine, what might happen when the engine is used in a slightly different context? Just like a road car that needs to be used as an all-terrain vehicle, you’d expect there needs to be some changes to its design and build. This illustrates the second type of variant testing we will discuss - making existing concepts work for different audiences, be that geographic, demographic or behavioral. 

Why is variant testing needed? 

The most important reason to do variant testing is to extend the life of a creative. The process of concept testing is expensive, with real ad dollars needing to be spent on testing. So when a winning concept is found, it needs to be nurtured and used for as long as possible. Unfortunately, mobile ads are “mean reverting” and a winning concept will likely get less effective over time due to competitors producing similar ads or due to “eyeball fatigue” in target audiences.

Second, variant testing lets you improve campaign KPIs such as ROI, CTR or install rate. Concepts are broad ideas of what works. Within a concept, there may be better performing variants. Academic literature often likens this to trying to reach the highest peak in a range of hills. Concept testing lets you find the highest hill in an area, while variant testing lets you finally reach the peak of a hill and get the best views (or, as in this case, the best ROI).

How should you perform variant testing? 

In our last article, we discussed the three main features (and related sub-features) of a creative that can be changed to find new concepts: design, messaging, and format. In its simplest form, variant testing aims to optimize a concept through small changes to several sub-features, or a large change on one or two sub-features. 

Creative refresh

Let’s look at an example of performing a creative refresh to improve a slowly degrading performance of an ad. Imagine you manage UA for a mahjong match-3 game, and after a few rounds of testing, the winning concept is identified. 

How do you know a creative refresh is needed? This should be done when there is a clear softening in the KPIs of a creative such as a decrease in ROI. Alternatively, perhaps you’ve mined the data and know that on average a creative will perform well for 6-weeks, and so as you approach that end date, you can start to make variants to extend that timeframe.

Variant 1

Variant 1 adjusts the call-to-action button and the wording of the creative. How do you choose what changes to make? In general, as it is inexpensive to make small adjustments, it is safe to use trial and error to refresh creatives. Alternatively, it is equally viable to go through the same hypothesis-based process discussed in the last blog.

Changing Audience

Now let’s look at the other type of variant testing - when you change your audience. Maybe you launch in a new geo or its heading into the end of year festive period. Variant adjustments may be adding Santa hats to characters and switching to a festive color scheme. 

Variant 2

Again, the creative concept stays the same but now it's more appropriate to the new audience and timeframe.

Putting variant testing into action

In this article we’ve discussed the role of small adaptations to existing creatives that can result in improving campaign KPIs or extending the life of a creative. Naturally, AB testing is also needed in variant testing - the original creative would be the control creative and the changed variant would be the test creative. 

Using variant testing, together with concept testing gives marketers the tools to generate, select and improve creatives. Want to boost your next app marketing campaign with Aarki's creative and analytic experience? Shoot us a message here.