by Nilay Patel, Product Manager (Creative) at Aarki
What is creative testing?
“Creative testing” is your recipe to finding the perfect advert. It lets you systematically refine the design, messaging, and format of a creative, to maximize your campaign performance. The simplest way you can do this is by running multiple, different creatives at the same time, and measuring performance of each of them.
A more sophisticated creative testing process is scientific - iterative and hypothesis based. It starts with an understanding of your app’s audience, and then refining it based on real user data. Testing should be statistically significant - run across a large enough number of users (or impressions) so that natural randomness is smoothed out. 
Each iteration should test a specific hypothesis you have about your app and audience. This can be achieved by changing the design, messaging, and format of the creative.
Below are several examples of these three levers of change:
Why is creative testing important?
Creative testing facilitates you in achieving your growth goals. There are three key reasons you may be seeing performance issues from creatives that were previously working well:
- Your app segment is likely competitive and so successful creatives can be copied by competitors, reducing your impact
- As users see the same creative multiple times, “eyeball fatigue” may mean there is less incremental impact from showing a creative
- User tastes and preferences change over time, so creative testing lets you remain current.
The net effect of the above is that mobile apps, and games in particular, need to constantly test creatives to continue to grow.
Creative testing also lets you optimize for the metrics that matter most to you and your business. There are varying performance metrics such as click-through-rate (CTR), average revenue per user (ARPU), retention, and return on ad spend (ROAS). Different creatives may impact different metrics.
For example, if you run a seasonal event and theme creatives, it may appeal to new users and early retention metrics as users play the event, while showing leaderboard and PVP features may appeal to hardcore players who are likely to be more engaged and make a purchase.
The subtle nature of a user’s understanding and engagement with ad creatives is all the more reason to test many different options, with your campaign performance goals in mind.
How to run creative testing
Aarki has the pleasure of working closely with the largest and most successful game makers in the industry. We collaborate with them to develop creatives at scale, following a scientific process. Below, we examine a two-stage testing method, that systematically identifies the best concept for a creative using market data, and then uses variant testing to extend the life of a creative once it is being used at scale.
We’ll work through an example of a “Mahjong match-3” (MM3) type game, where a player must clear a 3-D board with multiple layers of tiles, by matching 3 tiles in a 7-slot hand section.
Concept testing
Concept testing involves designing multiple creatives, all very different from each other. Your app likely has multiple value propositions; for example, games can be social, IQ building or leisure activities for downtime. Concept testing lets you identify various value propositions and use them in their creatives.
Concept testing is not simply screenshots or videos of the actual app content. It requires translating app value propositions into concepts that are relatable to the user base. Think of the now famous “IQ header” that spells out intellectual benefits of an app that may not be standalone features.
Creative testing processes generate concept ideas through:
- Understanding the value propositions of an app
- Performing user research to understand the interests of the user base
- Performing market research to understand what is working for competitors
- Looking at creatives that have worked for in the past
Going back to our example of a MM3 app, value propositions in these apps could be:
- Relaxation - Users play this app as a break, and want to be calmed by it
- Completion - Users like the contentment of clearing a filled board, and want a sense of accomplishment from the game
- Your meta mechanic - There are a lot of mahjong match-3 games and so a differentiator is the meta mechanic (e.g. collections, personalization, building, saga map, etc.)
The value proposition, user, and market research are then processed through the levers of change discussed earlier - design, messaging, and format. Applying these levers in a systematic way means you can create a wide range of creatives and test against testable hypotheses.
Let’s look at the value propositions of our MM3 app example above, and say you want to test the concepts of relaxation and completion. You can now make creative ads that focus the design and messaging on these value propositions. Here’s an example of a relaxing creative and a completionist creative.
How are these creatives different?
This approach stretches your creative way beyond the typical, limited campaign theme, and enables you to learn the most about what type of creatives will work. If concepts are too similar, there is a risk of missing out on finding the best creatives. This approach generates a volume of different creative concepts, so it is equally important that you allow enough media coverage to test them all.
Next step - variant testing
By following the steps in this blog, you should be able to make and test different creative concepts for your mobile app. However, often, test budgets can be difficult to stomach, even for the most experienced growth marketing teams.
By systematically stating and testing hypotheses, the cost can be seen as an investment decision and the learnings from concept tests will directly impact future ROI. Once a successful concept is identified, you can leverage it for longer and with a wider effect to recoup the investment involved - this is done during variant testing. Watch out for our next blog that will talk you through all the steps needed to successfully run variant testing.
.gif)
