Calculating the top performing ad—the “champion”—is easier if the best ad elements are determined. In order to show the impact of a single element on the campaign’s performance, we analyzed a programmatic advertising campaign for a casual game app over a period of two months. The objective of the case study was to find out which variant for a specific ad element would outperform the others.
The analysis was done in two phases:Phase 1
We started the test with 15 variants for a specific ad element. After testing and comparing the performance of different variants through machine learning technology and multivariate testing, we chose the top performing ones, narrowing down the number to seven. Once we narrowed it down, we were left with Variant 1, Variant 2, Variant 4, Variant 6, Variant 8, Variant 11, and Variant 12.
We then tested all these seven variants with equal allocation and gathered enough impressions that would allow us to make conclusions and changes.
Variant 8 showed the lowest cost per install (CPI), highest conversion rate (CR) and install rate (IR). It also had the most number of purchasers resulting in the lowest cost per purchase (CPP) in comparison to the other variants.
On the other hand, we received a very high amount of purchases for Variant 2. Both Variant 2 and Variant 8 outperformed the others which led us to the decision to remove the other variants.
Once we were down to two variants from the initial fifteen, we compared the performance and determined the best performing variant. During the initial phase of the campaign, there was no significant difference in performance between the two variants, so we continued running both until we gathered sufficient data. Here are the results:
- Variant 2 was delivering the best return on ad spend (ROAS) at ~50% and more frequent purchasers among all variants.
- Variant 8, on the other hand, had better numbers in terms of CPI, CR and IR.
- With the data gathered and analysis made, we can now keep and consider Variant 2 as the “champion.” CPI, CR and IRs were significant for the campaign but Variant 2 had more frequent purchasers and purchases. It also proved to be the “champion” as it defeated all the other variants throughout the testing phase.
This multivariate test campaign was to identify the best performing variant for an ad element in a display ad format. However, it is important to consider that we have seen different performance for these same variants for the other ad formats.
We do this multivariate testing of variants once the campaign is at scale to better understand the impact of small changes on campaigns, and to continuously provide superior performance. At Aarki, we keep creativity at the core of advertising and always try to find new and innovative solutions to advertise mobile apps. This helps us to provide the assurance that your marketing budget will always be spent on the best performing ad creative.