Multivariate Ad Testing To Improve Creative Data and Insights
The Challenge
A B2C client with a well-equipped creative department and full-funnel marketing strategy across all major social media platforms is interested in taking their creative to the next step, but feels like they lack direction. They’ve historically deployed AB tests for ads, but have found that to be difficult to manage with the number of creative in play and concurrent tests and they’re unsure how to incrementally improve on performance over time especially when AB tests give conflicting results or are impacted by confounding variables such as seasonality, platform, or audience.
the Approach
I recommended a multivariate testing strategy for several reasons that were specific to their business and marketing campaign setup.
What is multivariate creative analysis? While AB tests is used to test the impact of changing a single variable using versions A & B, multivariate testing changes several variables simultaneously to test the impact of each individually and the relationship between different combinations.
Example: Let’s say you’re interested in testing 3 variables in your ad:
Selling Point: Which selling point is most effective for your audience?
Ad Image Components: Does including a person in the ad image lead to improved creative?
Copy Length: Does using longer copy providing users with more information or content at the point of impression improve engagement?
If you were using an AB model, you would need one Control Ad (A) and test that against an ad with a different selling point (Ad B), then test the winner against an ad with a different ad image components (Ad C) and so forth. However, there are several things that you are not learning by choosing an AB approach over a multivariate approach, namely: how these variables interact with each other. You may find B to the best performing ad, but does it perform better or worse
The multivariate approach allows for multiple variables to tested simultaneously so that you get data on more variables faster.
It gives insight into how these variables interact with one another which can help discover synergies between a specific combination of variables.
By testing several variables concurrently, you reduce the possibility for time or seasonality to be a confounding variable that influences subsequent testing. The performance of some variables may be more (or less) important during some points in your seasonality.
It gives more time between testing periods. Multivariate tests require more time for data collection due to the data being split among a greater variety of ads. Because of this, it leads to less strain on designers and strategists by front-loading the development of new creative and not requiring constant iteration within a campaign.
Key Findings
By deploying a multivariate approach, the client was able to test the following attributes against each other:
Brand Themes and Product Selling Points, such as price, quality, prestige, and consistency.
Ad Formats, such as animated video/gif ads vs static imagery
There was also a test within the video variations on video length. Was engagement better or worse depending on whether the video was 6 seconds, 15 seconds, 30 seconds, or 2 minutes?
Copy Length, i.e. short ad copy that only contains text before clickable “read more” or long copy that had far more context on the ad post itself without requiring users to click.
The client was able to generate insights on all of these variables including which variables largely didn’t matter based on the available. In 3 months, the client had actionable insights on all of these variables and how they influence each other when it would have taken a full year of AB testing to get similar insights. Here are some of the high level findings:
Video length had little to no impact on CTR.
Copy Length had a positive correlation, with the longer copy variants performing 13% better than not.
Video ads performed substantially better than static image ads in most placements.
Selling points within the price and quality themes outperformed prestige and consistency.
On top of these high level insights, there were also more detailed insights related to the dynamics between these variables. For example, the prestige ads that highlighted brand reputation history were generally among the worst performers, but not when that ad was (a) a video (b) over 30 seconds in length. The message prestige ads were trying to communicate didn’t resonate in shorter or static placements, likely due to the fact that those placements lacked the information capacity required to convey such a nuanced idea.
Similarly, while quality was determined to be one of the better performing selling points, it performed better with longer copy variants, especially. Users that were interested in quality were interested enough to read beyond the “read more” buttons to learn more about the product.
