How to diagnose change in performance of ad campaigns?
Understanding why ad campaigns underperform is a common and critical application of Root Cause Analysis (RCA) in internet businesses.
The structured approach we learnt previously can pinpoint the specific factors contributing to lackluster results, allowing businesses to make data-driven decisions for future campaigns. Let's cover the 4 step approach in this use case:
1. Defining Metric Hierarchy:
The first step in diagnosing an underperforming ad campaign is defining the metric hierarchy based on the campaign's objectives. For instance, if the primary goal is increasing sales, top-level metrics might include overall sales and ROI.
These top-level metrics can then be broken down into more specific, independent components such as click-through rate (CTR), conversion rate (CVR or ECR), and average order value.
2. Exploratory Data Analysis (EDA):
Following the identification of the metric hierarchy, the next step involves conducting EDA to examine how these metrics have changed over the campaign period and isolate the input metrics that have shown significant variation.
This might include analyzing trends in user engagement, comparing pre and post-campaign performance, and looking at external factors that could have influenced the campaign's effectiveness.
3. Hypothesis Generation:
Based on the initial EDA findings, the team formulates hypotheses about the cause of the underperformance.
For instance, they might hypothesize that the constant CTR and low conversion rates are due to ineffective ad content or a non-optimized landing page experience.
4. Deciding on Statistical Technique:
To test the formulated hypotheses, the team decides to use specific analytical techniques.
For instance, they might choose correlation analysis to study the relationship between ad spend, CTR, and conversion rates, aiming to validate the hypothesis about ad content effectiveness.
Let's use these steps in an example:
Consider a scenario where a Bangalore-based e-commerce platform launched a campaign aiming to boost sales during the festive season. The campaign was expected to increase monthly sales by 30%, amounting to an additional ₹1 crore. However, the actual increase was only ₹25 lakhs.
Using the RCA approach:
- 1. Identify Metric Hierarchy: Sales growth and ROI are identified as the top-level metrics. These are further broken down into impressions, CTR, conversion rate, and average order value.
- 2. Conducting EDA: The team examines changes in these metrics during the campaign period. It's noted that while impressions increased, the CTR remained constant, and conversion rates were below expected levels.
- 3. Formulating Hypotheses: Based on the EDA findings, hypotheses are generated. For example, the team hypothesizes that the constant CTR combined with low conversion rates might be due to ineffective ad content or a suboptimal landing page experience.
- 4. Identifying Analytical Techniques: To test these hypotheses, the team decides to use correlation analysis to study the relationship between ad spend and conversion rate.
RCA insight: After conducting the correlation analysis, it was revealed that while the campaign successfully drove more traffic to the site, issues with ad content and user experience on the landing page led to lower-than-expected conversion rates.
Takeaway:
Diagnosing underperforming ad campaigns requires a methodical approach, starting with a clear understanding of the metric hierarchy and followed by targeted data analysis to uncover the root causes.
By applying RCA techniques systematically, businesses can derive actionable insights, ensuring more effective and successful campaigns in the future.