Assessing Marking Campaigns
Promotional Performance & A/B Testing
The selection of marketing campaigns for business is often challenging due to feature similarities and the sales results they yield.
In this analysis we will observe three marketing campaigns in a retail environment. Picture this — a new item has been introduced in several markets. The markets were randomly selected and different promotions are used at each location. How would you pick? What factors would you consider?
For our illustrative analysis, the market results were tracked and our analysis offers additional insight into the most effective marketing campaign.
The Data
The project data is made available via Udemy course Data Science & Deep Learning for Business. A full Udemy source citation is available below under sources.
The analysis was completed using Python libraries; Pandas, Numpy, Matplotlib and Seaborn.
Feature Details
The data includes 548 promotional records identified by the following features:
- MarketId: unique market identifier
- AgeOfStores: age of store
- LocationID: unique store identifier
- Promotion: promotion identifier
- Sales in Thousands: sales per store, promotion and week
- Market size: small, medium & large market ID
- Week: week 1–4 of the promotion
Note: The dataset is manufactured and as such requires no significant data processing. Typically, businesses closely guard their promotional data so real use cases are rarely available for public analysis. The following A/B testing and analysis is for educational and illustrative purposes only.
To continue; the dataset includes 7 features representing 3 promotions inclusive of 137 stores. Records were kept for each week the promotion was active.
Sales by Promotion
Total Sales
Promotion 1: $9,993.03 | Promotion 2: $8,897.93 | Promotion 3: $10,408.52
Promotion 1 has the smallest general distribution of promotions but has a significant number of higher value weeks recorded. It may be possible that promotion 1 ended up with a smaller amount of promotional opportunities due to the random selection.
Promotion 2 is showing most of its weekly values distributed closer to its average with fewer ‘high’ value sales weeks recorded.
Promotion 3 appears to be a front runner with a large distribution and a significant amount of the recorded sales being on the higher end of the spectrum.
We’ll continue by assessing market opportunity with respect to each small, medium and large market.
Promotions by Market Size
Promotion 1 has the lowest number of promotional counts across all three markets.
Promotion 2 has the most ‘large’ market promotional counts.
Promotion 3 has the most ‘small’ and ‘medium’ market promotional counts.
Sales, Mean, Standard Deviation and Total Occurrences
To fairly assess the three groups against each other we will break down a few statistics. The following values are in thousands:
Promotion 1
Mean Sales: 58.09 | Standard Deviation: 16.55 | Promotion Count: 172
Promotion 2
Mean Sales: 47.33 | Standard Deviation: 15.11 | Promotion Count: 188
Promotion 3
Mean Sales: 55.36 | Standard Deviation: 16.77 | Promotion Count: 188
A/B Testing Using Python
Our analysis will test T-Values and P-Values between promotions in order to assess performance. While the mean values have been compared, our T-Values will determine if there is a significant difference between the promotions with respect to variance and outliers. Our P-Value test will assess whether difference in promotional performance is significant (or not significant).
from scipy import statsPromotion 1 versus Promotion 2
# Calculate t & p / scipy stats
t, p = stats.ttest_ind(df.loc[df['Promotion'] == 1, 'SalesInThousands'].values, # isolate values as numpy array
df.loc[df['Promotion'] == 2, 'SalesInThousands'].values,
equal_var=False)
print('T-value: ' + str(t))
print('P-value: ' + str(p))Result
T-value: 6.42752867090748
P-value: 4.2903687179871785e-10The sales counts and mean values for promotion 1 are larger than those in group 2.
The T-Value of 6.23 illustrates a degree of difference (variation) between the two promotions. The P-Value of 0.0000000004 (when compared to a threshold of 0.05) means difference between the promotions is statistically significant.
Promotion 1 out performs promotion 2 in a significant way.
Promotion 1 versus Promotion 3
# Calculate t & p / scipy stats
t, p = stats.ttest_ind(df.loc[df['Promotion'] == 1, 'SalesInThousands'].values, # isolate values as numpy array
df.loc[df['Promotion'] == 3, 'SalesInThousands'].values,
equal_var=False)
print('T-value: ' + str(t))
print('P-value: ' + str(p))Result
T-value: 1.5560224307758634
P-value: 0.12059147742229478Promotion 1 and 3 have a very similar mean, standard deviation and total sales per promotion (difference is a few hundred dollars).
The T-Value and P-Value illustrate no real significance between the two groups. While promotion 3 ended up with more sales — this may be explained by the additional 16 promotional cites counted under promo 3.
Failed to reject the null hypothesis. There is no statistical difference between promo 1 and promo 3.
Interpreting Results
A/B test results showed promotion 1 outperformed promotion 2 in a significant manner. A/B test results did not show a significant difference between promotions 1 and 3. The mean and standard deviation values were very similar and while promotion 3 lead to more sales, the selection of promotions per market per location were done at random. There were 172 occurrences of promotion 1 and 188 172 occurrences of promotion 3. The difference in sales performance is more than likely due to random chance.
Limitations
The analysis was completed for illustrative purposes only. As mentioned at the top of our notebook — businesses vary rarely make sales data available to the public and as such this project was completed using data from a Udemy course.
The main challenge to this project is understanding how similar the provided dataset is to something that will be seen in the workplace. This may be effected by year (when compared to today’s data), region and many other industry factors.
Future Work
Future work should include a larger dataset with additional context. While fabricated, an ideal dataset would mimic current sales trends, in a specific market with multiple products and promotions to test for effects of markets or locations themselves on the the promotion. Understanding these effects will lead to a more functional determination of ‘best’ promotion for this business within their industry.
Further Information
For any additional questions, please view our analysis on Github or reach out via email at santana2.miguel@gmail.com, on LinkedIn or on Twitter.
Source | Udemy
Additional analysis, notes and file sources can be found on Udemy’s website. Course Name: Data Science & Deep Learning for Business by Rajeev Ratan
