A/B Testing Case Study Projects

Most people learn A/B testing from a textbook definition. Something like: "A/B testing is a method of comparing two versions of something to determine which performs better."

That's technically correct. And completely useless if you're trying to actually get a job doing it.

The companies that hire product analysts, growth marketers, and data scientists don't care if you can define A/B testing. They care if you've done it — or at least thought through it rigorously enough to simulate doing it. That's where A/B testing case study projects come in.

This blog walks you through what these projects really involve, the best case study ideas you can work on, real-world examples from top companies, and how to turn this skill into a career that pays well and keeps growing.

What Is an A/B Testing Case Study Project?

An A/B testing case study project is a structured analysis where you design, run (or simulate), and interpret an experiment meant to improve a specific business metric.

It's not just about splitting traffic 50/50 and checking which button color got more clicks. A proper A/B testing project involves framing a hypothesis, choosing the right metrics, calculating sample sizes, running statistical tests, and drawing actionable conclusions - even if you're working with a simulated dataset or a public dataset from Kaggle.

Done well, it shows you can think like a product person, reason like a statistician, and communicate like someone who actually cares about business outcomes. That combination is rare, and companies pay well for it.

Core Concepts You Must Understand Before Building a Case Study

Before jumping into project ideas, make sure your foundation is solid. Interviewers will dig into these concepts the moment you put "A/B testing" on your resume.

1. Hypothesis Formulation

Every test starts with a clear hypothesis. Not "let's see what happens if we change the button color," but "changing the CTA button from grey to orange will increase click-through rate on the pricing page because it creates higher visual contrast and urgency."

Good hypotheses have a change, a metric, and a reason.

2. Control and Treatment Groups

The control group sees the existing version 

A). The treatment group sees the new version

B). Groups must be randomly assigned and statistically comparable at the start.

3. Statistical Significance and P-Values

This is where most students get shaky. Statistical significance tells you whether the difference you observed is likely real or just random noise. The p-value is the probability of seeing your results (or more extreme ones) if there were actually no real difference. A p-value below 0.05 means there's less than a 5% chance the result happened by luck - the industry standard for "significant."

4. Sample Size and Power

Testing with too few users means your results are unreliable. Power analysis tells you how many users you need per variant to detect a meaningful difference with confidence. Tools like Evan Miller's A/B test calculator make this approachable, but you should understand the math behind it.

5. Metrics: Primary, Secondary, and Guardrail

Your primary metric is the one you're directly trying to move (e.g., conversion rate). Secondary metrics give context (e.g., average order value). Guardrail metrics make sure your change isn't breaking something else (e.g., making sure session time doesn't crash while conversion improves).

Common Biases to Watch For

• Novelty effect: Users engage more with anything new, temporarily inflating results
• Simpson's paradox: Aggregated results can hide opposite trends in subgroups
• Peeking problem: Checking results before the test is complete inflates false positives
• Network effects: When treating one user affects others (common in social apps)

Tools and Software for A/B Testing Projects

Knowing which tools to use - and mentioning them in your case study - signals professional readiness.

You don't need all of these. For a student project, Python + Jupyter + a clean dataset + good storytelling will get you 90% of the way there.

20 A/B Testing Case Study Project Ideas (From Beginner to Advanced)

Beginner Projects

1. Button Color and CTA Optimization

Use a simulated e-commerce dataset. Test whether changing a "Buy Now" button from blue to green increases click-through rate. Walk through hypothesis, sample size calculation, chi-square test, and conclusion.

Simple topic - but if you go deep on the statistics and present it cleanly, it's a surprisingly strong portfolio piece.

2. Email Subject Line A/B Test

Download a public email marketing dataset (many exist on Kaggle). Test whether a personalized subject line ("Hey Priya, your cart is waiting") outperforms a generic one ("Don't miss out - limited time offer"). Measure open rate and click rate.

3. Landing Page Headline Test

Design two versions of a landing page headline for a fictional SaaS product. Use simulated traffic data to test which headline drives more sign-ups. Great for marketing-focused portfolios.

4. Push Notification Copy Test

Test two push notification variants for a food delivery app: urgency-based ("Order in the next 10 minutes!") vs. value-based ("Your favorite restaurant is offering 20% off"). Analyze click rate and conversion using a z-test for proportions.

5. Form Length Optimization

Simulate data from a sign-up form with 5 fields (control) versus 3 fields (treatment). Test whether reducing form length increases completion rate. Discuss the trade-off between data collection and conversion.

Intermediate Projects

6. Pricing Page Layout Test

Design a case study around two pricing page layouts - one with a "Most Popular" badge (treatment) and one without (control). Use statistical testing to determine if social proof elements increase plan upgrades.

Great for product analyst and growth roles.

7. Search Algorithm Recommendation Test

Netflix, Spotify, and Amazon all A/B test their recommendation algorithms. Simulate a test where users are split between two recommendation models. Use engagement metrics (clicks, time spent, items added) as your outcome variables.

8. Onboarding Flow Optimization

Build a case study around a mobile app's onboarding experience. Test a 3-step onboarding flow versus a 5-step flow. Measure Day-1 retention rate and task completion as primary metrics. Handle drop-off analysis at each step.

This type of project is extremely common in product analyst interviews - build it well and you'll have a story to tell.

9. Checkout Flow Simplification

Analyze whether removing a "Create Account" step from checkout (replacing it with guest checkout) increases purchase completion rate. This mirrors a real Booking.com test that reportedly generated $300M in additional revenue - cite the case and build your own simulation.

10. Discount vs. Free Shipping Incentive Test

E-commerce classic. Test whether "10% off your order" or "Free shipping on your order" drives higher conversion and average order value. Handle the nuance that one metric may improve while the other declines.

11. Dark Mode UI Test

Test whether a dark mode toggle prominently placed in the app header increases feature adoption and 7-day retention compared to burying it in settings. Good for mobile-focused portfolios.

12. Content Personalization Test

Use a public news or e-commerce dataset. Build a simple personalization model and test it against a non-personalized baseline. Measure CTR and session depth. Shows ML + experimentation skills together.

Advanced Projects

13. Multi-Armed Bandit vs. Traditional A/B Test

This is a genuinely sophisticated project. Implement both a traditional A/B test and a multi-armed bandit (epsilon-greedy or Thompson Sampling) on the same simulated dataset. Compare their performance in terms of regret (missed conversions during the test). Show when each approach is appropriate.

This kind of project separates candidates who've gone deep from those who've just watched YouTube tutorials.

14. Bayesian A/B Testing Implementation

Implement a full Bayesian A/B test from scratch in Python using PyMC or scipy. Compare it to the frequentist approach on the same dataset. Write a clear explanation of when Bayesian methods are preferable - especially for small samples or continuous monitoring.

15. Sequential Testing for Early Stopping

Build a case study around sequential testing methods that allow you to peek at results without inflating false positives. Implement a simple sequential probability ratio test (SPRT) and show when it would have safely stopped an experiment early versus when continuing was necessary.

16. Feature Launch Experiment with Network Effects

Simulate a social platform where testing a new "follow recommendation" feature on some users affects the experience of users who weren't in the treatment group (because the people they follow might be in treatment). Model the network interference effect and propose a clustered randomization strategy.

17. Long-Term Holdout Experiment

Build a case study examining the difference between short-term A/B test results and long-term impact. Simulate a scenario where a feature shows positive results at Day-7 but neutral or negative impact at Day-30 due to novelty effect decay. Discuss how holdout groups are used at companies like LinkedIn and Facebook.

18. A/B Testing for Pricing Strategy

Build an end-to-end pricing experiment case study. Test three price points for a subscription tier. Use survival analysis to model churn at each price. Handle the ethical considerations of charging different users different prices. This is a real challenge growth and monetization teams face.

19. Experiment Analysis Audit

Take a real or simulated "failed" experiment and audit it for statistical errors - insufficient sample size, peeking, multiple comparisons problem, imbalanced groups, or biased segment analysis. Write up what went wrong and how it should have been run. Shows analytical depth and critical thinking.

20. End-to-End A/B Testing Dashboard

Build a full analytics dashboard (using Python Dash or Tableau) that ingests experiment data, calculates statistical significance in real time, visualizes conversion funnels, and flags experiments with potential bias issues. Package it as an open-source tool on GitHub.

This kind of project is a portfolio showstopper.

Real-World A/B Testing Case Studies From Top Companies

Understanding how top companies run experiments makes your own case studies more credible and context-aware.

1. Google: The 41 Shades of Blue

In the mid-2000s, Google couldn't decide between two shades of blue for its toolbar links. Rather than debate it in a meeting room, they tested 41 different shades of blue on real users. The winning shade reportedly generated an additional $200 million in annual revenue.

The lesson: even tiny UI changes at scale have massive financial impact. Every pixel is a hypothesis.

2. Booking.com: Guest Checkout

Booking.com removed the mandatory account creation step from their checkout flow, replacing it with a simple "continue as guest" option. This single experiment - later cited as one of the most valuable UX tests in history - dramatically increased booking completions.

The lesson: friction kills conversion. Reducing steps in a funnel almost always wins, but you need data to prove it.

3. Airbnb: Professional Photography

Early Airbnb noticed that listings with high-quality photos converted significantly better. They didn't just assume it - they ran a controlled experiment. Listings photographed professionally were tested against those with owner-uploaded photos. Conversion rates improved enough that Airbnb built an entire photography program around the finding.

The lesson: qualitative observations need quantitative validation before they become product decisions.

4. Netflix: Thumbnail Testing

Netflix tests every thumbnail you see. The image shown for a movie or show varies by user segment, viewing history, and device. They run hundreds of experiments simultaneously, with each thumbnail test measuring click-through rate and, crucially, whether clicking it leads to sustained viewing (not just a play that ends in 2 minutes).

The lesson: your success metric needs to be truly aligned with the business goal, not just the immediate action.

5. Duolingo: Notifications and Streak Mechanics

Duolingo famously optimized their push notification copy and streak mechanics through extensive A/B testing. They found that slightly guilt-inducing notifications ("You made Duo sad ") outperformed generic reminder copy in driving daily active use.

The lesson: emotional resonance in copy can be tested, not just debated.

How to Present Your A/B Testing Case Study

Building the project is half the job. How you present it determines whether a recruiter or interviewer actually engages with it.

Structure Every Case Study Like This:

1. Business Context: What product are you analyzing? What problem exists? What metric needs improvement? Ground everything in a real or realistic business scenario.

2. Hypothesis: State your hypothesis clearly. Change → Metric → Reason. Don't skip the reason.

3. Experiment Design: Who is the target user? What are the control and treatment variants? How long will the test run? What sample size is required and why? What are the primary, secondary, and guardrail metrics?

4. Data and Analysis: Show your statistical test (z-test, chi-square, t-test, or Bayesian equivalent). Show your p-value and confidence interval. Visualize the distributions. Don't just show the final number - show the work.

5. Results Interpretation: Did the test reach significance? What does that mean in practical terms? A 2% lift in conversion rate sounds small - until you scale it to a million users and calculate revenue impact.

6. Recommendation: What should the product team do based on this result? Ship the variant? Kill the idea? Run a follow-up test? Show that you think in business terms, not just statistical ones.

7. Limitations and Caveats: Every good analyst mentions what could go wrong. Was the test long enough? Was there a novelty effect? Were the groups truly comparable? Showing awareness of limitations is a sign of intellectual honesty - and interviewers love it.