Foundations for Development and QA Before Running A/B Tests

Foundations for Development and QA Before Running A/B Tests

A/B testing: the website equivalent of walking into a country pub and tasting all the beers on tap before committing to a full pint.
In this metaphor, the “beers” represent different variants of a particular area or function of a website.

Questions like:

• “Which will taste the best?”
• “Which one provides the best value for money?”
• And most importantly, “Which provides the best results?”
  …are what A/B testing helps us answer.

Start with the Six W’s of Experimentation

Before a single design sketch or line of code is created, the experiment planning process must begin. The foundations are built on the six W’s:

• What?
  Identify the purpose of the test. What are you trying to improve or validate?
• Why?
  Define clear primary objectives (e.g., increased click-through rates, uplift in revenue per visitor).
• Where?
  Which part of the website will host the A/B test?
• When?
  How long will the test run, and when will it conclude?
• Who?
  What percentage of users will see each variant? (e.g., Control: 50%, Variant A: 25%, Variant B: 25%).
• Win?
  What constitutes a “win”? For example, revenue per customer increased by 12%.

Answering these questions is crucial for gathering requirements and evolving them into clear, concise instructions for design, development, and QA teams.

Design Workshops

A design workshop is an excellent way to bring collaborative ideas to life. Whether it’s on a whiteboard, in an online document, or even scrawled on the back of a cigarette packet, you should leave the workshop with a canvas and colour palette that will allow the artist to create a masterpiece.

QA Starts Earlier Than You Think

What comes to mind when you hear “QA”? Probably something that happens late in development, right? Wrong! QA starts before the development journey begins.

• Write your test cases early, using the six W’s as a guide.
• Vigorously test the control variant (the original user experience) to ensure it meets baseline requirements and functions correctly.
• For your new variants, ensure all changes – whether minor tweaks or drastic redesigns – are equally functional.

Fair Traffic Distribution

Your variants deserve a fair chance to shine. In most cases, this means evenly distributing traffic across the control and test variants.
Think of it like a job interview: both candidates are dressed professionally (albeit in different outfits), and it’s only fair they get the same questions and equal amount of time to prove their value.

Timing Matters

The duration of your test is key. Avoid running tests during unpredictable periods like Christmas or New Year, as seasonal behavior can skew results.

• General sweet spot: 2–4 weeks.
• Adjust based on traffic levels and seasonality.

Feature Flags: Your Safety Net

Deployments can be nerve-wracking, but feature flags are like an emergency off-switch for your experiment.

• Bugs? Flip the switch.
• Unexpected issues? Flip the switch.

Wouldn’t life be easier if everything had a feature flag?

Monitoring: Beyond the Primary Metrics

Once your test is live, the real work begins. Monitoring isn’t just about making sure traffic splits are correct.
You need to dig deeper and ask out of the box questions. Did your test accidentally tank another metric? (e.g., bounce rate, page load speed)? Are users abandoning the checkout? Is revenue affected?

Don’t let early results mislead you—stay patient and let the test run its full course.

Crowning the Winner

When the test ends:

1. Verify the data.
2. Check if the results align with your goals.
3. Decide which variant achieved the desired objectives.

If the outcomes are unclear, don’t hesitate to re-run the test. A second round can sometimes reveal the real truth.

Learning from “Losers”

If all new variations flop statistically, don’t call it a failure. It’s valuable data. You’ve learned what doesn’t resonate with your users, and that’s just as important as finding what does.

With solid planning, rigorous QA, and careful monitoring, you’ll master the art of A/B testing and make data-driven decisions like a pro. Cheers to that—and to picking the right “beer” for your website!

Author: Kieran Priest, Project Manager