CRO data and the importance of statistical significance

The importance of CRO data is something we’ve highlighted before – but once you’ve got it, you need to use it correctly.

Which metrics are the right ones to use in order to determine if an outcome is significant?

Well, that depends on what type of test you’re designing and whereabouts in the user journey funnel it sits.

Obviously, key metrics for any CRO agency to measure are conversion and revenue.

These KPI’s are usually supported by secondary metrics such as engagement, bounce rate, click through rate and so on.

Ultimately, statistical significance simply means whether or not the difference between two numbers matters or is just an anomaly or fluke.

A reliable CRO specialist will know which statistics are important and which conclusion to draw to inform any website changes.

The magic 95 percent

Often, a rating of 95 percent significance is held up as the standard to prove that a test has a clear winner.

And while such conclusive evidence would be wonderful every time in a perfect world, in a commercial reality there are other factors to consider.

When you work in science or academia, there can be no room for risk and statistical significance of this magnitude has to be proven unequivocally.

But in the non lab condition world we work in, reaching that level of significance can take a long time, depending on traffic and conversions.

And that time costs money, which is why marketers will sometimes act on the probability of a winner.

The elusive 95 percent is an ideal goal – but the real prize for any CRO specialist should always be ROI for the client.

How long should I run an A/B test?

So, with the goal of statistical significance – but with the caveat of not having endless time and money – when is the right time to stop a test?

There are calculation tools available that can help predict both how long a test needs to be and its overall significance.

But even then, the person inputting the data at one end and interpreting at the other still needs a clear idea of what to look for before drawing any conclusions.

Put simply, there is no set length of time to run a test – a number of factors have to be considered each time, such as sample sizes and margins of error.

Cut an A/B test short too early and you can risk acting on false positives, or data that doesn’t reflect enough of your business cycle.

This is especially true of businesses selling products or services that are higher value or more complex.

But by running a test for too long, the likelihood of ‘length pollution’ increases, meaning that external factors can skew the data.

These might include seasonal holidays, campaign changes or unforeseen technical issues.

Ultimately, the more understanding a CRO specialist can gain of your business and customers, the better they’ll become at testing for the required duration.

And in turn, the statistical significance those tests produce will be better interpreted and turned into actions that improve your conversions.

Ready to make the best use of your CRO data?

Theorising a test, designing it, implementing it and generating results is a complex enough task.

But knowing how to interpret those, in terms of which statistics generated are meaningful, is another important discipline in itself.

Work with an experienced CRO partner who can explain the data to you and focus on the areas of your site that need attention.

Get in touch today, or if you know someone who might be interested, share this content and spread the love.