A/B testing has become popular among many websites and many customers.  However the client side solutions introduce performance delays so the whole process has always felt wrong to me. If you care about conversion so much that you want to optimise the usability as much as possible, why would you sacrifice performance and the negative affects on conversion in order to “hopefully” find this out?

I carried out a study into exactly what affect these tools had on performance and tried to provide some best practice advice if you do decide to use such tools.

The graph below shows the distribution of Sites with an A/B testing solution and those without and plots the Speed Index.  What is clear is that those with A/B testing tools have shifted to the right and as such introduce delays in building the page.  The same can be said for other metrics including Render Start, Content Load, On Load & Fully Loaded.

Looking at key metrics of Render Start and onLoad I applied thresholds for good, average & poor performance and reviewed the sites using some of the most popular tools.

It is clear to see that sites with these tools installed do not perform as well as those without.  However, what is interesting is how different tools vary so depending where your priorities lie you should be mindful when selecting a tool.

Using a demo website I trialed one of these tools and the results were consistent with the previous findings.  The site with the A/B testing tool is shown on the bottom of the following filmstrip.

Conclusions

Looking at these tools in more detail it was clear they definitely introduce performance delays so if you do decide to use one what should you consider?

A/B testing v’s Performance
What cost are you introducing to find out this information and is it worth it? If you find a solution that increases conversion by 1% but conversion falls 7% by introducing a performance delay is it worth it?

Server side AB testing
There are server side alternatives that route traffic at the server, proxy or load balancer that don’t introduce the same delays.  Investigate these.

Turn off when not using
Leaving a tag on a page with no live experiments still introduces delay.

Keep experiments short
1 week is plenty of time to gather data.

Keep experiments fair
Check the performance of each variation, if they differ the results could be skewed.

Only use one A/B tool
Some sites had more than one tool installed, I’m not even sure how you end up measuring that!

Try not to use on Mobile
Mobile performance was worse with these tools, if you can avoid then do.

Beware what else you break
The JavaScript used by these tools may break things on the page … Beware!

The full report can be found here: AB testing Report