Often missed part of successful email split test strategy

A mega test for sure, Marketing Experiments took 300 subject lines suggested by their readers, picked what they felt were the best ten and ran a split test.

The table of results is below. Read the original post for details of how it was run but before you do read on here for a quick lesson in email split test strategy not mentioned elsewhere.

Notice that the top three had the same level of performance, within level of confidence. The next five subject lines, lines 4 to 8, also shared the same performance level too.

Had only the top three subject lines been tested there would have been no winner! Had the test been just the five subject lines, lines 4 to 8, there would have been no winner – in five tests.

The crucial lesson is that testing is a game of skill and quantity. A key ingredient to email split test strategy is simply doing enough tests and different treatments to discover a big gain. Doing occasional tests won’t do it.

If you are running A/B subject line tests then be prepared for several tests showing no performance improvement. Don’t get despondent but keep persevering.

As a rule of thumb expect out of ten test treatments for one to make a big shift to the needle, 3 provide some learning and improvement, 3 no winner and 3 under perform (but you’ll still learn).

When you’ve enough data to run multiple tests concurrently then it’s great to do as it accelerates learning and optimisation. However, remember your sample sizes need to be large enough for statistical significance. A 10,000 list split into ten test cells of 1000 each won’t give any valid result.

Stay on the straight and narrow with this split test calculator. to work out sample sizes and check statistical significance of results.