Analyzing the Test
By now you either have a successful, a failed, or a null test result. After you’ ve concluded the test, you can dig into the data to analyze what happened during the test period and determine your next steps. To analyze your split-testing data, follow these steps:
1. Report all your findings.
Collect and put your testing data into words. You can use a test report sheet or PowerPoint deck for this. Considering breaking your report into the following sections:
Slide 1: Test title, URL, timeline, and metric( s) measured Slide 2: Hypothesis Slide 3: All the variants you tested Slide 4: In-depth results
Slide 5: Results showcasing the winning variant, conversion lift, and confidence rate
Slide 6: Analysis Slide 7: Other observations Slide 8: Recommendations
2. Report your conversion range.
The conversion range is the range between the lowest highest possible conversion rate. This range may be written in the form of a formula, as in 30 % lift ± 3 %, or you might say that you expect conversions to be between 27 and 33 percent. Be sure to report your conversion rate as a range. When you report a 40 percent conversion lift, but you really have a range of 35 – 43 percent, you’ re doing yourself a disservice by not properly setting expectations for your results or your recommendations.
Don’ t let your boss or client think that the conversion rate is static. It isn’ t. Set proper expectations by reporting on your conversion rate as a range. Tools such as Visual Website Optimizer create this range for you.
3. Look at each variant’ s heat map.
Observing each variant’ s heat map helps you find new things to optimize and test. Place these finding in the“ Other Observations” section of your report.
4. Analyze key segments in Google Analytics.
Here, you’ re determining whether the test indicates a higher or lower conversion rate for certain types of visitors.
5. Implement the successful variation. Ideally, thanks to the results of your split testing, you know what works. Now you can