Suppose you are running a campaign to your website landing page. Your team has just made a significant change to a landing page that is supposed to drastically improve your conversion rate. Everyone is wondering about the impact, and your statistical web marketing analyst has pulled some conversion rate data that visually looks like this:
Marketing and Web Initial Impression
Cheers. Hurrah. Things have drastically improved. Everyone gets a raise!
Statistical Web Marketing Analysts Initial Impression
I am confident that the changes made a positive impact, but the data is not conclusive. The time frame before the changes and after are not the same. After is shorter by a few days. There was no F-test that was ran to prove with 95% confidence that the changes provided a positive impact. We have not given enough time for the conversion rate to normalize.
Case for A/B Testing
This example is exactly why an A/B testing would answer your web marketing team questions. A/B testing eliminates the element of Time by dividing traffic to your pages during the same time period. This is vital, because doing it by hand and comparing time frame A to time frame B you are not considering all the changes that are happening such as Social, PR, Search, and Conferences. These data sources impact the mind frame of the consumer coming to your website landing page.
Conclusion
A/B Testing normalizes your data and removes the time element. Your statistical web analyst is correct by stating that the answer is inconclusive. In order for the above example to be evaluated without an A/B testing software, you need to give the post website changes more than 30 days to achieve statistical significance.
Related Posts: Web analytics platforms cannot prove causality, Small businesses need analytics, Case Study: Ecommerce store optimization with Google Analytics