In this post we’ll expand upon the first core challenge within A|B testing mentioned in the last post. Testing waste creation is measurable and avoidable. In fact, one of the central reasons why we built Tozan was to eliminate testing waste. Waste from A|B testing is harmful to business outcomes and also completely unnecessary now that Tozan can provide a waste-free solution to any business.
Let’s review a simple example of a typical A|B test:
An online retailer wants to see whether showcasing a different product set on the main landing page will result in higher conversions. The retailer wants to showcase four new products. Because V1 has been live and generating revenue for quite some time, the retailer keeps it as the primary product distribution at 90% and tests the variant at 10% of incoming traffic.
At the end of a pre-determined one week test period, the analyst team calculates the results and performs hypothesis testing to ensure that the differences in conversion rate are not likely due to random chance within the sample. V1 has received 100,000 visits and 10,000 conversions (10% conversion), whereas V2 has received 10,000 visits and 2,000 conversions (20% conversion). The common next step would be to launch V2 to all incoming traffic, or at the very least to a much larger distribution. All in all, this example illustrates a very run-of-the-mill and successful A|B test: simple test, clear and definitive result, and straightforward action step. Tests like this one happen everyday across many companies and industries.
However, the retailer expended real value to run this test: 10,000 missed conversions resulting from V1’s lower performance. The 10% conversion rate for V1 was unnecessarily consuming traffic for the duration of the test. Why is it considered an opportunity cost? Because we know that V2 performed better yet still permitted users to flow into what we knew was a lower quality version. Tozan rapidly identifies likely performance ranges from available versions and tunes traffic in real time to capture the most possible business value. While the experiment had a clean and actionable outcome, the typical cadence of running the experiment for a pre-determined time period before taking it down, calculating results, and launching the better version leads to a known quantity of missed opportunity.
The experiment was conducted within a business setting, not an academic laboratory, and Tozan’s priority is always to enable experimentation and revenue optimization simultaneously. The fact is that businesses do not need to sacrifice value for experimentation — Tozan balances versions perfectly to explore available options while achieving the highest possible output from those options. One of our central tenants is that all business activities should be aimed at generating value.
