Most organizations run tests but few tests advance organizational objectives. Why is this? The past three posts discussed the core challenges within traditional A|B testing, each of which contributes inefficiency. We built the Tozan platform to solve each of these challenges. But at a higher level, there is significant pre-planning necessary to build useful experiments, whether it’s a traditional A|B test or a Tozan experiment — 1) figuring out where to run an experiment and 2) identifying the right Key Performance Indicator. Let’s consider an example for an online media subscription service, like an HBO Now or Netflix.
Where to Experiment
Let’s assume that the company has decided to focus on growing its subscription base over the next 12 months. There are two primary sources of subscription growth: retaining previous subscriptions and acquiring new ones. We can dissect some of the example factors that could influence these two subscription pools. (In reality, the below list should be significantly longer).
The cleanest experiments occur within the factors over which the company has direct control (labeled with an X). Factors such as online advertising campaigns might appear controllable but are ultimately subject to advertising market conditions. Likewise, available content is a function of the licensable content market, so the ability to secure a given piece is not entirely controllable and hence subject to noise. The best place to start a test is within an independent factor over which the company can exercise significant independent control, like the subscription trial length.
Selecting a Key Performance Indicator
Now that we have identified the factor that we want to test, we need to identify the KPI around which to hinge our experiment. Free trial length for a subcription would likely impact three metrics:
- Trial take rate — more prospective subscribers likely come through the door if the trial length is extended.
- Trial conversion rate — a longer trial might bring in a lower quality/less interested user
- Retention rate — it is possible that the monthly retention rate will go down
Because our objective is to grow subscriptions, we should focus on net subscription growth rolling n months into the future. Here’s an example of the aggregated experiment results we’d look for:
Perhaps we opt to use 3 month mark as KPI. However, that would mean that we have to wait 3 months before making a decision (either through Tozan automated optimization or through a traditional test). It would be preferable to identify a good proxy for that KPI, even if there is variance around it. That would enable us to optimize it near term and reap the benefits longer term. The challenge with this last step of identifying a proxy is that we have to make assumptions without having seen experiment data. Perhaps the [Pay first month]/[Trial signups] rate is the best proxy, but it’s also possible that it wont be correlated to the three month subscription rate. Investigating a historical correlation matrix should reveal which metrics most closely align to our objective KPI, and we should select the most correlated metric to serve as our proxy KPI. In doing so, we can optimize our longer range KPI but more quickly.
This example highlights the type of thinking that companies must apply to identify the best product areas to test and the most relevant metric to use for evaluation. It is a rigorous process, requiring both logical analysis and intuition but it is a critical prerequisite to derive value from experimentation.
