In this post we’ll cover the last of the three major challenges that organizations face with traditional A|B testing. We refer to this as the Moving Target problem. The objective of an A|B test is to converge on an answer that leads to an action step. However, in most cases, this expectation is at odds with the fact that the product and market are in a state of continual evolution. How can we expect to impose a single, fixed conclusion onto a background of constant change?
Most A|B tests are run over a fixed time period. The typical processes goes something like this: 1) business identifies some critical product area for experimentation 2) product/marketing teams offer ideas on competing variants 3) analyst teams operationalize the test and execute it, including technical details around sample and time horizon 4) test concludes and analysts measure results 5) next steps: either setting the winning version live or circle back into more testing. Markets are fast moving, and by the time the company has moved from 1 → 5, there’s often been a sufficient change in product or market conditions to render the outcome less relevant. Obvious manifestations of this problem occur when running tests around holidays or tent pole events, but more subtle changes might occur throughout the year. For example, marketing teams might try new tactics that import different types of users or visitors, or competitors might release new products that alter the competitive landscape. As a result, the outcome from the test that was launched one or two months ago might not fit the new ecosystem. In general, stagnant test outcomes don’t work in an environment undergoing fast and unpredictable changes.
Tozan’s solution to this problem is baked into how we define experiments — Always-On. Once an experiment has been created, it becomes part of the product and a core means by which the product evolves. Tozan’s architecture was built to support an Always-On integration, but the technical solution is really only half the battle. The other half is a paradigm change about how to view A|B testing, overall. As testing transforms from a discrete activity to a continuous process, the nature of engagement around tests also changes. The mass of activity around testing shifts towards variant creation, since the main task of the teams involved is to put thoughtful variants into production among which Tozan will optimize.
