Tozan Experiment Simulation

We recently built an experiment simulation to showcase some of the most important attributes about the Tozan platform. You can find the simulation here:

Tozan-LLC

As we’ve mentioned in previous posts, Tozan significantly changes and improves the process of experimentation by 1) significantly reducing measurable waste generated from experiments 2) enabling significantly more versions to be tested over a wider/endless time horizon due to the Always-On nature of the platform connection and 3) enabling the experimentation process to be continuous. The simulation above really showcases the first point above all else.

The simulation prompts users to build UIs and then test them:

upon dropping into the example, you’ll see a mock product shelf and an initial version constructed

Users then add more versions — choose any combination of up to five products , and make any number of versions.

You’ll see that as you add versions, the starting allocation of the experiment automatically changes. In this example, each version will start at an allocation of 33.3%

When you’re done adding versions, hit ‘Run Tozan,’ and the browser is calling our simulation in real time. Actually, each time a user builds products, the app is generating a new target version, or optimal version, from the set of 12 available products. Versions are not created equal.

There are three charts on the right hand panel. The first chart, on top, shows how much Tozan is extracting from each version. In this example, Tozan has identified that the second version is stronger and exploits that version quite heavily. The second chart, on bottom above, shows how Tozan is adjusting the version allocations over time — the experiment starts at 33.3–33.3–33.3, goes through a period of variability in which Tozan explores the versions, and then begins to favor the second version. In the real world, once we see a steadily favored version, as version 2 is above, we would terminate the other two versions and consider launching yet more versions into production to compete with version 2.

The third chart showcases the incremental value between Tozan (allocation optimized) and what would have happened in the event of an AB test

The third chart on the bottom showcases the measurable KPI value that Tozan saves by optimizing between versions. It is the cumulative area between the two lines above. You can see above that there is a large cumulative impact.

One important note: above is the impact for a single experiment run. The natural process, because Tozan is Always-On, is to launch new versions into production, which then cycles through the same process of optimization pictures above. This iterative and continuous process generates better versions faster and it is the sum of the cumulative incremental improvements over the lifecycle of a product that generates Tozan’s measurable value.

Go ahead and play with the simulation — we built it to be fun and engaging. Let us know any questions.