“Candy Japan’s New Box A/B Test”, Gwern2016-05-06 (Haskell, R, design, ads, RL exploration, Bayes, power analysis, tutorial; backlinks)⁠:

Bayesian decision-theoretic analysis of the effect of fancier packaging on subscription cancellations & optimal experiment design.

I analyze an A/B test from a mail-order company of two different kinds of box packaging from a Bayesian decision-theory perspective, balancing posterior probability of improvements & greater profit against the cost of packaging & risk of worse results, finding that as the company’s analysis suggested, the new box is unlikely to be sufficiently better than the old. Calculating expected values of information shows that it is not worth experimenting on further, and that such fixed-sample trials are unlikely to ever be cost-effective for packaging improvements. However, adaptive experiments may be worthwhile.

• New Box A/B Test
• Analysis
  • NHST
  • Bayesian
    • Uninformative Priors
    • Informative Priors
• Decision
  • Benefit
  • Cost-Benefit
  • Value of Information
    • Expected Value of Perfect Information (EVPI)
    • Expected Value of Sample Information (EVSI)
      • Sampling to a Foregone Conclusion
  • Adaptive Trials
    • Decision Tree
    • Multi-Armed Bandits
• Conclusion
• See Also

Backlinks: