“A/B Testing Long-Form Readability on Gwern.net § Covariate Impact On Power”, Gwern2012-06-16 (CSS, JS, R, typography, ads, meta, decision theory, power analysis, Google)⁠:

A log of experiments done on the site design, intended to render pages more readable, focusing on the challenge of testing a static site, page width, fonts, plugins, and effects of advertising.

Is it important in randomized testing of A/B versions of websites to control for covariates, even powerful ones? A simulation using a website’s data suggests that data is sufficiently large that it is not critical the way it is in many applications.

• Background
• Problems With “Conversion” Metric
• Ideas For Testing
• Testing
  • max-width
  • TODO
• Resumption: ABalytics
  • max-width Redux
    • Implementation
    • Results
    • Analysis
  • Fonts
    • Power Analysis
    • Implementation
    • Results
    • Analysis
  • Line Height
    • Implementation
    • Analysis
  • Null Test
    • Results
    • Analysis
  • Text & Background Color
    • Implementation
    • Data
    • Analysis
  • List Symbol And Font-Size
    • Implementation
    • Data
    • Analysis
  • Blockquote Formatting
    • Implementation
    • Data
    • Analysis
  • Font Size & ToC Background
    • Implementation
    • Analysis
    • Multifactorial Roundup
  • Section Header Capitalization
  • ToC Formatting
  • BeeLine Reader Text Highlighting
    • Setup
    • Data
    • Analysis
    • Conclusion
  • Floating Footnotes
    • Implementation
    • Data
    • Analysis
  • Indented Paragraphs
    • Implementation
    • Data
    • Analysis
  • Sidebar Elements
    • Implementation
    • Data
    • Analysis
  • Moving Sidebar Metadata Into Page
    • Implementation
    • Data
    • Analysis
  • CSE
  • Banner Ad Effect on Total Traffic
• Deep Reinforcement Learning
  • Training A Neural Net To Generate CSS
    • char-rnn
    • GPU vs CPU
    • EC2
      • CSS
    • Evaluation
    • External Links
  • RNN: CSS → HTML
    • Creating a Corpus
      • Personal
      • HN
      • CSS Zen Garden
      • Downloading
      • Data Augmentation
• Indentation + Left-Justified Text
• Appendix
  • Covariate Impact On Power
    • Power Simulation
      • Large N
      • Small N
      • Larger Differences
    • Sample Size Implication
      • Gwern.net