“On the Near Impossibility of Measuring the Returns to Advertising”, 2013-05-23 (; backlinks):
[superseded by 2015] Classical theories of the firm assume access to reliable signals to measure the causal impact of choice variables on profit.
For advertising expenditure we show, using 25 online field experiments (representing $3.8$2.82013 million total) with major US retailers and brokerages, that this assumption typically does not hold.
Statistical evidence from the randomized trials is very weak because individual-level sales are incredibly volatile relative to the per capita cost of a campaign—a “small” impact on a noisy dependent variable can generate positive returns. A concise statistical argument shows that the required sample size for an experiment to generate sufficiently informative confidence intervals is typically in excess of 10 million person-weeks.
This also implies that heterogeneity bias (or model misspecification) unaccounted for by observational methods only needs to explain a tiny fraction of the variation in sales to severely bias estimates.
The weak informational feedback means most firms cannot even approach profit maximization.
[Keywords: advertising, field experiments, causal inference, electronic commerce, return on investment, information]
…we conditioned on the user level covariates listed in the column labeled by the vector W in Table 1 using several methods to strengthen statistical power; such panel techniques predict and absorb residual variation. Lagged sales are the best predictor and are used wherever possible, reducing variance in the dependent variable by as much as 40%…However, seemingly large improvements in R2 lead to only modest reductions in standard errors. A little math shows that going from R2 = 0 in the univariate regression to R2{|w} = 50% yields a sublinear reduction in standard errors of 29%. Hence, the modeling is as valuable as doubling the sample—a substantial improvement, but one that does not materially change the measurement difficulty. An order-of-magnitude reduction in standard errors would require R2{|w} = 99%, perhaps a “nearly impossible” goal.