“Roosevelt Predicted to Win: Revisiting the 1936 Literary Digest Poll”, 2017-03-31 (; similar):
The Literary Digest poll of 1936, which incorrectly predicted that Landon would defeat Roosevelt in the 1936 US presidential election, has long been held up as an example of how not to sample.
The sampling frame was constructed from telephone directories and automobile registration lists, and the survey had a 24% response rate. But if information collected by the poll about votes cast in 1932 had been used to weight the results, the poll would have predicted a majority of electoral votes for Roosevelt in 1936, and thus would have correctly predicted the winner of the election.
We explore alternative weighting methods for the 1936 poll and the models that support them. While weighting would have resulted in Roosevelt being projected as the winner, the bias in the estimates is still very large.
We discuss implications of these results for today’s low-response rate surveys and how the accuracy of the modeling might be reflected better than current practice.
…After every election in which polls err, numerous commentators publish articles about what went wrong with the polls. Gallup’s (193886ya) commentary was of this type. Deming (198638ya: p. 319) observed that “Dr. George Gallup remarked in a speech one time (after a fiasco) that he made his prediction in advance of the election. Other people, smarter, made their predictions after the election, explaining how it all happened.” In some respects, post hoc explanations view the polling inadequacies as what Deming called a “special cause” attributable to unusual features of that particular election. However, since the outcomes of elections typically differ from the poll estimates by considerably more than sampling error, Deming would argue that this is a system-level, or “common” cause. Our system of assessing the uncertainty of estimates from surveys is inadequate and this needs to be addressed systematically rather than trying to explain what is wrong with a particular outcome. Unfortunately, the main lesson from 1936 has not yet been learned.