Political scientists increasingly recognize that multiple imputation represents a superior strategy for analyzing missing data to the widely used method of listwise deletion. However, there has been little systematic investigation ofhowmultiple imputation affects existing empirical knowledge in the discipline.
This article presents the first large-scale examination of the empirical effects of substituting multiple imputation for listwise deletion in political science. The examination focuses on research in the major subfield of comparative and international political economy (CIPE) as an illustrative example. Specifically, I use multiple imputation to reanalyze the results of almost every quantitative CIPE study published during a recent five-year period inInternational OrganizationandWorld Politics, two of the leading subfield journals in CIPE.
The outcome is striking: in almost half of the studies, key results “disappear” (by conventional statistical-significance standards) when reanalyzed.
…I argue that, in addition to being highly inefficient, listwise deletion tends to produce biased inferences in CIPE because the pattern of missing values is not completely random. Most notably, poorer and less democratic countries are more likely to have missing data, causing listwise deletion to give rise to a particular selection problem that I call advanced democracy bias. Despite these problems, however, use of listwise deletion remains widespread in CIPE. A review of almost 100 CIPE studies recently published in 5 leading political science journals indicates that 90% continue to employ listwise deletion as their primary missing-data method, while only 5% have switched to multiple imputation. [The review covers all CIPE studies published in the American Political Science Review, the American Journal of Political Science, the British Journal of Political Science, International Organization, and World Politics between July 2007 and July 2012. The remaining 5% of studies employ another ad-hoc technique, such as averaging observed data or substituting zero for missing values. Worryingly, more than 3-quarters of studies—all of which used listwise deletion—were not explicit about how they dealt with missing data.]
Specifically, I use multiple imputation to reanalyze the results of almost every quantitative CIPE study published during a recent 5% period in International Organization and World Politics, two of the leading subfield journals in CIPE.4 The outcome of the reanalysis, previewed in Figure 1, is striking. In almost half of the studies, key results “disappear” when the main statistical analysis is re-estimated using multiply imputed data (shaded portion of bars, corresponding to left y-axis). That is, at least half of the regression coefficients on the key explanatory variable(s) that were previously statistically-significant at the 10% level either cease to be statistically-significant or experience a change in sign; alternatively, in the case of “negative” findings, at least half of the coefficients on the key explanatory variable(s) that were previously non-statistically-significant become statistically-significant (regardless of sign).5 The reanalysis also sheds light on the considerable scale of the missing-data problem in CIPE: an average of 48% of eligible observations are excluded from the main analysis due to listwise deletion (hollow circles, corresponding to right y-axis), resulting in the loss of 43% of available observed data (solid circles).
In addition to challenging the results of a number of prominent recent studies in CIPE, the article’s findings have important implications for quantitative work in other areas of political science, many of which are likely to be similarly ill suited to listwise deletion and have paid equally little attention to missing-data issues. In the concluding section, I offer some brief speculations on whether and how substituting multiple imputation for listwise deletion might affect empirical knowledge in different subfields.