“The InterModel Vigorish (IMV): A Flexible and Portable Approach for Quantifying Predictive Accuracy With Binary Outcomes”, 2022-01-12 (; similar):
[Twitter; app] Understanding the “fit” of models designed to predict binary outcomes has been a long-standing problem.
We propose a flexible, portable, and intuitive metric for quantifying the change in accuracy between 2 predictive systems in the case of a binary outcome, the InterModel Vigorish (IMV). The IMV is based on an analogy to well-characterized physical systems with tractable probabilities: weighted coins. The IMV is always a statement about the change in fit relative to some baseline—which can be as simple as the prevalence—whereas other metrics are stand-alone measures that need to be further manipulated to yield indices related to differences in fit across models. Moreover, the IMV is consistently interpretable independent of baseline prevalence.
We illustrate the flexible properties of this metric in numerous simulations and showcase its flexibility across examples spanning the social, biomedical, and physical sciences.
[Keywords: binary outcomes, fit index, logistic regression, prediction, Kelly criterion, entropy, coherence]