“Datamodels: Predicting Predictions from Training Data”, Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, Aleksander Madry2022-02-01 (NN sparsity, meta-learning; similar)⁠:

We present a conceptual framework, datamodeling, for analyzing the behavior of a model class in terms of the training data.

For any fixed “target” example x, training set S, and learning algorithm, a datamodel is a parameterized function 2^S ⟶ ℝ that for any subset ofS′ ⊂ S—using only information about which examples of S are contained in S′—predicts the outcome of training a model on S′ and evaluating on x. Despite the potential complexity of the underlying process being approximated (eg. end-to-end training and evaluation of deep neural networks), we show that even simple linear datamodels can successfully predict model outputs.

We then demonstrate that datamodels give rise to a variety of applications, such as: accurately predicting the effect of dataset counterfactuals; identifying brittle predictions; finding semantically similar examples; quantifying train-test leakage; and embedding data into a well-behaved and feature-rich representation space.

Data for this paper (including pre-computed datamodels as well as raw predictions from 4 million trained deep neural networks) is available at Github.

Similar Links: