“An Empirical Study of Example Forgetting during Deep Neural Network Learning”, Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, Geoffrey J. Gordon2018-12-12 (neural net; similar)⁠:

Inspired by the phenomenon of catastrophic forgetting, we investigate the learning dynamics of neural networks as they train on single classification tasks.

Our goal is to understand whether a related phenomenon occurs when data does not undergo a clear distributional shift. We define a ‘forgetting event’ to have occurred when an individual training example transitions from being classified correctly to incorrectly over the course of learning.

Across several benchmark data sets, we find that: (1) certain examples are forgotten with high frequency, and some not at all; (2) a data set’s (un)forgettable examples generalize across neural architectures; and (3) based on forgetting dynamics, a large fraction of examples can be omitted from the training data set while still maintaining state-of-the-art generalization performance.

Similar Links: