“Memorization Without Generalization in a Multilayered Neural Network”, D. Hansel, G. Mato, C. Meunier1992 (neural net)⁠:

The supervised learning of a rule that can be realized by a multilayer neural network (the teacher) functioning as a parity machine with K = 2 hidden units and non-overlapping receptive fields is studied. The student network is supposed to have the same architecture as the teacher.

Application of statistical mechanics shows that when the number of examples is smaller than a critical value P_✱, the trained network is unable to generalize the rule from the examples [phase transition].

Numerical simulations exhibiting this phenomenon are discussed.

See Also:

• Learning and generalization in a two-layer neural network: The role of the Vapnik-Chervonvenkis dimension

• Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior

• Learning through atypical “phase transitions” in overparameterized neural networks

• Statistical mechanics of learning from examples

• The statistical mechanics of learning a rule

• On the ability of the optimal perceptron to generalize

• In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning