[code] We aim to understand grokking, a phenomenon where models generalize long after overfitting their training set.
We present both a âmicroscopicâ analysis anchored by an effective theory and a âmacroscopicâ analysis of phase diagrams describing learning performance across hyperparameters.
We find that generalization originates from structured representations whose training dynamics and dependence on training set size can be predicted by our effective theory in a toy setting.
We observe empirically the presence of four learning phases: comprehension, grokking, memorization, and confusion.
We find representation learning to occur only in a âGoldilocks zoneâ (including comprehension and grokking) between memorization and confusion. Compared to the comprehension phase, the grokking phase stays closer to the memorization phase, leading to delayed generalization.
The Goldilocks phase is reminiscent of âintelligence from starvationâ in Darwinian evolution, where resource limitations drive discovery of more efficient solutions.
This study not only provides intuitive explanations of the origin of grokking, but also highlights the usefulness of physics-inspired tools, eg. effective theories and phase diagrams, for understanding deep learning.
âŚUniversality of phase diagrams: We fix the embedding learning rate to be 10â3 and sweep instead decoder weight decay in Figure 6bâd. The phase diagrams correspond to addition regression (b), addition classification (c) and permutation regression (d), respectively.
Common phenomena emerge from these different tasks: (1) they all include 4 phases; (2) The top right corner (a fast and capable decoder) is the memorization phase; (3) the bottom right corner (a fast and simple decoder) is the confusion phase; (4) grokking is sandwiched between comprehension and memorization, which seems to imply that it is an undesirable phase that stems from improperly tuned hyperparameters.
