“‘Grokking (NN)’ Tag”,2020-03-04

Bibliography for tag ai/scaling/emergence/grokking, most recent first: 40 annotations & 27 links (parent).

• See Also
• Gwern
  • “Hardware Hedging Against Scaling Regime Shifts”, Gwern2024
• Links
  • “Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond”, Jeffares et al 2024
  • “The Slingshot Helps With Learning”, Wu2024
  • “Emergent Properties With Repeated Examples”, Charton & Kempe2024
  • “Grokking Modular Polynomials”, Doshi et al 2024
  • “Learning to Grok: Emergence of In-Context Learning and Skill Composition in Modular Arithmetic Tasks”, He et al 2024
  • “Grokfast: Accelerated Grokking by Amplifying Slow Gradients”, Lee et al 2024
  • “Deep Grokking: Would Deep Neural Networks Generalize Better?”, Fan et al 2024
  • “Grokked Transformers Are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization”, Wang et al 2024
  • “Unified View of Grokking, Double Descent and Emergent Abilities: A Perspective from Circuits Competition”, Huang et al 2024
  • “A Tale of Tails: Model Collapse As a Change of Scaling Laws”, Dohmatob et al 2024
  • “Critical Data Size of Language Models from a Grokking Perspective”, Zhu et al 2024
  • “Grokking Group Multiplication With Cosets”, Stander et al 2023
  • “Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokking”, Lyu et al 2023
  • “Outliers With Opposing Signals Have an Outsized Effect on Neural Network Optimization”, Rosenfeld & Risteski2023
  • “Grokking Beyond Neural Networks: An Empirical Exploration With Model Complexity”, Miller et al 2023
  • “Grokking in Linear Estimators—A Solvable Model That Groks without Understanding”, Levi et al 2023
  • “To Grok or Not to Grok: Disentangling Generalization and Memorization on Corrupted Algorithmic Datasets”, Doshi et al 2023
  • “Grokking As the Transition from Lazy to Rich Training Dynamics”, Kumar et al 2023
  • “PassUntil: Predicting Emergent Abilities With Infinite Resolution Evaluation”, Hu et al 2023
  • “Explaining Grokking through Circuit Efficiency”, Varma et al 2023
  • “Latent State Models of Training Dynamics”, Hu et al 2023
  • “The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks”, Zhong et al 2023
  • “Predicting Grokking Long Before It Happens: A Look into the Loss Landscape of Models Which Grok”, Notsawo et al 2023
  • “A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations”, Chughtai et al 2023
  • “Progress Measures for Grokking via Mechanistic Interpretability”, Nanda et al 2023
  • “Grokking Phase Transitions in Learning Local Rules With Gradient Descent”, Žunkovič & Ilievski2022
  • “Omnigrok: Grokking Beyond Algorithmic Data”, Liu et al 2022
  • “The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon”, Thilak et al 2022
  • “Towards Understanding Grokking: An Effective Theory of Representation Learning”, Liu et al 2022
  • “Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets [Paper]”, Power et al 2022
  • “Learning through Atypical “Phase Transitions” in Overparameterized Neural Networks”, Baldassi et al 2021
  • “Knowledge Distillation: A Good Teacher Is Patient and Consistent”, Beyer et al 2021
  • “Grokking: Generalization Beyond Overfitting On Small Algorithmic Datasets”, Power et al 2021
  • “The Large Learning Rate Phase of Deep Learning: the Catapult Mechanism”, Lewkowycz et al 2020
  • “A Recipe for Training Neural Networks”, Karpathy2019
  • “Sea-Snell/grokking: Unofficial Re-Implementation of “Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets””
  • “Openai/grok”
  • “Teddykoker/grokking: PyTorch Implementation of “Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets””
  • “Hypothesis: Gradient Descent Prefers General Circuits”
  • “Grokking: Generalization beyond Overfitting on Small Algorithmic Datasets (Paper Explained)”
  • Sort By Magic
    • phase-transition emergence dynamics optimization learning-mechanisms adaptive-learning
    • grokking-theory learning-dynamics representation-learning circuit-efficiency generalization
    • grokking-ai
• Miscellaneous
• Bibliography