‘grokking (NN)’ directory

Annotations sorted by machine learning into inferred 'tags'. This provides an alternative way to browse: instead of by date order, one can browse in topic order. The 'sorted' list has been automatically clustered into multiple sections & auto-labeled for easier browsing.

Beginning with the newest annotation, it uses the embedding of each annotation to attempt to create a list of nearest-neighbor annotations, creating a progression of topics. For more details, see the link.

`teachers-persistence distillation-concept knowledge-optimization distillation-strategies teacher-student model-relationship`

[see previous entry]

Miscellaneous

Bibliography

https://www.lesswrong.com/posts/LncYobrn3vRr7qkZW/the-slingshot-helps-with-learning: “The Slingshot Helps With Learning”, Wilson Wu

link-bibliography
https://arxiv.org/abs/2405.20233: “Grokfast: Accelerated Grokking by Amplifying Slow Gradients”, Jaerin Lee, Bong Gyun Kang, Kihoon Kim, Kyoung Mu Lee

link-bibliography
https://arxiv.org/abs/2405.19454: “Deep Grokking: Would Deep Neural Networks Generalize Better?”, Simin Fan, Razvan Pascanu, Martin Jaggi

link-bibliography
https://arxiv.org/abs/2405.15071: “Grokked Transformers Are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization”, Boshi Wang, Xiang Yue, Yu Su, Huan Sun

link-bibliography
https://arxiv.org/abs/2402.15175: “Unified View of Grokking, Double Descent and Emergent Abilities: A Perspective from Circuits Competition”, Yufei Huang, Shengding Hu, Xu Han, Zhiyuan Liu, Maosong Sun

link-bibliography
https://arxiv.org/abs/2401.10463: “Critical Data Size of Language Models from a Grokking Perspective”, Xuekai Zhu, Yao Fu, Bowen Zhou, Zhouhan Lin

link-bibliography
https://arxiv.org/abs/2310.13061: “To Grok or Not to Grok: Disentangling Generalization and Memorization on Corrupted Algorithmic Datasets”, Darshil Doshi, Aritra Das, Tianyu He, Andrey Gromov

link-bibliography
https://arxiv.org/abs/2310.03262: “PassUntil: Predicting Emergent Abilities With Infinite Resolution Evaluation”, Shengding Hu, Xin Liu, Xu Han, Xinrong Zhang, Chaoqun He, Weilin Zhao, Yankai Lin, Ning Ding, Zebin Ou, Guoyang Zeng, Zhiyuan Liu, Maosong Sun

link-bibliography
https://arxiv.org/abs/2306.13253: “Predicting Grokking Long Before It Happens: A Look into the Loss Landscape of Models Which Grok”, Pascal Junior Tikeng Notsawo, Hattie Zhou, Mohammad Pezeshki, Irina Rish, Guillaume Dumas

link-bibliography
https://arxiv.org/abs/2301.05217: “Progress Measures for Grokking via Mechanistic Interpretability”, Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, Jacob Steinhardt

link-bibliography
https://arxiv.org/abs/2210.15435: “Grokking Phase Transitions in Learning Local Rules With Gradient Descent”, Bojan Žunkovič, Enej Ilievski

link-bibliography
https://arxiv.org/abs/2210.01117: “Omnigrok: Grokking Beyond Algorithmic Data”, Ziming Liu, Eric J. Michaud, Max Tegmark

link-bibliography
https://arxiv.org/abs/2205.10343: “Towards Understanding Grokking: An Effective Theory of Representation Learning”, Ziming Liu, Ouail Kitouni, Niklas Nolte, Eric J. Michaud, Max Tegmark, Mike Williams

link-bibliography
https://arxiv.org/abs/2110.00683: “Learning through Atypical "Phase Transitions" in Overparameterized Neural Networks”, Carlo Baldassi, Clarissa Lauditi, Enrico M. Malatesta, Rosalba Pacelli, Gabriele Perugini, Riccardo Zecchina

link-bibliography
https://arxiv.org/abs/2106.05237#google: “Knowledge Distillation: A Good Teacher Is Patient and Consistent”, Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, Alexander Kolesnikov

link-bibliography
2021-power.pdf#openai: “Grokking: Generalization Beyond Overfitting On Small Algorithmic Datasets”, Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, Vedant Misra

link-bibliography
https://karpathy.github.io/2019/04/25/recipe/: “A Recipe for Training Neural Networks”, Andrej Karpathy

link-bibliography