Machine Learning Scaling

• ⁠The Blessings of Scale⁠: when ⁠more is different⁠

• “Do Deep Convolutional Nets Really Need to be Deep and Convolutional?”⁠, Urbanet al2016 (negative result, particularly ⁠on scaling⁠—wrong, but why?)

• “Revisiting Unreasonable Effectiveness of Data in Deep Learning Era”⁠, Sunet al2017

• “Deep Learning Scaling is Predictable, Empirically”⁠, Hestnesset al2017

• “Learning Visual Features from Large Weakly Supervised Data”⁠, Joulinet al2015; “Exploring the Limits of Weakly Supervised Pretraining”⁠, Mahajanet al2018; “Revisiting Weakly Supervised Pre-Training of Visual Perception Models”⁠, Singhet al2022 (CNNs⁠ scale to billions of hashtagged⁠ Instagram images)

• WebVision: “WebVision Challenge: Visual Learning and Understanding With Web Data”⁠, Liet al2017a/“WebVision Database: Visual Learning and Understanding from Web Data”⁠, Liet al2017b/“CurriculumNet: Weakly Supervised Learning from Large-Scale Web Images”⁠, Guoet al2018

• “Measuring the Effects of Data Parallelism on Neural Network Training”⁠, Shallueet al2018

• “Gradient Noise Scale: An Empirical Model of Large-Batch Training”⁠, McCandlishet al2018

• “A Constructive Prediction of the Generalization Error Across Scales”⁠, Rosenfeldet al2019

• “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks”⁠, Tan & Le2019

• “One Epoch Is All You Need”⁠, Komatsuzaki2019

• “Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers”⁠, Liet al2020

• “Small Data, Big Decisions: Model Selection in the Small-Data Regime”⁠, Bornscheinet al2020

• Key GPT papers:

  • “Scaling Laws for Neural Language Models”⁠, Kaplanet al2020

  • “Scaling Laws from the Data Manifold Dimension”⁠, Sharma & Kaplan2020

  • “Scaling Laws for Autoregressive Generative Modeling”⁠, Henighanet al2020 (noise & resolution); “Broken Neural Scaling Laws”⁠, Caballeroet al2022

  • “GPT-3: Language Models are Few-Shot Learners”⁠, Brownet al2020

  • “Measuring Massive Multitask Language Understanding”⁠, Hendryckset al2020; “Measuring Mathematical Problem Solving With the MATH Dataset”⁠, Hendryckset al2021

  • “Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers”⁠, Hendrickset al2021

  • “Scaling Laws for Transfer”⁠, Hernandezet al2021; “Scaling Laws for Language Transfer Learning”⁠, Christina Kim (Hernandezet al2021 followup: smooth scaling for En → De/Es/Zh); “When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method”⁠, Zhanget al2024

  • “Scaling Laws for Neural Machine Translation”⁠, Ghorbaniet al2021; “Data and Parameter Scaling Laws for Neural Machine Translation”⁠, Gordonet al2021; “Unsupervised Neural Machine Translation with Generative Language Models Only”⁠, Hanet al2021; “Data Scaling Laws in NMT: The Effect of Noise and Architecture”⁠, Bansalet al2022

  • “How Many Data Points is a Prompt Worth?”⁠, Le Scao & Rush2021

  • “Recursively Summarizing Books with Human Feedback”⁠, Wuet al2021

  • “Codex: Evaluating Large Language Models Trained on Code”⁠, Chenet al2021 (small versions of GitHub Copilot⁠, solves simple linear algebra⁠/statistics problems⁠ too); “Program Synthesis with Large Language Models”⁠, Austinet al2021; “Show Your Work: Scratchpads for Intermediate Computation with Language Models”⁠, Anonymouset al2021; “Few-Shot Self-Rationalization with Natural Language Prompts”⁠, Marasovićet al2021

  • “Scarecrow: A Framework for Scrutinizing Machine Text”⁠, Douet al2021

  • “A Recipe For Arbitrary Text Style Transfer with Large Language Models”⁠, Reifet al2021

  • Instruction tuning⁠/multi-task finetuning

  • “M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining”⁠, Linet al2021

  • “Training Verifiers to Solve Math Word Problems”⁠, Cobbeet al2021

  • “Symbolic Knowledge Distillation: from General Language Models to Commonsense Models”⁠, Westet al2021

  • “An Explanation of In-Context Learning as Implicit Bayesian Inference”⁠, Xieet al2021

• “Blender: Recipes for building an open-domain chatbot”⁠, Rolleret al2020

• “Big Self-Supervised Models are Strong Semi-Supervised Learners”⁠, Chenet al2020a

• “iGPT: Generative Pretraining from Pixels”⁠, Chenet al2020b

• “GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding”⁠, Lepikhinet al2020; “Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity”⁠, Feduset al2021; “Exploring Sparse Expert Models and Beyond”⁠, Yanget al2021

  • “On the Predictability of Pruning Across Scales”⁠, Rosenfeldet al2020 (scaling laws for sparsity⁠: initially large size reductions are free, then power-law⁠ worsening, then plateau at tiny but bad models)

• “How big should my language model be?”⁠, Huggingface2020

• “When Do You Need Billions of Words of Pretraining Data?”⁠, Zhanget al2020; “Learning Which Features Matter: RoBERTa Acquires a Preference forLinguistic Generalizations (Eventually)”⁠, Warstadtet al2020; “Probing Across Time: What Does RoBERTa Know and When?”⁠, Liuet al2021

• CLIP⁠; “ALIGN: Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision”⁠, Jiaet al2021 (see also CC-12M⁠; EfficientNet trained on 1.8 billion images on a TPUv3-1024⁠); “WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training”⁠, Huoet al2021; “Multimodal Few-Shot Learning with Frozen Language Models”⁠, Tsimpoukelliet al2021; “GrokNet: Unified Computer Vision Model Trunk and Embeddings For Commerce”⁠, Bellet al2020; “Billion-Scale (Pinterest) Pretraining with Vision Transformers for Multi-Task Visual Representations”⁠, Bealeet al2021

• “DALL·E 1: Zero-Shot Text-to-Image Generation”⁠, Rameshet al2021 (blog⁠); “M6: A Chinese Multimodal Pretrainer”⁠, Linet al2021 (Chinese DALL·E 1: 1.9TB images/0.29TB text for 10b-parameter dense/100b-parameter MoE Transformer; shockingly fast Chinese replication of DALL·E 1/CLIP)

• “Improved Denoising Diffusion Probabilistic Models”⁠, Nichol & Dhariwal2021 (DDPM⁠ scaling laws for FID⁠ & likelihood)

• “Automatic Curation of Large-Scale Datasets for Audio-Visual Representation Learning”⁠, Leeet al2021

• “Scaling Laws for Acoustic Models”⁠, Droppo & Elibol2021

  • “XLSR: Unsupervised Cross-lingual Representation Learning for Speech Recognition”⁠, Conneauet al2020

  • “Scaling End-to-End Models for Large-Scale Multilingual ASR”⁠, Liet al2021; “Scaling ASR Improves Zero and Few Shot Learning”⁠, Xiaoet al2021

  • “VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation”⁠, Wanget al2021; “wav2vec: Large-Scale Self-Supervised and Semi-Supervised Learning for Speech Translation”⁠, Wanget al2021 (fMRI⁠); “XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale”⁠, Babuet al2021

  • “HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units”⁠, Hsuet al2021

  • Whisper⁠

• “SEER: Self-supervised Pretraining of Visual Features in the Wild”⁠, Goyalet al2021; “Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervision”⁠, Goyalet al2022

• “Fast and Accurate Model Scaling”⁠, Dolláret al2021; “Revisiting ResNets: Improved Training and Scaling Strategies”⁠, Belloet al2021

• “XLM-R: Unsupervised Cross-lingual Representation Learning at Scale”⁠, Conneauet al2019; “XLM-R XL/XLM-R XXL: Larger-Scale Transformers for Multilingual Masked Language Modeling”⁠, Goyalet al2021; “Facebook AI WMT21 News Translation Task Submission”⁠, Tranet al2021

• “ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation”⁠, Sunet al2021

• “LEMON: Scaling Up Vision-Language Pre-training for Image Captioning”⁠, Huet al2021

• “Flamingo: a Visual Language Model for Few-Shot Learning”⁠, Alayracet al2022

• “Scaling Vision Transformers”⁠, Zhaiet al2021

• “CoAtNet: Marrying Convolution and Attention for All Data Sizes”⁠, Daiet al2021

• “BEiT: BERT Pre-Training of Image Transformers”⁠, Baoet al2021; “Masked Autoencoders Are Scalable Vision Learners”⁠, Heet al2021

• “A Universal Law of Robustness via Isoperimetry”⁠, Bubeck & Sellke2021; “Exploring the Limits of Out-of-Distribution Detection”⁠, Fortet al2021; “Partial success in closing the gap between human and machine vision”⁠, Geirhoset al2021

• “Effect of scale on catastrophic forgetting in neural networks”⁠, Anonymous2021

• “On the Opportunities and Risks of Foundation Models”⁠, Bommasaniet al2021 (review)

• “Exploring the Limits of Large Scale Pre-training”⁠, Abnaret al2021

• “Scaling Laws for the Few-Shot Adaptation of Pre-trained Image Classifiers”⁠, Pratoet al2021

• “E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials”⁠, Batzneret al2021

• Face recognition: “WebFace260M: A Benchmark for Million-Scale Deep Face Recognition”⁠, Zhuet al2022

• “Fine-tuned Language Models are Continual Learners”⁠, Scialom at al 2022

• Embeddings: “DynamicEmbedding: Extending TensorFlow for Colossal-Scale Applications”⁠, Zenget al2020; “DLRM: High-performance, Distributed Training of Large-scale Deep Learning Recommendation Models”⁠, Mudigereet al2021; ⁠“Make Every feature Binary (MEB): A 135b-parameter sparse neural network for massively improved search relevance”⁠; “Persia: A Hybrid System Scaling Deep Learning Based Recommenders up to 100 Trillion Parameters”⁠, Lianet al2021 (Kuaisho⁠)

  • “Scaling Law for Recommendation Models: Towards General-purpose User Representations”⁠, Shinet al2021; “Understanding Scaling Laws for Recommendation Models”⁠, Ardalaniet al2022

• MLPs/FCs: from the “Fully-Connected Neural Nets” bibliography⁠: ⁠Urbanet al2016⁠; “MLP-Mixer: An all-MLP Architecture for Vision”⁠, Tolstikhinet al2021; “gMLP: Pay Attention to MLPs”⁠, Liuet al2021

• Reinforcement Learning:

  • “Fine-Tuning Language Models from Human Preferences”⁠, Ziegleret al2019; “Learning to summarize from human feedback”⁠, Stiennonet al2020

  • “Measuring hardware overhang”⁠, hippke (the curves cross: “with today’s [trained] algorithms, computers would have beat the world chess champion already in 1994_31ya on a contemporary desk computer”)

  • “Scaling Scaling Laws with Board Games”⁠, Jones2021 (AlphaZero⁠/Hex⁠: ⁠highly-optimized⁠ GPU implementation enables showing smooth scaling across 6 OOM of compute—2× FLOPS = 66% victory; amortization of training → runtime tree-search, where 10× training = 15× runtime)

  • “MuZero Unplugged: Online and Offline Reinforcement Learning by Planning with a Learned Model”⁠, Schrittwieseret al2021

  • “From Motor Control to Team Play in Simulated Humanoid Football”⁠, Liuet al2021

  • “Open-Ended Learning Leads to Generally Capable Agents”⁠, Open Ended Learning Teamet al2021; “Procedural Generalization by Planning with Self-Supervised World Models”⁠, Anandet al2021

  • “Fictitious Co-Play: Collaborating with Humans without Human Data”⁠, Strouseet al2021

  • “Gato: A Generalist Agent”⁠, Reedet al2022 (small Decision Transformer⁠ can learn >500 tasks; scaling smoothly)

  • “Multi-Game Decision Transformers”⁠, Leeet al2022 (near-human offline single-checkpoint ALE agent with scaling & rapid transfer)

• Theory:

  • “Does Learning Require Memorization? A Short Tale about a Long Tail”⁠, Feldman2019

  • “Generalization bounds for deep learning”⁠, Valle-Pérez & Louis2020

  • “The Deep Bootstrap Framework: Good Online Learners are Good Offline Generalizers”⁠, Nakkiranet al2020

  • “Explaining Neural Scaling Laws”⁠, Bahriet al2021

  • “Learning Curve Theory”⁠, Hutter2021 (⁠Rohin Shah commentary⁠; more on the manifold hypothesis)

  • “The Shape of Learning Curves: a Review”⁠, Viering & Loog2021

  • “A mathematical theory of semantic development in deep neural networks”⁠, Saxeet al2019 (are jumps in NN capabilities to be expected when scaling? see also ⁠Viering & Loog2021’s⁠ discussion of phase transitions & averaging of exponentials giving power-laws, human “vocabulary spurts”⁠, and “Acquisition of Chess Knowledge in AlphaZero”, McGrathet al2021⁠ ⁠§6 “Rapid increase of basic knowledge”⁠); sequential learning in OpenFold⁠

  • “A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning”⁠, Daret al2021

• Historical:

  • “Toward A Universal Law Of Generalization For Psychological Science”⁠, Shepard1987

  • “Scaling to Very Very Large Corpora for Natural Language Disambiguation”⁠, Banko & Brill2001

  • ⁠“Large Scale Online Learning”⁠, Bottou & LeCun2003 (“We argue that suitably designed online learning algorithms asymptotically outperform any batch learning algorithm.”)

  • “Tree Induction vs. Logistic Regression: A Learning-Curve Analysis”⁠, Perlichet al2003

  • “Large Language Models in Machine Translation”⁠, Brantset al2007; Koehn & Knowles2017⁠ (⁠Figure 3⁠)

  • “The Unreasonable Effectiveness of Data”⁠, Halevyet al2009

  • “The Tradeoffs of Large-Scale Learning”⁠, Bottou & Bousquet2007/2012; ⁠“Large-Scale Machine Learning Revisited [slides]”⁠, Bottou2013

• See Also: For more ML scaling research, follow the /r/MLScaling⁠ subreddit; “It Looks Like You’re Trying To Take Over The World”⁠

⁠null⁠