- See Also
-
Links
- “Mamba: Linear-Time Sequence Modeling With Selective State Spaces”, Gu & Dao 2023
- “Instruction-tuning Aligns LLMs to the Human Brain”, Aw et al 2023
- “First Tragedy, Then Parse: History Repeats Itself in the New Era of Large Language Models”, Saphra et al 2023
- “I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models”, Zhang et al 2023
- “Sam Altman Accepts the 2023 Hawking Fellowship Award § Is There Another Breakthrough That’s Needed to Reach AGI?”, Altman 2023
- “A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models”, Eisape et al 2023
- “ConvNets Match Vision Transformers at Scale”, Smith et al 2023
- “PaLI-3 Vision Language Models: Smaller, Faster, Stronger”, Chen et al 2023
- “Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning”, Xia et al 2023
- “Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition”, Chen et al 2023
- “GeoLLM: Extracting Geospatial Knowledge from Large Language Models”, Manvi et al 2023
- “FreshLLMs: Refreshing Large Language Models With Search Engine Augmentation”, Vu et al 2023
- “SeamlessM4T: Massively Multilingual & Multimodal Machine Translation”, Communication et al 2023
- “Simple Synthetic Data Reduces Sycophancy in Large Language Models”, Wei et al 2023
- “Absolute Unit NNs: Regression-Based MLPs for Everything”, Gwern 2023
- “LLaMA 2: Open Foundation and Fine-Tuned Chat Models”, Touvron et al 2023
- “Introducing Superalignment”, Leike & Sutskever 2023
- “Gödel, Escher, Bach Author Douglas Hofstadter on the State of AI Today § What about AI Terrifies You?”, Hofstadter & Kim 2023
- “Pretraining Task Diversity and the Emergence of Non-Bayesian In-context Learning for Regression”, Raventós et al 2023
- “Scaling MLPs: A Tale of Inductive Bias”, Bachmann et al 2023
- “Image Captioners Are Scalable Vision Learners Too”, Tschannen et al 2023
- “PaLI-X: On Scaling up a Multilingual Vision and Language Model”, Chen et al 2023
- “Scaling Data-Constrained Language Models”, Muennighoff et al 2023
- “The False Promise of Imitating Proprietary LLMs”, Gudibande et al 2023
- “Scaling Laws for Language Encoding Models in FMRI”, Antonello et al 2023
- “LIMA: Less Is More for Alignment”, Zhou et al 2023
- “Google's Newest A.I. Model Uses Nearly 5× More Text Data for Training Than Its Predecessor”, Elias 2023
- “TinyStories: How Small Can Language Models Be and Still Speak Coherent English?”, Eldan & Li 2023
- “TorToise: Better Speech Synthesis through Scaling”, Betker 2023
- “ImageBind: One Embedding Space To Bind Them All”, Girdhar et al 2023
- “Finding Neurons in a Haystack: Case Studies With Sparse Probing”, Gurnee et al 2023
- “Emergent and Predictable Memorization in Large Language Models”, Biderman et al 2023
- “Even The Politicians Thought the Open Letter Made No Sense In The Senate Hearing on AI Today's Hearing on Ai Covered Ai Regulation and Challenges, and the Infamous Open Letter, Which Nearly Everyone in the Room Thought Was Unwise”, Gorrell 2023
- “Power Law Trends in Speedrunning and Machine Learning”, Erdil & Sevilla 2023
- “DINOv2: Learning Robust Visual Features without Supervision”, Oquab et al 2023
- “Segment Anything”, Kirillov et al 2023
- “Sigmoid Loss for Language Image Pre-Training”, Zhai et al 2023
- “How Well Do Large Language Models Perform in Arithmetic Tasks?”, Yuan et al 2023
- “GPT-4 Technical Report”, OpenAI 2023
- “Securing Liberal Democratic Control of AGI through UK Leadership”, Phillips 2023
- “Language Is Not All You Need: Aligning Perception With Language Models (Kosmos-1)”, Huang et al 2023
- “Scaling Vision Transformers to 22 Billion Parameters”, Dehghani et al 2023
- “Large Language Models As Fiduciaries: A Case Study Toward Robustly Communicating With Artificial Intelligence Through Legal Standards”, Nay 2023
- “ClimaX: A Foundation Model for Weather and Climate”, Nguyen et al 2023
- “StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis”, Sauer et al 2023
- “MUG: Vision Learners Meet Web Image-Text Pairs”, Zhao et al 2023
- “GPT-3 As Knowledge Worker: A Zero-Shot Evaluation of (AI)CPA Capabilities”, Bommarito et al 2023
- “Scaling Laws for Generative Mixed-Modal Language Models”, Aghajanyan et al 2023
- “VALL-E: Neural Codec Language Models Are Zero-Shot Text to Speech Synthesizers”, Wang et al 2023
- “GPT-3 Takes the Bar Exam”, II & Katz 2022
- “One Embedder, Any Task: Instruction-Finetuned Text Embeddings (INSTRUCTOR)”, Su et al 2022
- “Discovering Language Model Behaviors With Model-Written Evaluations”, Perez et al 2022
- “Reproducible Scaling Laws for Contrastive Language-image Learning”, Cherti et al 2022
- “ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languages”, Chai et al 2022
- “VindLU: A Recipe for Effective Video-and-Language Pretraining”, Cheng et al 2022
- “Video-Text Modeling With Zero-Shot Transfer from Contrastive Captioners”, Yan et al 2022
- “Robust Speech Recognition via Large-Scale Weak Supervision”, Radford et al 2022
- “Scaling Language-Image Pre-training via Masking”, Li et al 2022
- “MultiRay: Optimizing Efficiency for Large-scale AI Models”, Gupta et al 2022
- “Galactica: A Large Language Model for Science”, Taylor et al 2022
- “EVA: Exploring the Limits of Masked Visual Representation Learning at Scale”, Fang et al 2022
- “MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation”, Feng et al 2022
- “Increments Podcast: #45—4 Central Fallacies of AI Research (with Melanie Mitchell)”, Mitchell & Chugg 2022
- “Will We Run out of Data? An Analysis of the Limits of Scaling Datasets in Machine Learning”, Villalobos et al 2022
- “Evaluating Parameter Efficient Learning for Generation”, Xu et al 2022
- “FLAN: Scaling Instruction-Finetuned Language Models”, Chung et al 2022
- “BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining”, Luo et al 2022
- “Vision-Language Pre-training: Basics, Recent Advances, and Future Trends”, Gan et al 2022
- “Foundation Transformers”, Wang et al 2022
- “Self-Ask: Measuring and Narrowing the Compositionality Gap in Language Models (Bamboogle)”, Press et al 2022
- “Ask Me Anything (AMA): A Simple Strategy for Prompting Language Models”, Arora et al 2022
- “GLM-130B: An Open Bilingual Pre-trained Model”, Zeng et al 2022
- “Do Current Multi-Task Optimization Methods in Deep Learning Even Help?”, Xin et al 2022
- “Monolith: Real Time Recommendation System With Collisionless Embedding Table”, Liu et al 2022
- “Machine Reading, Fast and Slow: When Do Models "Understand" Language?”, Choudhury et al 2022
- “PaLI: A Jointly-Scaled Multilingual Language-Image Model”, Chen et al 2022
- “Using Large Language Models to Simulate Multiple Humans”, Aher et al 2022
-
“
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale”, Dettmers et al 2022 - “Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP”, Nguyen et al 2022
- “Efficient Training of Language Models to Fill in the Middle”, Bavarian et al 2022
- “ESMfold: Language Models of Protein Sequences at the Scale of Evolution Enable Accurate Structure Prediction”, Lin et al 2022
- “Why Do Tree-based Models Still Outperform Deep Learning on Tabular Data?”, Grinsztajn et al 2022
- “PIXEL: Language Modelling With Pixels”, Rust et al 2022
- “High-performing Neural Network Models of Visual Cortex Benefit from High Latent Dimensionality”, Elmoznino & Bonner 2022
- “Language Models (Mostly) Know What They Know”, Kadavath et al 2022
- “Exploring Length Generalization in Large Language Models”, Anil et al 2022
- “On-Device Training Under 256KB Memory”, Lin et al 2022
- “Beyond Neural Scaling Laws: Beating Power Law Scaling via Data Pruning”, Sorscher et al 2022
- “ProGen2: Exploring the Boundaries of Protein Language Models”, Nijkamp et al 2022
- “RST: ReStructured Pre-training”, Yuan & Liu 2022
- “Limitations of the NTK for Understanding Generalization in Deep Learning”, Vyas et al 2022
- “Modeling Transformative AI Risks (MTAIR) Project—Summary Report”, Clarke et al 2022
- “BigVGAN: A Universal Neural Vocoder With Large-Scale Training”, Lee et al 2022
- “An Improved One Millisecond Mobile Backbone”, Vasu et al 2022
- “A Neural Corpus Indexer for Document Retrieval”, Wang et al 2022
- “Toward a Realistic Model of Speech Processing in the Brain With Self-supervised Learning”, Millet et al 2022
- “Teaching Models to Express Their Uncertainty in Words”, Lin et al 2022
- “M3AE: Multimodal Masked Autoencoders Learn Transferable Representations”, Geng et al 2022
- “Why Robust Generalization in Deep Learning Is Difficult: Perspective of Expressive Power”, Li et al 2022
- “InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning”, Gupta et al 2022
- “Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models”, Tirumala et al 2022
- “Least-to-Most Prompting Enables Complex Reasoning in Large Language Models”, Zhou et al 2022
- “Towards Understanding Grokking: An Effective Theory of Representation Learning”, Liu et al 2022
- “Continual Pre-Training Mitigates Forgetting in Language and Vision”, Cossu et al 2022
- “Dialog Inpainting: Turning Documents into Dialogues”, Dai et al 2022
- “Unifying Language Learning Paradigms”, Tay et al 2022
- “When Does Dough Become a Bagel? Analyzing the Remaining Mistakes on ImageNet”, Vasudevan et al 2022
- “Building Machine Translation Systems for the Next Thousand Languages”, Bapna et al 2022
- “CoCa: Contrastive Captioners Are Image-Text Foundation Models”, Yu et al 2022
- “Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)”, Fang et al 2022
- “Continual Learning With Foundation Models: An Empirical Study of Latent Replay”, Ostapenko et al 2022
- “Flamingo: a Visual Language Model for Few-Shot Learning”, Alayrac et al 2022
- “WebFace260M: A Benchmark for Million-Scale Deep Face Recognition”, Zhu et al 2022
- “What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?”, Wang et al 2022
- “DeepMind: The Podcast—Excerpts on AGI”, Kiely 2022
- “Can Language Models Learn from Explanations in Context?”, Lampinen et al 2022
- “Chinchilla: Training Compute-Optimal Large Language Models”, Hoffmann et al 2022
- “A Roadmap for Big Model”, Yuan et al 2022
- “A Conversational Paradigm for Program Synthesis”, Nijkamp et al 2022
- “Self-Consistency Improves Chain-of-Thought Reasoning in Language Models”, Wang et al 2022
- “Effect of Scale on Catastrophic Forgetting in Neural Networks”, Ramasesh et al 2022
- “Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer”, Yang et al 2022
- “FastFold: Reducing AlphaFold Training Time from 11 Days to 67 Hours”, Cheng et al 2022
- “Variational Autoencoders Without the Variation”, Daly et al 2022
- “Performance Reserves in Brain-imaging-based Phenotype Prediction”, Schulz et al 2022
- “Self-Distilled StyleGAN: Towards Generation from Internet Photos”, Mokady et al 2022
- “UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Training”, Khashabi et al 2022
- “Brains and Algorithms Partially Converge in Natural Language Processing”, Caucheteux & King 2022
- “Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervision”, Goyal et al 2022
- “Wukong: 100 Million Large-scale Chinese Cross-modal Pre-training Dataset and A Foundation Framework”, Gu et al 2022
- “OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework”, Wang et al 2022
- “Webly Supervised Concept Expansion for General Purpose Vision Models”, Kamath et al 2022
- “Data Scaling Laws in NMT: The Effect of Noise and Architecture”, Bansal et al 2022
- “Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model”, Smith et al 2022
- “Reasoning Like Program Executors”, Pi et al 2022
- “Text and Code Embeddings by Contrastive Pre-Training”, Neelakantan et al 2022
- “SWAG: Revisiting Weakly Supervised Pre-Training of Visual Perception Models”, Singh et al 2022
- “LaMDA: Language Models for Dialog Applications”, Thoppilan et al 2022
- “CM3: A Causal Masked Multimodal Model of the Internet”, Aghajanyan et al 2022
- “ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalization”, Xu et al 2022
- “The Defeat of the Winograd Schema Challenge”, Kocijan et al 2022
- “Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets [paper]”, Power et al 2022
- “AV-HuBERT: Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction”, Shi et al 2022
- “Robust Self-Supervised Audio-Visual Speech Recognition”, Shi et al 2022
- “Self-supervised Learning from 100 Million Medical Images”, Ghesu et al 2022
- “The Evolution of Quantitative Sensitivity”, Bryer et al 2021
- “ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation”, Wang et al 2021
- “XGLM: Few-shot Learning With Multilingual Language Models”, Lin et al 2021
- “An Empirical Investigation of the Role of Pre-training in Lifelong Learning”, Mehta et al 2021
- “Knowledge-Rich Self-Supervised Entity Linking”, Zhang et al 2021
- “Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biases”, Prabhumoye et al 2021
- “EBERT: Epigenomic Language Models Powered by Cerebras”, Trotter et al 2021
- “You Only Need One Model for Open-domain Question Answering”, Lee et al 2021
- “MAGMA—Multimodal Augmentation of Generative Models through Adapter-based Finetuning”, Eichenberg et al 2021
- “MLP Architectures for Vision-and-Language Modeling: An Empirical Study”, Nie et al 2021
- “Improving Language Models by Retrieving from Trillions of Tokens”, Borgeaud et al 2021
- “Sparse Is Enough in Scaling Transformers”, Jaszczur et al 2021
- “LEMON: Scaling Up Vision-Language Pre-training for Image Captioning”, Hu et al 2021
- “Can Pre-trained Language Models Be Used to Resolve Textual and Semantic Merge Conflicts?”, Zhang et al 2021
- “Florence: A New Foundation Model for Computer Vision”, Yuan et al 2021
- “RedCaps: Web-curated Image-text Data Created by the People, for the People”, Desai et al 2021
- “L-Verse: Bidirectional Generation Between Image and Text”, Kim et al 2021
- “ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning”, Aribandi et al 2021
- “BASIC: Combined Scaling for Open-Vocabulary Image Classification”, Pham et al 2021
- “Swin Transformer V2: Scaling Up Capacity and Resolution”, Liu et al 2021
- “XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale”, Babu et al 2021
- “INTERN: A New Learning Paradigm Towards General Vision”, Shao et al 2021
- “Few-Shot Self-Rationalization With Natural Language Prompts”, Marasović et al 2021
- “Solving Probability and Statistics Problems by Program Synthesis”, Tang et al 2021
- “Covariate Shift in High-Dimensional Random Feature Regression”, Tripuraneni et al 2021
- “Solving Linear Algebra by Program Synthesis”, Drori & Verma 2021
- “MAE: Masked Autoencoders Are Scalable Vision Learners”, He et al 2021
- “Scaling ASR Improves Zero and Few Shot Learning”, Xiao et al 2021
- “Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters”, Lian et al 2021
- “LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs”, Schuhmann et al 2021
- “Training Verifiers to Solve Math Word Problems”, Cobbe et al 2021
- “When in Doubt, Summon the Titans: Efficient Inference With Large Models”, Rawat et al 2021
- “The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail”, Bowman 2021
- “LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5”, Qin & Joty 2021
- “Symbolic Knowledge Distillation: from General Language Models to Commonsense Models”, West et al 2021
- “Scaling Laws for the Few-Shot Adaptation of Pre-trained Image Classifiers”, Prato et al 2021
- “Unsupervised Neural Machine Translation With Generative Language Models Only”, Han et al 2021
- “Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning”, Wu et al 2021
- “Universal Paralinguistic Speech Representations Using Self-Supervised Conformers”, Shor et al 2021
- “M6–10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining”, Lin et al 2021
- “Show Your Work: Scratchpads for Intermediate Computation With Language Models”, Nye et al 2021
- “Exploring the Limits of Large Scale Pre-training”, Abnar et al 2021
- “Learning through Atypical "phase Transitions" in Overparameterized Neural Networks”, Baldassi et al 2021
- “Mining for Strong Gravitational Lenses With Self-supervised Learning”, Stein et al 2021
- “Stochastic Training Is Not Necessary for Generalization”, Geiping et al 2021
- “Evaluating Machine Accuracy on ImageNet”, Shankar et al 2021
- “BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition”, Zhang et al 2021
- “Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers”, Tay et al 2021
- “Scaling Laws for Neural Machine Translation”, Ghorbani et al 2021
- “What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers”, Kim et al 2021
- “TruthfulQA: Measuring How Models Mimic Human Falsehoods”, Lin et al 2021
- “A Recipe For Arbitrary Text Style Transfer With Large Language Models”, Reif et al 2021
- “General-Purpose Question-Answering With Macaw”, Tafjord & Clark 2021
- “A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning”, Dar et al 2021
- “An Empirical Exploration in Quality Filtering of Text Data”, Gao 2021
- “Data and Parameter Scaling Laws for Neural Machine Translation”, Gordon et al 2021
- “Want To Reduce Labeling Cost? GPT-3 Can Help”, Wang et al 2021
- “Do Vision Transformers See Like Convolutional Neural Networks?”, Raghu et al 2021
- “Scaling Laws for Deep Learning”, Rosenfeld 2021
- “Modeling Protein Using Large-scale Pretrain Language Model”, Xiao et al 2021
- “Billion-Scale Pretraining With Vision Transformers for Multi-Task Visual Representations”, Beal et al 2021
- “Facebook AI WMT21 News Translation Task Submission”, Tran et al 2021
- “EVA: An Open-Domain Chinese Dialogue System With Large-Scale Generative Pre-Training”, Zhou et al 2021
- “HTLM: Hyper-Text Pre-Training and Prompting of Language Models”, Aghajanyan et al 2021
- “A Field Guide to Federated Optimization”, Wang et al 2021
- “Brain-like Functional Specialization Emerges Spontaneously in Deep Neural Networks”, Dobs et al 2021
- “ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation”, Sun et al 2021
- “Scarecrow: A Framework for Scrutinizing Machine Text”, Dou et al 2021
- “Revisiting the Calibration of Modern Neural Networks”, Minderer et al 2021
- “HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units”, Hsu et al 2021
- “Partial Success in Closing the Gap between Human and Machine Vision”, Geirhos et al 2021
- “Scaling Laws for Acoustic Models”, Droppo & Elibol 2021
- “Knowledge Distillation: A Good Teacher Is Patient and Consistent”, Beyer et al 2021
- “CoAtNet: Marrying Convolution and Attention for All Data Sizes”, Dai et al 2021
- “Scaling Vision Transformers”, Zhai et al 2021
- “Exploring the Limits of Out-of-Distribution Detection”, Fort et al 2021
- “Effect of Pre-Training Scale on Intra/Inter-Domain Full and Few-Shot Transfer Learning for Natural and Medical X-Ray Chest Images”, Cherti & Jitsev 2021
- “A Universal Law of Robustness via Isoperimetry”, Bubeck & Sellke 2021
- “Naver Unveils First ‘hyperscale’ AI Platform”, Jae-eun 2021
- “One4all User Representation for Recommender Systems in E-commerce”, Shin et al 2021
- “Unsupervised Speech Recognition”, Baevski et al 2021
- “Google Details New AI Accelerator Chips”, Wiggers 2021
- “RecPipe: Co-designing Models and Hardware to Jointly Optimize Recommendation Quality and Performance”, Gupta et al 2021
- “MLP-Mixer: An All-MLP Architecture for Vision”, Tolstikhin et al 2021
- “XLM-R XL: Larger-Scale Transformers for Multilingual Masked Language Modeling”, Goyal et al 2021
- “Scaling End-to-End Models for Large-Scale Multilingual ASR”, Li et al 2021
- “What Are Bayesian Neural Network Posteriors Really Like?”, Izmailov et al 2021
- “DINO: Emerging Properties in Self-Supervised Vision Transformers”, Caron et al 2021
- “Machine Learning Scaling”, Gwern 2021
- “Fully-Connected Neural Nets”, Gwern 2021
- “Computer Optimization: Your Computer Is Faster Than You Think”, Gwern 2021
- “[Ali Released PLUG: 27 Billion Parameters, the Largest Pre-trained Language Model in the Chinese Community]”, Yuying 2021
- “CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP”, Ye et al 2021
- “Revealing Persona Biases in Dialogue Systems”, Sheng et al 2021
- “The Power of Scale for Parameter-Efficient Prompt Tuning”, Lester et al 2021
- “Memorization versus Generalisation in Pre-trained Language Models”, Tänzer et al 2021
- “Probing Across Time: What Does RoBERTa Know and When?”, Liu et al 2021
- “Large-Scale Self-Supervised and Semi-Supervised Learning for Speech Translation”, Wang et al 2021
- “Scaling Laws for Language Transfer Learning”, Kim 2021
- “Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections”, Zhong et al 2021
- “SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network”, Chan et al 2021
- “Understanding Robustness of Transformers for Image Classification”, Bhojanapalli et al 2021
- “UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark”, Lourie et al 2021
- “Efficient Visual Pretraining With Contrastive Detection”, Hénaff et al 2021
- “The Shape of Learning Curves: a Review”, Viering & Loog 2021
- “Controllable Generation from Pre-trained Language Models via Inverse Prompting”, Zou et al 2021
- “Revisiting ResNets: Improved Training and Scaling Strategies”, Bello et al 2021
- “Learning from Videos to Understand the World”, Zweig et al 2021
- “Fast and Accurate Model Scaling”, Dollár et al 2021
- “WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training”, Huo et al 2021
- “Pretrained Transformers As Universal Computation Engines”, Lu et al 2021
- “Greedy Hierarchical Variational Autoencoders (GHVAEs) for Large-Scale Video Prediction”, Wu et al 2021
- “A Law of Robustness for Two-layers Neural Networks”, Bubeck et al 2021
- “Measuring Mathematical Problem Solving With the MATH Dataset”, Hendrycks et al 2021
- “SEER: Self-supervised Pretraining of Visual Features in the Wild”, Goyal et al 2021
- “M6: A Chinese Multimodal Pretrainer”, Lin et al 2021
- “Zero-Shot Text-to-Image Generation”, Ramesh et al 2021
- “Improved Denoising Diffusion Probabilistic Models”, Nichol & Dhariwal 2021
- “Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts”, Changpinyo et al 2021
- “Explaining Neural Scaling Laws”, Bahri et al 2021
- “A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes”, Nado et al 2021
- “NFNet: High-Performance Large-Scale Image Recognition Without Normalization”, Brock et al 2021
- “ALIGN: Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision”, Jia et al 2021
- “Learning Curve Theory”, Hutter 2021
- “1-bit Adam: Communication Efficient Large-Scale Training With Adam’s Convergence Speed”, Tang et al 2021
- “Mind the Gap: Assessing Temporal Generalization in Neural Language Models § Scaling”, Lazaridou et al 2021
- “Scaling Laws for Transfer”, Hernandez et al 2021
- “Muppet: Massive Multi-task Representations With Pre-Finetuning”, Aghajanyan et al 2021
- “Automatic Curation of Large-Scale Datasets for Audio-Visual Representation Learning”, Lee et al 2021
- “Language Processing in Brains and Deep Neural Networks: Computational Convergence and Its Limits”, Caucheteux & King 2021
- “CLIP: Learning Transferable Visual Models From Natural Language Supervision”, Radford et al 2021
- “Meta Pseudo Labels”, Pham et al 2021
- “VinVL: Revisiting Visual Representations in Vision-Language Models”, Zhang et al 2021
- “CDLM: Cross-Document Language Modeling”, Caciularu et al 2021
- “VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation”, Wang et al 2021
- “Process for Adapting Language Models to Society (PALMS) With Values-Targeted Datasets”, Solaiman & Dennison 2021
- “Extrapolating GPT-N Performance”, Finnveden 2020
- “Biological Structure and Function Emerge from Scaling Unsupervised Learning to 250 Million Protein Sequences”, Rives et al 2020
- “CPM: A Large-scale Generative Chinese Pre-trained Language Model”, Zhang et al 2020
- “Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images”, Child 2020
- “When Do You Need Billions of Words of Pretraining Data?”, Zhang et al 2020
- “ML Scaling Subreddit”, Branwen 2020
- “Scaling Laws for Autoregressive Generative Modeling”, Henighan et al 2020
- “Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus”, Caswell et al 2020
- “MT5: A Massively Multilingual Pre-trained Text-to-text Transformer”, Xue et al 2020
- “Beyond English-Centric Multilingual Machine Translation”, Fan et al 2020
- “Towards End-to-End In-Image Neural Machine Translation”, Mansimov et al 2020
- “Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition”, Zhang et al 2020
- “The First AI Model That Translates 100 Languages without Relying on English Data”, Fan 2020
- “The Deep Bootstrap Framework: Good Online Learners Are Good Offline Generalizers”, Nakkiran et al 2020
- “WinoGrande: An Adversarial Winograd Schema Challenge at Scale”, Sakaguchi et al 2020
- “Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)”, Warstadt et al 2020
- “The Neural Architecture of Language: Integrative Reverse-engineering Converges on a Model for Predictive Processing”, Schrimpf et al 2020
- “Fast Stencil-Code Computation on a Wafer-Scale Processor”, Rocki et al 2020
- “Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples”, Gowal et al 2020
- “Vision Transformer: An Image Is Worth 16×16 Words: Transformers for Image Recognition at Scale”, Dosovitskiy et al 2020
- “Small Data, Big Decisions: Model Selection in the Small-Data Regime”, Bornschein et al 2020
- “New Report on How Much Computational Power It Takes to Match the Human Brain”, Carlsmith 2020
- “Generative Language Modeling for Automated Theorem Proving”, Polu & Sutskever 2020
- “GrokNet: Unified Computer Vision Model Trunk and Embeddings For Commerce”, Bell et al 2020
- “Accuracy and Performance Comparison of Video Action Recognition Approaches”, Hutchinson et al 2020
- “Generative Models Are Unsupervised Predictors of Page Quality: A Colossal-Scale Study”, Bahri et al 2020
- “Matt Botvinick on the Spontaneous Emergence of Learning Algorithms”, Scholl 2020
- “Self-supervised Learning through the Eyes of a Child”, Orhan et al 2020
- “Hopfield Networks Is All You Need”, Ramsauer et al 2020
- “On Robustness and Transferability of Convolutional Neural Networks”, Djolonga et al 2020
- “ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Deep Learning and High Performance Computing”, Elnaggar et al 2020
- “NVAE: A Deep Hierarchical Variational Autoencoder”, Vahdat & Kautz 2020
- “Measuring Robustness to Natural Distribution Shifts in Image Classification”, Taori et al 2020
- “Is SGD a Bayesian Sampler? Well, Almost”, Mingard et al 2020
- “Unsupervised Cross-lingual Representation Learning for Speech Recognition”, Conneau et al 2020
- “Logarithmic Pruning Is All You Need”, Orseau et al 2020
- “Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations”, Baevski et al 2020
- “Denoising Diffusion Probabilistic Models”, Ho et al 2020
- “GPT-3 Creative Fiction”, Gwern 2020
- “On the Predictability of Pruning Across Scales”, Rosenfeld et al 2020
- “Image GPT (iGPT): We Find That, Just As a Large Transformer Model Trained on Language Can Generate Coherent Text, the Same Exact Model Trained on Pixel Sequences Can Generate Coherent Image Completions and Samples”, Chen et al 2020
- “SimCLRv2: Big Self-Supervised Models Are Strong Semi-Supervised Learners”, Chen et al 2020
- “SwAV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments”, Caron et al 2020
- “IGPT: Generative Pretraining from Pixels”, Chen et al 2020
- “Are We Done With ImageNet?”, Beyer et al 2020
- “OpenAI API”, Brockman et al 2020
- “How Big Should My Language Model Be?”, Scao 2020
- “Object Segmentation Without Labels With Large-Scale Generative Models”, Voynov et al 2020
- “GPT-3 Paper § Figure F.1: Four Uncurated Completions from a Context Suggesting the Model Compose a Poem in the Style of Wallace Stevens With the Title ‘Shadows on the Way’”, GPT-3 2020 (page 48)
- “The Scaling Hypothesis”, Gwern 2020
- “Danny Hernandez on Forecasting and the Drivers of AI Progress”, Koehler et al 2020
- “ZeRO-2 & DeepSpeed: Shattering Barriers of Deep Learning Speed & Scale”, Team 2020
- “Powered by AI: Advancing Product Understanding and Building New Shopping Experiences”, Berg et al 2020
- “Measuring the Algorithmic Efficiency of Neural Networks”, Hernandez & Brown 2020
- “Pushing the Limit of Molecular Dynamics With ab Initio Accuracy to 100 Million Atoms With Machine Learning”, Jia et al 2020
- “Jukebox: We’re Introducing Jukebox, a Neural Net That Generates Music, including Rudimentary Singing, As Raw Audio in a Variety of Genres and Artist Styles. We’re Releasing the Model Weights and Code, along With a Tool to Explore the Generated Samples.”, Dhariwal et al 2020
- “Blender: A State-of-the-art Open Source Chatbot”, Roller et al 2020
- “A Review of Winograd Schema Challenge Datasets and Approaches”, Kocijan et al 2020
- “Scaling Laws from the Data Manifold Dimension”, Sharma & Kaplan 2020
- “DynamicEmbedding: Extending TensorFlow for Colossal-Scale Applications”, Zeng et al 2020
- “PALM: Pre-training an Autoencoding & Autoregressive Language Model for Context-conditioned Generation”, Bi et al 2020
- “Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems”, Naumov et al 2020
- “A Metric Learning Reality Check”, Musgrave et al 2020
- “TTTTTackling WinoGrande Schemas”, Lin et al 2020
- “Zoom In: An Introduction to Circuits—By Studying the Connections between Neurons, We Can Find Meaningful Algorithms in the Weights of Neural Networks”, Olah et al 2020
- “Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited”, Maddox et al 2020
- “Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers”, Li et al 2020
- “Rethinking Bias-Variance Trade-off for Generalization of Neural Networks”, Yang et al 2020
- “The Messy, Secretive Reality behind OpenAI’s Bid to save the World: The AI Moonshot Was Founded in the Spirit of Transparency. This Is the inside Story of How Competitive Pressure Eroded That Idealism”, Hao 2020
- “A Simple Framework for Contrastive Learning of Visual Representations”, Chen et al 2020
- “Turing-NLG: A 17-billion-parameter Language Model by Microsoft”, Rosset 2020
- “How Much Knowledge Can You Pack Into the Parameters of a Language Model?”, Roberts et al 2020
- “Impact of ImageNet Model Selection on Domain Adaptation”, Zhang & Davison 2020
- “Direct Fit to Nature: An Evolutionary Perspective on Biological and Artificial Neural Networks”, Hasson et al 2020
- “Towards a Conversational Agent That Can Chat About…Anything”, Adiwardana & Luong 2020
- “Towards a Human-like Open-Domain Chatbot”, Adiwardana et al 2020
- “Scaling Laws for Neural Language Models”, Kaplan et al 2020
- “Big Transfer (BiT): General Visual Representation Learning”, Kolesnikov et al 2019
- “Deep Double Descent: We Show That the Double Descent Phenomenon Occurs in CNNs, ResNets, and Transformers: Performance First Improves, Then Gets Worse, and Then Improves Again With Increasing Model Size, Data Size, or Training Time. This Effect Is Often Avoided through Careful Regularization. While This Behavior Appears to Be Fairly Universal, We Don’t yet Fully Understand Why It Happens, and View Further Study of This Phenomenon As an Important Research Direction.”, Nakkiran et al 2019
- “12-in-1: Multi-Task Vision and Language Representation Learning”, Lu et al 2019
- “Deep Double Descent: Where Bigger Models and More Data Hurt”, Nakkiran et al 2019
- “Understanding the Generalization of ‘lottery Tickets’ in Neural Networks”, Morcos & Tian 2019
- “Momentum Contrast for Unsupervised Visual Representation Learning”, He et al 2019
- “The Deep Learning Revolution and Its Implications for Computer Architecture and Chip Design”, Dean 2019
- “Self-training With Noisy Student Improves ImageNet Classification”, Xie et al 2019
- “CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs”, El-Kishky et al 2019
- “CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB”, Schwenk et al 2019
- “XLM-R: State-of-the-art Cross-lingual Understanding through Self-supervision”, FAIR 2019
- “Unsupervised Cross-lingual Representation Learning at Scale”, Conneau et al 2019
- “High Fidelity Video Prediction With Large Stochastic Recurrent Neural Networks”, Villegas et al 2019
- “T5: Exploring the Limits of Transfer Learning With a Unified Text-to-Text Transformer”, Raffel et al 2019
- “ZeRO: Memory Optimizations Toward Training Trillion Parameter Models”, Rajbhandari et al 2019
- “Environmental Drivers of Systematicity and Generalization in a Situated Agent”, Hill et al 2019
- “A Constructive Prediction of the Generalization Error Across Scales”, Rosenfeld et al 2019
- “Large-scale Pretraining for Neural Machine Translation With Tens of Billions of Sentence Pairs”, Meng et al 2019
- “UNITER: UNiversal Image-TExt Representation Learning”, Chen et al 2019
- “Exascale Deep Learning for Scientific Inverse Problems”, Laanait et al 2019
- “Simple, Scalable Adaptation for Neural Machine Translation”, Bapna et al 2019
- “CTRL: A Conditional Transformer Language Model For Controllable Generation”, Keskar et al 2019
- “Show Your Work: Improved Reporting of Experimental Results”, Dodge et al 2019
- “MegatronLM: Training Billion+ Parameter Language Models Using GPU Model Parallelism”, ADLR 2019
- “RoBERTa: A Robustly Optimized BERT Pretraining Approach”, Liu et al 2019
- “Robustness Properties of Facebook’s ResNeXt WSL Models”, Orhan 2019
- “Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges”, Arivazhagan et al 2019
- “Large Scale Adversarial Representation Learning”, Donahue & Simonyan 2019
- “One Epoch Is All You Need”, Komatsuzaki 2019
- “Does Learning Require Memorization? A Short Tale about a Long Tail”, Feldman 2019
- “Intriguing Properties of Adversarial Training at Scale”, Xie & Yuille 2019
- “Scaling Autoregressive Video Models”, Weissenborn et al 2019
- “A Mathematical Theory of Semantic Development in Deep Neural Networks”, Saxe et al 2019
- “Adversarially Robust Generalization Just Requires More Unlabeled Data”, Zhai et al 2019
- “ICML 2019 Notes”, Abel 2019
- “SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers”, Fedorov et al 2019
- “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks”, Tan & Le 2019
- “UniLM: Unified Language Model Pre-training for Natural Language Understanding and Generation”, Dong et al 2019
- “Billion-scale Semi-supervised Learning for Image Classification”, Yalniz et al 2019
- “VideoBERT: A Joint Model for Video and Language Representation Learning”, Sun et al 2019
- “Benchmarking Neural Network Robustness to Common Corruptions and Perturbations”, Hendrycks & Dietterich 2019
- “Surprises in High-Dimensional Ridgeless Least Squares Interpolation”, Hastie et al 2019
- “The Bitter Lesson”, Sutton 2019
- “Deep Learning Hardware: Past, Present, & Future”, LeCun 2019
- “Better Language Models and Their Implications”, Radford et al 2019
- “Language Models Are Unsupervised Multitask Learners”, Radford et al 2019
- “Do ImageNet Classifiers Generalize to ImageNet?”, Recht et al 2019
- “Cross-lingual Language Model Pretraining”, Lample & Conneau 2019
- “High Fidelity Video Prediction With Large Stochastic Recurrent Neural Networks: Videos”, Villegas et al 2019
- “Artificial Intelligence: A Guide for Thinking Humans § Prologue: Terrified”, Mitchell 2019
- “Reconciling Modern Machine Learning Practice and the Bias-variance Trade-off”, Belkin et al 2018
- “Nocaps: Novel Object Captioning at Scale”, Agrawal et al 2018
- “How AI Training Scales”, McCandlish et al 2018
- “Is Science Slowing Down?”, Alexander 2018
- “WBE and DRL: a Middle Way of Imitation Learning from the Human Brain”, Branwen 2018
- “BigGAN: Large Scale GAN Training For High Fidelity Natural Image Synthesis § 5.2 Additional Evaluation On JFT-300M”, Brock et al 2018 (page 8 org deepmind)
- “Large Scale GAN Training for High Fidelity Natural Image Synthesis”, Brock et al 2018
- “Measurement Invariance Explains the Universal Law of Generalization for Psychological Perception”, Frank 2018
- “CurriculumNet: Weakly Supervised Learning from Large-Scale Web Images”, Guo et al 2018
- “Large-Scale Visual Speech Recognition”, Shillingford et al 2018
- “Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations”, Hendrycks & Dietterich 2018
- “Neural Scene Representation and Rendering”, Eslami et al 2018
- “GPT-1: Improving Language Understanding With Unsupervised Learning”, OpenAI 2018
- “GPT-1: Improving Language Understanding by Generative Pre-Training § Model Specifications”, Radford et al 2018 (page 5)
- “GPT-1: Improving Language Understanding by Generative Pre-Training”, Radford et al 2018
- “Do CIFAR-10 Classifiers Generalize to CIFAR-10?”, Recht et al 2018
- “Deep Learning Generalizes Because the Parameter-function Map Is Biased towards Simple Functions”, Valle-Pérez et al 2018
- “Google DeepMind Founder and Leader in Artificial Intelligence Returns to Hamilton”, Tantau 2018
- “Exploring the Limits of Weakly Supervised Pretraining”, Mahajan et al 2018
- “One Big Net For Everything”, Schmidhuber 2018
- “Sensitivity and Generalization in Neural Networks: an Empirical Study”, Novak et al 2018
- “ULMFiT: Universal Language Model Fine-tuning for Text Classification”, Howard & Ruder 2018
- “Deep Image Reconstruction from Human Brain Activity”, Shen et al 2017
- “Deep Learning Scaling Is Predictable, Empirically”, Hestness et al 2017
- “Are GANs Created Equal? A Large-Scale Study”, Lucic et al 2017
- “Knowledge Concentration: Learning 100K Object Classifiers in a Single CNN”, Gao et al 2017
- “Rethinking Generalization Requires Revisiting Old Ideas: Statistical Mechanics Approaches and Complex Learning Behavior”, Martin & Mahoney 2017
- “There’s No Fire Alarm for Artificial General Intelligence”, Yudkowsky 2017
- “WebVision Database: Visual Learning and Understanding from Web Data”, Li et al 2017
- “Revisiting Unreasonable Effectiveness of Data in Deep Learning Era”, Sun et al 2017
- “Towards Deep Learning Models Resistant to Adversarial Attacks”, Madry et al 2017
- “Learning to Learn from Noisy Web Videos”, Yeung et al 2017
- “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour”, Goyal et al 2017
- “A Simple Neural Network Module for Relational Reasoning”, Santoro et al 2017
- “Deep Learning Is Robust to Massive Label Noise”, Rolnick et al 2017
- “Quo Vadis, Action Recognition? A New Model I3D and the Kinetics Dataset”, Carreira & Zisserman 2017
- “WebVision Challenge: Visual Learning and Understanding With Web Data”, Li et al 2017
- “Geometry of Optimization and Implicit Regularization in Deep Learning”, Neyshabur et al 2017
- “On the Impossibility of Supersized Machines”, Garfinkel et al 2017
- “Research Ideas”, Gwern 2017
- “Parallel Multiscale Autoregressive Density Estimation”, Reed et al 2017
- “Universal Representations:The Missing Link between Faces, Text, Planktons, and Cat Breeds”, Bilen & Vedaldi 2017
- “Estimation of Gap Between Current Language Models and Human Performance”, Shen et al 2017
- “Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles”, Lakshminarayanan et al 2016
- “Understanding Deep Learning Requires Rethinking Generalization”, Zhang et al 2016
- “Ra”, Constantin 2016
- “The LAMBADA Dataset: Word Prediction Requiring a Broad Discourse Context”, Paperno et al 2016
- “Do Deep Convolutional Nets Really Need to Be Deep and Convolutional?”, Urban et al 2016
- “PlaNet—Photo Geolocation With Convolutional Neural Networks”, Weyand et al 2016
- “Exploring the Limits of Language Modeling”, Jozefowicz et al 2016
- “The Singularity: A Philosophical Analysis”, Chalmers 2016
- “The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition”, Krause et al 2015
- “Net2Net: Accelerating Learning via Knowledge Transfer”, Chen et al 2015
- “Generative Concatenative Nets Jointly Learn to Write and Classify Reviews”, Lipton et al 2015
- “Learning Visual Features from Large Weakly Supervised Data”, Arm et al 2015
- “LSUN: Construction of a Large-scale Image Dataset Using Deep Learning With Humans in the Loop”, Yu et al 2015
- “Clothing-1M: Learning from Massive Noisy Labeled Data for Image Classification”, Xiao et al 2015
- “The Unreasonable Effectiveness of Recurrent Neural Networks”, Karpathy 2015
- “YFCC100M: The New Data in Multimedia Research”, Thomee et al 2015
- “Evolution of the Human Brain: From Matter to Mind”, Hofman 2015
- “In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning”, Neyshabur et al 2014
- “Technology Forecasting: The Garden of Forking Paths”, Gwern 2014
- “Jumping NLP Curves: A Review of Natural Language Processing Research [Review Article]”, Cambria & White 2014
- “Neural Networks, Manifolds, and Topology”, Olah 2014
- “Computing’s Energy Problem (and What We Can Do about It)”, Horowitz 2014b
- “N-gram Counts and Language Models from the Common Crawl”, Buck et al 2014
- “One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling”, Chelba et al 2013
- “Algorithmic Progress in Six Domains”, Grace 2013
- “Large–Scale Machine Learning Revisited [slides]”, Bottou 2013
- “Intelligence Explosion Microeconomics”, Yudkowsky 2013
- “Scalable Modified Kneser-Ney Language Model Estimation”, Heafield et al 2013
- “The Remarkable, yet Not Extraordinary, Human Brain As a Scaled-up Primate Brain and Its Associated Cost”, Herculano-Houzel 2012
- “Advantages of Artificial Intelligences, Uploads, and Digital Minds”, Sotala 2012
- “Recurrent Neural Network Based Language Model”, Mikolov et al 2010
- “How Complex Are Individual Differences?”, Gwern 2010
- “Understanding Sources of Inefficiency in General-purpose Chips”, Hameed et al 2010
- “The Teenies”, Legg 2009
- “Tick, Tock, Tick, Tock… BING”, Legg 2009
- “Halloween Nightmare Scenario, Early 2020’s”, Wood 2009
- “The Unreasonable Effectiveness of Data”, Halevy et al 2009
- “Economics Of The Singularity: Stuffed into Skyscrapers by the Billion, Brainy Bugbots Will Be the Knowledge Workers of the Future”, Hanson 2008
- “Large Language Models in Machine Translation”, Brants et al 2007
- “Cellular Scaling Rules for Primate Brains”, Herculano-Houzel et al 2007
- “The Tradeoffs of Large-Scale Learning”, Bottou & Bousquet 2007
- “Robot Predictions Evolution”, Moravec 2004
- “Tree Induction vs. Logistic Regression: A Learning-Curve Analysis”, Perlich et al 2003
- “Scaling to Very Very Large Corpora for Natural Language Disambiguation”, Banko & Brill 2001
- “On Discriminative vs. Generative Classifiers: A Comparison of Logistic Regression and Naive Bayes”, Ng & Jordan 2001
- “A Survey of Methods for Scaling Up Inductive Algorithms”, Provost & Kolluri 1999
- “On The Effect of Data Set Size on Bias And Variance in Classification Learning”, Brain & Webb 1999
- “The Effects of Training Set Size on Decision Tree Complexity”, Oates & Jensen 1997
- “Rigorous Learning Curve Bounds from Statistical Mechanics”, Haussler et al 1996
- “Scaling up the Accuracy of Naive-Bayes Classifiers: a Decision-tree Hybrid”, Kohavi 1996
- “Reflections After Refereeing Papers for NIPS”, Breiman 1995
- “Building a Large Annotated Corpus of English: The Penn Treebank”, Marcus et al 1993
- “Statistical Theory of Learning Curves under Entropic Loss Criterion”, Amari & Murata 1993
- “Learning Curves: Asymptotic Values and Rate of Convergence”, Cortes et al 1993
- “Exhaustive Learning”, Schwartz et al 1990
- “Computing With Connections”, Sejnowski 1987
- “The Role Of RAW POWER In INTELLIGENCE”, Moravec 1976
- “Don’t Worry—It Can’t Happen”, Harrington 1940
- “Homepage of Paul F. Christiano”, Christiano 2023
- “Ilya Sutskever: Deep Learning | AI Podcast #94 With Lex Fridman”
- “A Universal Law of Robustness”
- “A Law of Robustness and the Importance of Overparameterization in Deep Learning”
- Sort By Magic
- Wikipedia
- Miscellaneous
- Link Bibliography
Tagged links on machine learning scaling.
For the bibliography of ML scaling papers showing smooth scaling of neural net performance in general with increasingly large parameters, data, & compute organized by topic, see “Machine Learning Scaling” notes. For the essay on NN scaling implications, see “The Scaling Hypothesis”.
See Also
Links
“Mamba: Linear-Time Sequence Modeling With Selective State Spaces”, Gu & Dao 2023
“Mamba: Linear-Time Sequence Modeling with Selective State Spaces”
“Instruction-tuning Aligns LLMs to the Human Brain”, Aw et al 2023
“First Tragedy, Then Parse: History Repeats Itself in the New Era of Large Language Models”, Saphra et al 2023
“First Tragedy, then Parse: History Repeats Itself in the New Era of Large Language Models”
“I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models”, Zhang et al 2023
“I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models”
“Sam Altman Accepts the 2023 Hawking Fellowship Award § Is There Another Breakthrough That’s Needed to Reach AGI?”, Altman 2023
“A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models”, Eisape et al 2023
“A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models”
“ConvNets Match Vision Transformers at Scale”, Smith et al 2023
“PaLI-3 Vision Language Models: Smaller, Faster, Stronger”, Chen et al 2023
“Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning”, Xia et al 2023
“Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning”
“Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition”, Chen et al 2023
“Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition”
“GeoLLM: Extracting Geospatial Knowledge from Large Language Models”, Manvi et al 2023
“GeoLLM: Extracting Geospatial Knowledge from Large Language Models”
“FreshLLMs: Refreshing Large Language Models With Search Engine Augmentation”, Vu et al 2023
“FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation”
“SeamlessM4T: Massively Multilingual & Multimodal Machine Translation”, Communication et al 2023
“SeamlessM4T: Massively Multilingual & Multimodal Machine Translation”
“Simple Synthetic Data Reduces Sycophancy in Large Language Models”, Wei et al 2023
“Simple synthetic data reduces sycophancy in large language models”
“Absolute Unit NNs: Regression-Based MLPs for Everything”, Gwern 2023
“LLaMA 2: Open Foundation and Fine-Tuned Chat Models”, Touvron et al 2023
“Introducing Superalignment”, Leike & Sutskever 2023
“Gödel, Escher, Bach Author Douglas Hofstadter on the State of AI Today § What about AI Terrifies You?”, Hofstadter & Kim 2023
“Pretraining Task Diversity and the Emergence of Non-Bayesian In-context Learning for Regression”, Raventós et al 2023
“Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression”
“Scaling MLPs: A Tale of Inductive Bias”, Bachmann et al 2023
“Image Captioners Are Scalable Vision Learners Too”, Tschannen et al 2023
“PaLI-X: On Scaling up a Multilingual Vision and Language Model”, Chen et al 2023
“PaLI-X: On Scaling up a Multilingual Vision and Language Model”
“Scaling Data-Constrained Language Models”, Muennighoff et al 2023
“The False Promise of Imitating Proprietary LLMs”, Gudibande et al 2023
“Scaling Laws for Language Encoding Models in FMRI”, Antonello et al 2023
“LIMA: Less Is More for Alignment”, Zhou et al 2023
“Google's Newest A.I. Model Uses Nearly 5× More Text Data for Training Than Its Predecessor”, Elias 2023
“Google's newest A.I. model uses nearly 5× more text data for training than its predecessor”
“TinyStories: How Small Can Language Models Be and Still Speak Coherent English?”, Eldan & Li 2023
“TinyStories: How Small Can Language Models Be and Still Speak Coherent English?”
“TorToise: Better Speech Synthesis through Scaling”, Betker 2023
“ImageBind: One Embedding Space To Bind Them All”, Girdhar et al 2023
“Finding Neurons in a Haystack: Case Studies With Sparse Probing”, Gurnee et al 2023
“Finding Neurons in a Haystack: Case Studies with Sparse Probing”
“Emergent and Predictable Memorization in Large Language Models”, Biderman et al 2023
“Emergent and Predictable Memorization in Large Language Models”
“Even The Politicians Thought the Open Letter Made No Sense In The Senate Hearing on AI Today's Hearing on Ai Covered Ai Regulation and Challenges, and the Infamous Open Letter, Which Nearly Everyone in the Room Thought Was Unwise”, Gorrell 2023
“Power Law Trends in Speedrunning and Machine Learning”, Erdil & Sevilla 2023
“DINOv2: Learning Robust Visual Features without Supervision”, Oquab et al 2023
“DINOv2: Learning Robust Visual Features without Supervision”
“Segment Anything”, Kirillov et al 2023
“Sigmoid Loss for Language Image Pre-Training”, Zhai et al 2023
“How Well Do Large Language Models Perform in Arithmetic Tasks?”, Yuan et al 2023
“How well do Large Language Models perform in Arithmetic tasks?”
“GPT-4 Technical Report”, OpenAI 2023
“Securing Liberal Democratic Control of AGI through UK Leadership”, Phillips 2023
“Securing Liberal Democratic Control of AGI through UK Leadership”
“Language Is Not All You Need: Aligning Perception With Language Models (Kosmos-1)”, Huang et al 2023
“Language Is Not All You Need: Aligning Perception with Language Models (Kosmos-1)”
“Scaling Vision Transformers to 22 Billion Parameters”, Dehghani et al 2023
“Large Language Models As Fiduciaries: A Case Study Toward Robustly Communicating With Artificial Intelligence Through Legal Standards”, Nay 2023
“ClimaX: A Foundation Model for Weather and Climate”, Nguyen et al 2023
“StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis”, Sauer et al 2023
“StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis”
“MUG: Vision Learners Meet Web Image-Text Pairs”, Zhao et al 2023
“GPT-3 As Knowledge Worker: A Zero-Shot Evaluation of (AI)CPA Capabilities”, Bommarito et al 2023
“GPT-3 as Knowledge Worker: A Zero-Shot Evaluation of (AI)CPA Capabilities”
“Scaling Laws for Generative Mixed-Modal Language Models”, Aghajanyan et al 2023
“VALL-E: Neural Codec Language Models Are Zero-Shot Text to Speech Synthesizers”, Wang et al 2023
“VALL-E: Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers”
“GPT-3 Takes the Bar Exam”, II & Katz 2022
“One Embedder, Any Task: Instruction-Finetuned Text Embeddings (INSTRUCTOR)”, Su et al 2022
“One Embedder, Any Task: Instruction-Finetuned Text Embeddings (INSTRUCTOR)”
“Discovering Language Model Behaviors With Model-Written Evaluations”, Perez et al 2022
“Discovering Language Model Behaviors with Model-Written Evaluations”
“Reproducible Scaling Laws for Contrastive Language-image Learning”, Cherti et al 2022
“Reproducible scaling laws for contrastive language-image learning”
“ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languages”, Chai et al 2022
“ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languages”
“VindLU: A Recipe for Effective Video-and-Language Pretraining”, Cheng et al 2022
“VindLU: A Recipe for Effective Video-and-Language Pretraining”
“Video-Text Modeling With Zero-Shot Transfer from Contrastive Captioners”, Yan et al 2022
“Video-Text Modeling with Zero-Shot Transfer from Contrastive Captioners”
“Robust Speech Recognition via Large-Scale Weak Supervision”, Radford et al 2022
“Robust Speech Recognition via Large-Scale Weak Supervision”
“Scaling Language-Image Pre-training via Masking”, Li et al 2022
“MultiRay: Optimizing Efficiency for Large-scale AI Models”, Gupta et al 2022
“Galactica: A Large Language Model for Science”, Taylor et al 2022
“EVA: Exploring the Limits of Masked Visual Representation Learning at Scale”, Fang et al 2022
“EVA: Exploring the Limits of Masked Visual Representation Learning at Scale”
“MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation”, Feng et al 2022
“MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation”
“Increments Podcast: #45—4 Central Fallacies of AI Research (with Melanie Mitchell)”, Mitchell & Chugg 2022
“Increments Podcast: #45—4 Central Fallacies of AI Research (with Melanie Mitchell)”
“Will We Run out of Data? An Analysis of the Limits of Scaling Datasets in Machine Learning”, Villalobos et al 2022
“Will we run out of data? An analysis of the limits of scaling datasets in Machine Learning”
“Evaluating Parameter Efficient Learning for Generation”, Xu et al 2022
“FLAN: Scaling Instruction-Finetuned Language Models”, Chung et al 2022
“BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining”, Luo et al 2022
“BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining”
“Vision-Language Pre-training: Basics, Recent Advances, and Future Trends”, Gan et al 2022
“Vision-Language Pre-training: Basics, Recent Advances, and Future Trends”
“Foundation Transformers”, Wang et al 2022
“Self-Ask: Measuring and Narrowing the Compositionality Gap in Language Models (Bamboogle)”, Press et al 2022
“Self-Ask: Measuring and Narrowing the Compositionality Gap in Language Models (Bamboogle)”
“Ask Me Anything (AMA): A Simple Strategy for Prompting Language Models”, Arora et al 2022
“Ask Me Anything (AMA): A simple strategy for prompting language models”
“GLM-130B: An Open Bilingual Pre-trained Model”, Zeng et al 2022
“Do Current Multi-Task Optimization Methods in Deep Learning Even Help?”, Xin et al 2022
“Do Current Multi-Task Optimization Methods in Deep Learning Even Help?”
“Monolith: Real Time Recommendation System With Collisionless Embedding Table”, Liu et al 2022
“Monolith: Real Time Recommendation System With Collisionless Embedding Table”
“Machine Reading, Fast and Slow: When Do Models "Understand" Language?”, Choudhury et al 2022
“Machine Reading, Fast and Slow: When Do Models "Understand" Language?”
“PaLI: A Jointly-Scaled Multilingual Language-Image Model”, Chen et al 2022
“Using Large Language Models to Simulate Multiple Humans”, Aher et al 2022
“LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale”, Dettmers et al 2022
“LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale”
“Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP”, Nguyen et al 2022
“Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP”
“Efficient Training of Language Models to Fill in the Middle”, Bavarian et al 2022
“Efficient Training of Language Models to Fill in the Middle”
“ESMfold: Language Models of Protein Sequences at the Scale of Evolution Enable Accurate Structure Prediction”, Lin et al 2022
“Why Do Tree-based Models Still Outperform Deep Learning on Tabular Data?”, Grinsztajn et al 2022
“Why do tree-based models still outperform deep learning on tabular data?”
“PIXEL: Language Modelling With Pixels”, Rust et al 2022
“High-performing Neural Network Models of Visual Cortex Benefit from High Latent Dimensionality”, Elmoznino & Bonner 2022
“High-performing neural network models of visual cortex benefit from high latent dimensionality”
“Language Models (Mostly) Know What They Know”, Kadavath et al 2022
“Exploring Length Generalization in Large Language Models”, Anil et al 2022
“On-Device Training Under 256KB Memory”, Lin et al 2022
“Beyond Neural Scaling Laws: Beating Power Law Scaling via Data Pruning”, Sorscher et al 2022
“Beyond neural scaling laws: beating power law scaling via data pruning”
“ProGen2: Exploring the Boundaries of Protein Language Models”, Nijkamp et al 2022
“ProGen2: Exploring the Boundaries of Protein Language Models”
“RST: ReStructured Pre-training”, Yuan & Liu 2022
“Limitations of the NTK for Understanding Generalization in Deep Learning”, Vyas et al 2022
“Limitations of the NTK for Understanding Generalization in Deep Learning”
“Modeling Transformative AI Risks (MTAIR) Project—Summary Report”, Clarke et al 2022
“Modeling Transformative AI Risks (MTAIR) Project—Summary Report”
“BigVGAN: A Universal Neural Vocoder With Large-Scale Training”, Lee et al 2022
“BigVGAN: A Universal Neural Vocoder with Large-Scale Training”
“An Improved One Millisecond Mobile Backbone”, Vasu et al 2022
“A Neural Corpus Indexer for Document Retrieval”, Wang et al 2022
“Toward a Realistic Model of Speech Processing in the Brain With Self-supervised Learning”, Millet et al 2022
“Toward a realistic model of speech processing in the brain with self-supervised learning”
“Teaching Models to Express Their Uncertainty in Words”, Lin et al 2022
“M3AE: Multimodal Masked Autoencoders Learn Transferable Representations”, Geng et al 2022
“M3AE: Multimodal Masked Autoencoders Learn Transferable Representations”
“Why Robust Generalization in Deep Learning Is Difficult: Perspective of Expressive Power”, Li et al 2022
“Why Robust Generalization in Deep Learning is Difficult: Perspective of Expressive Power”
“InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning”, Gupta et al 2022
“InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning”
“Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models”, Tirumala et al 2022
“Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models”
“Least-to-Most Prompting Enables Complex Reasoning in Large Language Models”, Zhou et al 2022
“Least-to-Most Prompting Enables Complex Reasoning in Large Language Models”
“Towards Understanding Grokking: An Effective Theory of Representation Learning”, Liu et al 2022
“Towards Understanding Grokking: An Effective Theory of Representation Learning”
“Continual Pre-Training Mitigates Forgetting in Language and Vision”, Cossu et al 2022
“Continual Pre-Training Mitigates Forgetting in Language and Vision”
“Dialog Inpainting: Turning Documents into Dialogues”, Dai et al 2022
“Unifying Language Learning Paradigms”, Tay et al 2022
“When Does Dough Become a Bagel? Analyzing the Remaining Mistakes on ImageNet”, Vasudevan et al 2022
“When does dough become a bagel? Analyzing the remaining mistakes on ImageNet”
“Building Machine Translation Systems for the Next Thousand Languages”, Bapna et al 2022
“Building Machine Translation Systems for the Next Thousand Languages”
“CoCa: Contrastive Captioners Are Image-Text Foundation Models”, Yu et al 2022
“CoCa: Contrastive Captioners are Image-Text Foundation Models”
“Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)”, Fang et al 2022
“Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)”
“Continual Learning With Foundation Models: An Empirical Study of Latent Replay”, Ostapenko et al 2022
“Continual Learning with Foundation Models: An Empirical Study of Latent Replay”
“Flamingo: a Visual Language Model for Few-Shot Learning”, Alayrac et al 2022
“WebFace260M: A Benchmark for Million-Scale Deep Face Recognition”, Zhu et al 2022
“WebFace260M: A Benchmark for Million-Scale Deep Face Recognition”
“What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?”, Wang et al 2022
“What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?”
“DeepMind: The Podcast—Excerpts on AGI”, Kiely 2022
“Can Language Models Learn from Explanations in Context?”, Lampinen et al 2022
“Chinchilla: Training Compute-Optimal Large Language Models”, Hoffmann et al 2022
“Chinchilla: Training Compute-Optimal Large Language Models”
“A Roadmap for Big Model”, Yuan et al 2022
“A Conversational Paradigm for Program Synthesis”, Nijkamp et al 2022
“Self-Consistency Improves Chain-of-Thought Reasoning in Language Models”, Wang et al 2022
“Self-Consistency Improves Chain-of-Thought Reasoning in Language Models”
“Effect of Scale on Catastrophic Forgetting in Neural Networks”, Ramasesh et al 2022
“Effect of scale on catastrophic forgetting in neural networks”
“Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer”, Yang et al 2022
“Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer”
“FastFold: Reducing AlphaFold Training Time from 11 Days to 67 Hours”, Cheng et al 2022
“FastFold: Reducing AlphaFold Training Time from 11 Days to 67 Hours”
“Variational Autoencoders Without the Variation”, Daly et al 2022
“Performance Reserves in Brain-imaging-based Phenotype Prediction”, Schulz et al 2022
“Performance reserves in brain-imaging-based phenotype prediction”
“Self-Distilled StyleGAN: Towards Generation from Internet Photos”, Mokady et al 2022
“Self-Distilled StyleGAN: Towards Generation from Internet Photos”
“UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Training”, Khashabi et al 2022
“UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Training”
“Brains and Algorithms Partially Converge in Natural Language Processing”, Caucheteux & King 2022
“Brains and algorithms partially converge in natural language processing”
“Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervision”, Goyal et al 2022
“Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervision”
“Wukong: 100 Million Large-scale Chinese Cross-modal Pre-training Dataset and A Foundation Framework”, Gu et al 2022
“OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework”, Wang et al 2022
“Webly Supervised Concept Expansion for General Purpose Vision Models”, Kamath et al 2022
“Webly Supervised Concept Expansion for General Purpose Vision Models”
“Data Scaling Laws in NMT: The Effect of Noise and Architecture”, Bansal et al 2022
“Data Scaling Laws in NMT: The Effect of Noise and Architecture”
“Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model”, Smith et al 2022
“Reasoning Like Program Executors”, Pi et al 2022
“Text and Code Embeddings by Contrastive Pre-Training”, Neelakantan et al 2022
“SWAG: Revisiting Weakly Supervised Pre-Training of Visual Perception Models”, Singh et al 2022
“SWAG: Revisiting Weakly Supervised Pre-Training of Visual Perception Models”
“LaMDA: Language Models for Dialog Applications”, Thoppilan et al 2022
“CM3: A Causal Masked Multimodal Model of the Internet”, Aghajanyan et al 2022
“ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalization”, Xu et al 2022
“ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalization”
“The Defeat of the Winograd Schema Challenge”, Kocijan et al 2022
“Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets [paper]”, Power et al 2022
“Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets [paper]”
“AV-HuBERT: Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction”, Shi et al 2022
“AV-HuBERT: Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction”
“Robust Self-Supervised Audio-Visual Speech Recognition”, Shi et al 2022
“Self-supervised Learning from 100 Million Medical Images”, Ghesu et al 2022
“The Evolution of Quantitative Sensitivity”, Bryer et al 2021
“ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation”, Wang et al 2021
“XGLM: Few-shot Learning With Multilingual Language Models”, Lin et al 2021
“An Empirical Investigation of the Role of Pre-training in Lifelong Learning”, Mehta et al 2021
“An Empirical Investigation of the Role of Pre-training in Lifelong Learning”
“Knowledge-Rich Self-Supervised Entity Linking”, Zhang et al 2021
“Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biases”, Prabhumoye et al 2021
“Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biases”
“EBERT: Epigenomic Language Models Powered by Cerebras”, Trotter et al 2021
“You Only Need One Model for Open-domain Question Answering”, Lee et al 2021
“You Only Need One Model for Open-domain Question Answering”
“MAGMA—Multimodal Augmentation of Generative Models through Adapter-based Finetuning”, Eichenberg et al 2021
“MAGMA—Multimodal Augmentation of Generative Models through Adapter-based Finetuning”
“MLP Architectures for Vision-and-Language Modeling: An Empirical Study”, Nie et al 2021
“MLP Architectures for Vision-and-Language Modeling: An Empirical Study”
“Improving Language Models by Retrieving from Trillions of Tokens”, Borgeaud et al 2021
“Improving language models by retrieving from trillions of tokens”
“Sparse Is Enough in Scaling Transformers”, Jaszczur et al 2021
“LEMON: Scaling Up Vision-Language Pre-training for Image Captioning”, Hu et al 2021
“LEMON: Scaling Up Vision-Language Pre-training for Image Captioning”
“Can Pre-trained Language Models Be Used to Resolve Textual and Semantic Merge Conflicts?”, Zhang et al 2021
“Can Pre-trained Language Models be Used to Resolve Textual and Semantic Merge Conflicts?”
“Florence: A New Foundation Model for Computer Vision”, Yuan et al 2021
“RedCaps: Web-curated Image-text Data Created by the People, for the People”, Desai et al 2021
“RedCaps: web-curated image-text data created by the people, for the people”
“L-Verse: Bidirectional Generation Between Image and Text”, Kim et al 2021
“ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning”, Aribandi et al 2021
“ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning”
“BASIC: Combined Scaling for Open-Vocabulary Image Classification”, Pham et al 2021
“BASIC: Combined Scaling for Open-Vocabulary Image Classification”
“Swin Transformer V2: Scaling Up Capacity and Resolution”, Liu et al 2021
“XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale”, Babu et al 2021
“XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale”
“INTERN: A New Learning Paradigm Towards General Vision”, Shao et al 2021
“Few-Shot Self-Rationalization With Natural Language Prompts”, Marasović et al 2021
“Few-Shot Self-Rationalization with Natural Language Prompts”
“Solving Probability and Statistics Problems by Program Synthesis”, Tang et al 2021
“Solving Probability and Statistics Problems by Program Synthesis”
“Covariate Shift in High-Dimensional Random Feature Regression”, Tripuraneni et al 2021
“Covariate Shift in High-Dimensional Random Feature Regression”
“Solving Linear Algebra by Program Synthesis”, Drori & Verma 2021
“MAE: Masked Autoencoders Are Scalable Vision Learners”, He et al 2021
“Scaling ASR Improves Zero and Few Shot Learning”, Xiao et al 2021
“Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters”, Lian et al 2021
“LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs”, Schuhmann et al 2021
“LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs”
“Training Verifiers to Solve Math Word Problems”, Cobbe et al 2021
“When in Doubt, Summon the Titans: Efficient Inference With Large Models”, Rawat et al 2021
“When in Doubt, Summon the Titans: Efficient Inference with Large Models”
“The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail”, Bowman 2021
“The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail”
“LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5”, Qin & Joty 2021
“LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5”
“Symbolic Knowledge Distillation: from General Language Models to Commonsense Models”, West et al 2021
“Symbolic Knowledge Distillation: from General Language Models to Commonsense Models”
“Scaling Laws for the Few-Shot Adaptation of Pre-trained Image Classifiers”, Prato et al 2021
“Scaling Laws for the Few-Shot Adaptation of Pre-trained Image Classifiers”
“Unsupervised Neural Machine Translation With Generative Language Models Only”, Han et al 2021
“Unsupervised Neural Machine Translation with Generative Language Models Only”
“Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning”, Wu et al 2021
“Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning”
“Universal Paralinguistic Speech Representations Using Self-Supervised Conformers”, Shor et al 2021
“Universal Paralinguistic Speech Representations Using Self-Supervised Conformers”
“M6–10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining”, Lin et al 2021
“M6–10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining”
“Show Your Work: Scratchpads for Intermediate Computation With Language Models”, Nye et al 2021
“Show Your Work: Scratchpads for Intermediate Computation with Language Models”
“Exploring the Limits of Large Scale Pre-training”, Abnar et al 2021
“Learning through Atypical "phase Transitions" in Overparameterized Neural Networks”, Baldassi et al 2021
“Learning through atypical "phase transitions" in overparameterized neural networks”
“Mining for Strong Gravitational Lenses With Self-supervised Learning”, Stein et al 2021
“Mining for strong gravitational lenses with self-supervised learning”
“Stochastic Training Is Not Necessary for Generalization”, Geiping et al 2021
“Evaluating Machine Accuracy on ImageNet”, Shankar et al 2021
“BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition”, Zhang et al 2021
“Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers”, Tay et al 2021
“Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers”
“Scaling Laws for Neural Machine Translation”, Ghorbani et al 2021
“What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers”, Kim et al 2021
“TruthfulQA: Measuring How Models Mimic Human Falsehoods”, Lin et al 2021
“A Recipe For Arbitrary Text Style Transfer With Large Language Models”, Reif et al 2021
“A Recipe For Arbitrary Text Style Transfer with Large Language Models”
“General-Purpose Question-Answering With Macaw”, Tafjord & Clark 2021
“A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning”, Dar et al 2021
“An Empirical Exploration in Quality Filtering of Text Data”, Gao 2021
“An Empirical Exploration in Quality Filtering of Text Data”
“Data and Parameter Scaling Laws for Neural Machine Translation”, Gordon et al 2021
“Data and Parameter Scaling Laws for Neural Machine Translation”
“Want To Reduce Labeling Cost? GPT-3 Can Help”, Wang et al 2021
“Do Vision Transformers See Like Convolutional Neural Networks?”, Raghu et al 2021
“Do Vision Transformers See Like Convolutional Neural Networks?”
“Scaling Laws for Deep Learning”, Rosenfeld 2021
“Modeling Protein Using Large-scale Pretrain Language Model”, Xiao et al 2021
“Modeling Protein Using Large-scale Pretrain Language Model”
“Billion-Scale Pretraining With Vision Transformers for Multi-Task Visual Representations”, Beal et al 2021
“Billion-Scale Pretraining with Vision Transformers for Multi-Task Visual Representations”
“Facebook AI WMT21 News Translation Task Submission”, Tran et al 2021
“EVA: An Open-Domain Chinese Dialogue System With Large-Scale Generative Pre-Training”, Zhou et al 2021
“EVA: An Open-Domain Chinese Dialogue System with Large-Scale Generative Pre-Training”
“HTLM: Hyper-Text Pre-Training and Prompting of Language Models”, Aghajanyan et al 2021
“HTLM: Hyper-Text Pre-Training and Prompting of Language Models”
“A Field Guide to Federated Optimization”, Wang et al 2021
“Brain-like Functional Specialization Emerges Spontaneously in Deep Neural Networks”, Dobs et al 2021
“Brain-like functional specialization emerges spontaneously in deep neural networks”
“ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation”, Sun et al 2021
“ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation”
“Scarecrow: A Framework for Scrutinizing Machine Text”, Dou et al 2021
“Revisiting the Calibration of Modern Neural Networks”, Minderer et al 2021
“HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units”, Hsu et al 2021
“HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units”
“Partial Success in Closing the Gap between Human and Machine Vision”, Geirhos et al 2021
“Partial success in closing the gap between human and machine vision”
“Scaling Laws for Acoustic Models”, Droppo & Elibol 2021
“Knowledge Distillation: A Good Teacher Is Patient and Consistent”, Beyer et al 2021
“Knowledge distillation: A good teacher is patient and consistent”
“CoAtNet: Marrying Convolution and Attention for All Data Sizes”, Dai et al 2021
“CoAtNet: Marrying Convolution and Attention for All Data Sizes”
“Scaling Vision Transformers”, Zhai et al 2021
“Exploring the Limits of Out-of-Distribution Detection”, Fort et al 2021
“Effect of Pre-Training Scale on Intra/Inter-Domain Full and Few-Shot Transfer Learning for Natural and Medical X-Ray Chest Images”, Cherti & Jitsev 2021
“A Universal Law of Robustness via Isoperimetry”, Bubeck & Sellke 2021
“Naver Unveils First ‘hyperscale’ AI Platform”, Jae-eun 2021
“One4all User Representation for Recommender Systems in E-commerce”, Shin et al 2021
“One4all User Representation for Recommender Systems in E-commerce”
“Unsupervised Speech Recognition”, Baevski et al 2021
“Google Details New AI Accelerator Chips”, Wiggers 2021
“RecPipe: Co-designing Models and Hardware to Jointly Optimize Recommendation Quality and Performance”, Gupta et al 2021
“MLP-Mixer: An All-MLP Architecture for Vision”, Tolstikhin et al 2021
“XLM-R XL: Larger-Scale Transformers for Multilingual Masked Language Modeling”, Goyal et al 2021
“XLM-R XL: Larger-Scale Transformers for Multilingual Masked Language Modeling”
“Scaling End-to-End Models for Large-Scale Multilingual ASR”, Li et al 2021
“Scaling End-to-End Models for Large-Scale Multilingual ASR”
“What Are Bayesian Neural Network Posteriors Really Like?”, Izmailov et al 2021
“DINO: Emerging Properties in Self-Supervised Vision Transformers”, Caron et al 2021
“DINO: Emerging Properties in Self-Supervised Vision Transformers”
“Machine Learning Scaling”, Gwern 2021
“Fully-Connected Neural Nets”, Gwern 2021
“Computer Optimization: Your Computer Is Faster Than You Think”, Gwern 2021
“Computer Optimization: Your Computer Is Faster Than You Think”
“[Ali Released PLUG: 27 Billion Parameters, the Largest Pre-trained Language Model in the Chinese Community]”, Yuying 2021
“CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP”, Ye et al 2021
“CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP”
“Revealing Persona Biases in Dialogue Systems”, Sheng et al 2021
“The Power of Scale for Parameter-Efficient Prompt Tuning”, Lester et al 2021
“Memorization versus Generalisation in Pre-trained Language Models”, Tänzer et al 2021
“Memorization versus Generalisation in Pre-trained Language Models”
“Probing Across Time: What Does RoBERTa Know and When?”, Liu et al 2021
“Large-Scale Self-Supervised and Semi-Supervised Learning for Speech Translation”, Wang et al 2021
“Large-Scale Self-Supervised and Semi-Supervised Learning for Speech Translation”
“Scaling Laws for Language Transfer Learning”, Kim 2021
“Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections”, Zhong et al 2021
“Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections”
“SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network”, Chan et al 2021
“SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network”
“Understanding Robustness of Transformers for Image Classification”, Bhojanapalli et al 2021
“Understanding Robustness of Transformers for Image Classification”
“UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark”, Lourie et al 2021
“UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark”
“Efficient Visual Pretraining With Contrastive Detection”, Hénaff et al 2021
“The Shape of Learning Curves: a Review”, Viering & Loog 2021
“Controllable Generation from Pre-trained Language Models via Inverse Prompting”, Zou et al 2021
“Controllable Generation from Pre-trained Language Models via Inverse Prompting”
“Revisiting ResNets: Improved Training and Scaling Strategies”, Bello et al 2021
“Revisiting ResNets: Improved Training and Scaling Strategies”
“Learning from Videos to Understand the World”, Zweig et al 2021
“Fast and Accurate Model Scaling”, Dollár et al 2021
“WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training”, Huo et al 2021
“WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training”
“Pretrained Transformers As Universal Computation Engines”, Lu et al 2021
“Greedy Hierarchical Variational Autoencoders (GHVAEs) for Large-Scale Video Prediction”, Wu et al 2021
“Greedy Hierarchical Variational Autoencoders (GHVAEs) for Large-Scale Video Prediction”
“A Law of Robustness for Two-layers Neural Networks”, Bubeck et al 2021
“Measuring Mathematical Problem Solving With the MATH Dataset”, Hendrycks et al 2021
“Measuring Mathematical Problem Solving With the MATH Dataset”
“SEER: Self-supervised Pretraining of Visual Features in the Wild”, Goyal et al 2021
“SEER: Self-supervised Pretraining of Visual Features in the Wild”
“M6: A Chinese Multimodal Pretrainer”, Lin et al 2021
“Zero-Shot Text-to-Image Generation”, Ramesh et al 2021
“Improved Denoising Diffusion Probabilistic Models”, Nichol & Dhariwal 2021
“Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts”, Changpinyo et al 2021
“Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts”
“Explaining Neural Scaling Laws”, Bahri et al 2021
“A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes”, Nado et al 2021
“A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes”
“NFNet: High-Performance Large-Scale Image Recognition Without Normalization”, Brock et al 2021
“NFNet: High-Performance Large-Scale Image Recognition Without Normalization”
“ALIGN: Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision”, Jia et al 2021
“ALIGN: Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision”
“Learning Curve Theory”, Hutter 2021
“1-bit Adam: Communication Efficient Large-Scale Training With Adam’s Convergence Speed”, Tang et al 2021
“1-bit Adam: Communication Efficient Large-Scale Training with Adam’s Convergence Speed”
“Mind the Gap: Assessing Temporal Generalization in Neural Language Models § Scaling”, Lazaridou et al 2021
“Mind the Gap: Assessing Temporal Generalization in Neural Language Models § Scaling”
“Scaling Laws for Transfer”, Hernandez et al 2021
“Muppet: Massive Multi-task Representations With Pre-Finetuning”, Aghajanyan et al 2021
“Muppet: Massive Multi-task Representations with Pre-Finetuning”
“Automatic Curation of Large-Scale Datasets for Audio-Visual Representation Learning”, Lee et al 2021
“Automatic Curation of Large-Scale Datasets for Audio-Visual Representation Learning”
“Language Processing in Brains and Deep Neural Networks: Computational Convergence and Its Limits”, Caucheteux & King 2021
“Language processing in brains and deep neural networks: computational convergence and its limits”
“CLIP: Learning Transferable Visual Models From Natural Language Supervision”, Radford et al 2021
“CLIP: Learning Transferable Visual Models From Natural Language Supervision”
“Meta Pseudo Labels”, Pham et al 2021
“VinVL: Revisiting Visual Representations in Vision-Language Models”, Zhang et al 2021
“VinVL: Revisiting Visual Representations in Vision-Language Models”
“CDLM: Cross-Document Language Modeling”, Caciularu et al 2021
“VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation”, Wang et al 2021
“Process for Adapting Language Models to Society (PALMS) With Values-Targeted Datasets”, Solaiman & Dennison 2021
“Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets”
“Extrapolating GPT-N Performance”, Finnveden 2020
“Biological Structure and Function Emerge from Scaling Unsupervised Learning to 250 Million Protein Sequences”, Rives et al 2020
“CPM: A Large-scale Generative Chinese Pre-trained Language Model”, Zhang et al 2020
“CPM: A Large-scale Generative Chinese Pre-trained Language Model”
“Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images”, Child 2020
“Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images”
“When Do You Need Billions of Words of Pretraining Data?”, Zhang et al 2020
“ML Scaling Subreddit”, Branwen 2020
“Scaling Laws for Autoregressive Generative Modeling”, Henighan et al 2020
“Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus”, Caswell et al 2020
“Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus”
“MT5: A Massively Multilingual Pre-trained Text-to-text Transformer”, Xue et al 2020
“mT5: A massively multilingual pre-trained text-to-text transformer”
“Beyond English-Centric Multilingual Machine Translation”, Fan et al 2020
“Towards End-to-End In-Image Neural Machine Translation”, Mansimov et al 2020
“Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition”, Zhang et al 2020
“Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition”
“The First AI Model That Translates 100 Languages without Relying on English Data”, Fan 2020
“The first AI model that translates 100 languages without relying on English data”
“The Deep Bootstrap Framework: Good Online Learners Are Good Offline Generalizers”, Nakkiran et al 2020
“The Deep Bootstrap Framework: Good Online Learners are Good Offline Generalizers”
“WinoGrande: An Adversarial Winograd Schema Challenge at Scale”, Sakaguchi et al 2020
“WinoGrande: An Adversarial Winograd Schema Challenge at Scale”
“Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)”, Warstadt et al 2020
“The Neural Architecture of Language: Integrative Reverse-engineering Converges on a Model for Predictive Processing”, Schrimpf et al 2020
“Fast Stencil-Code Computation on a Wafer-Scale Processor”, Rocki et al 2020
“Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples”, Gowal et al 2020
“Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples”
“Vision Transformer: An Image Is Worth 16×16 Words: Transformers for Image Recognition at Scale”, Dosovitskiy et al 2020
“Vision Transformer: An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale”
“Small Data, Big Decisions: Model Selection in the Small-Data Regime”, Bornschein et al 2020
“Small Data, Big Decisions: Model Selection in the Small-Data Regime”
“New Report on How Much Computational Power It Takes to Match the Human Brain”, Carlsmith 2020
“New Report on How Much Computational Power It Takes to Match the Human Brain”
“Generative Language Modeling for Automated Theorem Proving”, Polu & Sutskever 2020
“Generative Language Modeling for Automated Theorem Proving”
“GrokNet: Unified Computer Vision Model Trunk and Embeddings For Commerce”, Bell et al 2020
“GrokNet: Unified Computer Vision Model Trunk and Embeddings For Commerce”
“Accuracy and Performance Comparison of Video Action Recognition Approaches”, Hutchinson et al 2020
“Accuracy and Performance Comparison of Video Action Recognition Approaches”
“Generative Models Are Unsupervised Predictors of Page Quality: A Colossal-Scale Study”, Bahri et al 2020
“Generative Models are Unsupervised Predictors of Page Quality: A Colossal-Scale Study”
“Matt Botvinick on the Spontaneous Emergence of Learning Algorithms”, Scholl 2020
“Matt Botvinick on the spontaneous emergence of learning algorithms”
“Self-supervised Learning through the Eyes of a Child”, Orhan et al 2020
“Hopfield Networks Is All You Need”, Ramsauer et al 2020
“On Robustness and Transferability of Convolutional Neural Networks”, Djolonga et al 2020
“On Robustness and Transferability of Convolutional Neural Networks”
“ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Deep Learning and High Performance Computing”, Elnaggar et al 2020
“NVAE: A Deep Hierarchical Variational Autoencoder”, Vahdat & Kautz 2020
“Measuring Robustness to Natural Distribution Shifts in Image Classification”, Taori et al 2020
“Measuring Robustness to Natural Distribution Shifts in Image Classification”
“Is SGD a Bayesian Sampler? Well, Almost”, Mingard et al 2020
“Unsupervised Cross-lingual Representation Learning for Speech Recognition”, Conneau et al 2020
“Unsupervised Cross-lingual Representation Learning for Speech Recognition”
“Logarithmic Pruning Is All You Need”, Orseau et al 2020
“Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations”, Baevski et al 2020
“wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations”
“Denoising Diffusion Probabilistic Models”, Ho et al 2020
“GPT-3 Creative Fiction”, Gwern 2020
“On the Predictability of Pruning Across Scales”, Rosenfeld et al 2020
“Image GPT (iGPT): We Find That, Just As a Large Transformer Model Trained on Language Can Generate Coherent Text, the Same Exact Model Trained on Pixel Sequences Can Generate Coherent Image Completions and Samples”, Chen et al 2020
“SimCLRv2: Big Self-Supervised Models Are Strong Semi-Supervised Learners”, Chen et al 2020
“SimCLRv2: Big Self-Supervised Models are Strong Semi-Supervised Learners”
“SwAV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments”, Caron et al 2020
“SwAV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments”
“IGPT: Generative Pretraining from Pixels”, Chen et al 2020
“Are We Done With ImageNet?”, Beyer et al 2020
“OpenAI API”, Brockman et al 2020
“How Big Should My Language Model Be?”, Scao 2020
“Object Segmentation Without Labels With Large-Scale Generative Models”, Voynov et al 2020
“Object Segmentation Without Labels with Large-Scale Generative Models”
“GPT-3 Paper § Figure F.1: Four Uncurated Completions from a Context Suggesting the Model Compose a Poem in the Style of Wallace Stevens With the Title ‘Shadows on the Way’”, GPT-3 2020 (page 48)
“The Scaling Hypothesis”, Gwern 2020
“Danny Hernandez on Forecasting and the Drivers of AI Progress”, Koehler et al 2020
“Danny Hernandez on forecasting and the drivers of AI progress”
“ZeRO-2 & DeepSpeed: Shattering Barriers of Deep Learning Speed & Scale”, Team 2020
“ZeRO-2 & DeepSpeed: Shattering barriers of deep learning speed & scale”
“Powered by AI: Advancing Product Understanding and Building New Shopping Experiences”, Berg et al 2020
“Powered by AI: Advancing product understanding and building new shopping experiences”
“Measuring the Algorithmic Efficiency of Neural Networks”, Hernandez & Brown 2020
“Pushing the Limit of Molecular Dynamics With ab Initio Accuracy to 100 Million Atoms With Machine Learning”, Jia et al 2020
“Jukebox: We’re Introducing Jukebox, a Neural Net That Generates Music, including Rudimentary Singing, As Raw Audio in a Variety of Genres and Artist Styles. We’re Releasing the Model Weights and Code, along With a Tool to Explore the Generated Samples.”, Dhariwal et al 2020
“Blender: A State-of-the-art Open Source Chatbot”, Roller et al 2020
“A Review of Winograd Schema Challenge Datasets and Approaches”, Kocijan et al 2020
“A Review of Winograd Schema Challenge Datasets and Approaches”
“DynamicEmbedding: Extending TensorFlow for Colossal-Scale Applications”, Zeng et al 2020
“DynamicEmbedding: Extending TensorFlow for Colossal-Scale Applications”
“PALM: Pre-training an Autoencoding & Autoregressive Language Model for Context-conditioned Generation”, Bi et al 2020
“Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems”, Naumov et al 2020
“Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems”
“A Metric Learning Reality Check”, Musgrave et al 2020
“TTTTTackling WinoGrande Schemas”, Lin et al 2020
“Zoom In: An Introduction to Circuits—By Studying the Connections between Neurons, We Can Find Meaningful Algorithms in the Weights of Neural Networks”, Olah et al 2020
“Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited”, Maddox et al 2020
“Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited”
“Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers”, Li et al 2020
“Rethinking Bias-Variance Trade-off for Generalization of Neural Networks”, Yang et al 2020
“Rethinking Bias-Variance Trade-off for Generalization of Neural Networks”
“The Messy, Secretive Reality behind OpenAI’s Bid to save the World: The AI Moonshot Was Founded in the Spirit of Transparency. This Is the inside Story of How Competitive Pressure Eroded That Idealism”, Hao 2020
“A Simple Framework for Contrastive Learning of Visual Representations”, Chen et al 2020
“A Simple Framework for Contrastive Learning of Visual Representations”
“Turing-NLG: A 17-billion-parameter Language Model by Microsoft”, Rosset 2020
“Turing-NLG: A 17-billion-parameter language model by Microsoft”
“How Much Knowledge Can You Pack Into the Parameters of a Language Model?”, Roberts et al 2020
“How Much Knowledge Can You Pack Into the Parameters of a Language Model?”
“Impact of ImageNet Model Selection on Domain Adaptation”, Zhang & Davison 2020
“Direct Fit to Nature: An Evolutionary Perspective on Biological and Artificial Neural Networks”, Hasson et al 2020
“Direct Fit to Nature: An Evolutionary Perspective on Biological and Artificial Neural Networks”
“Towards a Conversational Agent That Can Chat About…Anything”, Adiwardana & Luong 2020
“Towards a Conversational Agent that Can Chat About…Anything”
“Towards a Human-like Open-Domain Chatbot”, Adiwardana et al 2020
“Scaling Laws for Neural Language Models”, Kaplan et al 2020
“Big Transfer (BiT): General Visual Representation Learning”, Kolesnikov et al 2019
“Big Transfer (BiT): General Visual Representation Learning”
“Deep Double Descent: We Show That the Double Descent Phenomenon Occurs in CNNs, ResNets, and Transformers: Performance First Improves, Then Gets Worse, and Then Improves Again With Increasing Model Size, Data Size, or Training Time. This Effect Is Often Avoided through Careful Regularization. While This Behavior Appears to Be Fairly Universal, We Don’t yet Fully Understand Why It Happens, and View Further Study of This Phenomenon As an Important Research Direction.”, Nakkiran et al 2019
“12-in-1: Multi-Task Vision and Language Representation Learning”, Lu et al 2019
“12-in-1: Multi-Task Vision and Language Representation Learning”
“Deep Double Descent: Where Bigger Models and More Data Hurt”, Nakkiran et al 2019
“Deep Double Descent: Where Bigger Models and More Data Hurt”
“Understanding the Generalization of ‘lottery Tickets’ in Neural Networks”, Morcos & Tian 2019
“Understanding the generalization of ‘lottery tickets’ in neural networks”
“Momentum Contrast for Unsupervised Visual Representation Learning”, He et al 2019
“Momentum Contrast for Unsupervised Visual Representation Learning”
“The Deep Learning Revolution and Its Implications for Computer Architecture and Chip Design”, Dean 2019
“The Deep Learning Revolution and Its Implications for Computer Architecture and Chip Design”
“Self-training With Noisy Student Improves ImageNet Classification”, Xie et al 2019
“Self-training with Noisy Student improves ImageNet classification”
“CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs”, El-Kishky et al 2019
“CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs”
“CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB”, Schwenk et al 2019
“CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB”
“XLM-R: State-of-the-art Cross-lingual Understanding through Self-supervision”, FAIR 2019
“XLM-R: State-of-the-art cross-lingual understanding through self-supervision”
“Unsupervised Cross-lingual Representation Learning at Scale”, Conneau et al 2019
“Unsupervised Cross-lingual Representation Learning at Scale”
“High Fidelity Video Prediction With Large Stochastic Recurrent Neural Networks”, Villegas et al 2019
“High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks”
“T5: Exploring the Limits of Transfer Learning With a Unified Text-to-Text Transformer”, Raffel et al 2019
“T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”
“ZeRO: Memory Optimizations Toward Training Trillion Parameter Models”, Rajbhandari et al 2019
“ZeRO: Memory Optimizations Toward Training Trillion Parameter Models”
“Environmental Drivers of Systematicity and Generalization in a Situated Agent”, Hill et al 2019
“Environmental drivers of systematicity and generalization in a situated agent”
“A Constructive Prediction of the Generalization Error Across Scales”, Rosenfeld et al 2019
“A Constructive Prediction of the Generalization Error Across Scales”
“Large-scale Pretraining for Neural Machine Translation With Tens of Billions of Sentence Pairs”, Meng et al 2019
“Large-scale Pretraining for Neural Machine Translation with Tens of Billions of Sentence Pairs”
“UNITER: UNiversal Image-TExt Representation Learning”, Chen et al 2019
“Exascale Deep Learning for Scientific Inverse Problems”, Laanait et al 2019
“Simple, Scalable Adaptation for Neural Machine Translation”, Bapna et al 2019
“Simple, Scalable Adaptation for Neural Machine Translation”
“CTRL: A Conditional Transformer Language Model For Controllable Generation”, Keskar et al 2019
“CTRL: A Conditional Transformer Language Model For Controllable Generation”
“Show Your Work: Improved Reporting of Experimental Results”, Dodge et al 2019
“Show Your Work: Improved Reporting of Experimental Results”
“MegatronLM: Training Billion+ Parameter Language Models Using GPU Model Parallelism”, ADLR 2019
“MegatronLM: Training Billion+ Parameter Language Models Using GPU Model Parallelism”
“RoBERTa: A Robustly Optimized BERT Pretraining Approach”, Liu et al 2019
“Robustness Properties of Facebook’s ResNeXt WSL Models”, Orhan 2019
“Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges”, Arivazhagan et al 2019
“Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges”
“Large Scale Adversarial Representation Learning”, Donahue & Simonyan 2019
“One Epoch Is All You Need”, Komatsuzaki 2019
“Does Learning Require Memorization? A Short Tale about a Long Tail”, Feldman 2019
“Does Learning Require Memorization? A Short Tale about a Long Tail”
“Intriguing Properties of Adversarial Training at Scale”, Xie & Yuille 2019
“Scaling Autoregressive Video Models”, Weissenborn et al 2019
“A Mathematical Theory of Semantic Development in Deep Neural Networks”, Saxe et al 2019
“A mathematical theory of semantic development in deep neural networks”
“Adversarially Robust Generalization Just Requires More Unlabeled Data”, Zhai et al 2019
“Adversarially Robust Generalization Just Requires More Unlabeled Data”
“ICML 2019 Notes”, Abel 2019
“SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers”, Fedorov et al 2019
“SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers”
“EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks”, Tan & Le 2019
“EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks”
“UniLM: Unified Language Model Pre-training for Natural Language Understanding and Generation”, Dong et al 2019
“UniLM: Unified Language Model Pre-training for Natural Language Understanding and Generation”
“Billion-scale Semi-supervised Learning for Image Classification”, Yalniz et al 2019
“Billion-scale semi-supervised learning for image classification”
“VideoBERT: A Joint Model for Video and Language Representation Learning”, Sun et al 2019
“VideoBERT: A Joint Model for Video and Language Representation Learning”
“Benchmarking Neural Network Robustness to Common Corruptions and Perturbations”, Hendrycks & Dietterich 2019
“Benchmarking Neural Network Robustness to Common Corruptions and Perturbations”
“Surprises in High-Dimensional Ridgeless Least Squares Interpolation”, Hastie et al 2019
“Surprises in High-Dimensional Ridgeless Least Squares Interpolation”
“The Bitter Lesson”, Sutton 2019
“Deep Learning Hardware: Past, Present, & Future”, LeCun 2019
“Better Language Models and Their Implications”, Radford et al 2019
“Language Models Are Unsupervised Multitask Learners”, Radford et al 2019
“Do ImageNet Classifiers Generalize to ImageNet?”, Recht et al 2019
“Cross-lingual Language Model Pretraining”, Lample & Conneau 2019
“High Fidelity Video Prediction With Large Stochastic Recurrent Neural Networks: Videos”, Villegas et al 2019
“High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks: Videos”
“Artificial Intelligence: A Guide for Thinking Humans § Prologue: Terrified”, Mitchell 2019
“Artificial Intelligence: A Guide for Thinking Humans § Prologue: Terrified”
“Reconciling Modern Machine Learning Practice and the Bias-variance Trade-off”, Belkin et al 2018
“Reconciling modern machine learning practice and the bias-variance trade-off”
“Nocaps: Novel Object Captioning at Scale”, Agrawal et al 2018
“How AI Training Scales”, McCandlish et al 2018
“Is Science Slowing Down?”, Alexander 2018
“WBE and DRL: a Middle Way of Imitation Learning from the Human Brain”, Branwen 2018
“WBE and DRL: a Middle Way of imitation learning from the human brain”
“BigGAN: Large Scale GAN Training For High Fidelity Natural Image Synthesis § 5.2 Additional Evaluation On JFT-300M”, Brock et al 2018 (page 8 org deepmind)
“Large Scale GAN Training for High Fidelity Natural Image Synthesis”, Brock et al 2018
“Large Scale GAN Training for High Fidelity Natural Image Synthesis”
“Measurement Invariance Explains the Universal Law of Generalization for Psychological Perception”, Frank 2018
“Measurement invariance explains the universal law of generalization for psychological perception”
“CurriculumNet: Weakly Supervised Learning from Large-Scale Web Images”, Guo et al 2018
“CurriculumNet: Weakly Supervised Learning from Large-Scale Web Images”
“Large-Scale Visual Speech Recognition”, Shillingford et al 2018
“Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations”, Hendrycks & Dietterich 2018
“Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations”
“Neural Scene Representation and Rendering”, Eslami et al 2018
“GPT-1: Improving Language Understanding With Unsupervised Learning”, OpenAI 2018
“GPT-1: Improving Language Understanding with Unsupervised Learning”
“GPT-1: Improving Language Understanding by Generative Pre-Training § Model Specifications”, Radford et al 2018 (page 5)
“GPT-1: Improving Language Understanding by Generative Pre-Training § Model specifications”
“GPT-1: Improving Language Understanding by Generative Pre-Training”, Radford et al 2018
“GPT-1: Improving Language Understanding by Generative Pre-Training”
“Do CIFAR-10 Classifiers Generalize to CIFAR-10?”, Recht et al 2018
“Deep Learning Generalizes Because the Parameter-function Map Is Biased towards Simple Functions”, Valle-Pérez et al 2018
“Deep learning generalizes because the parameter-function map is biased towards simple functions”
“Google DeepMind Founder and Leader in Artificial Intelligence Returns to Hamilton”, Tantau 2018
“Google DeepMind founder and leader in artificial intelligence returns to Hamilton”
“Exploring the Limits of Weakly Supervised Pretraining”, Mahajan et al 2018
“One Big Net For Everything”, Schmidhuber 2018
“Sensitivity and Generalization in Neural Networks: an Empirical Study”, Novak et al 2018
“Sensitivity and Generalization in Neural Networks: an Empirical Study”
“ULMFiT: Universal Language Model Fine-tuning for Text Classification”, Howard & Ruder 2018
“ULMFiT: Universal Language Model Fine-tuning for Text Classification”
“Deep Image Reconstruction from Human Brain Activity”, Shen et al 2017
“Deep Learning Scaling Is Predictable, Empirically”, Hestness et al 2017
“Are GANs Created Equal? A Large-Scale Study”, Lucic et al 2017
“Knowledge Concentration: Learning 100K Object Classifiers in a Single CNN”, Gao et al 2017
“Knowledge Concentration: Learning 100K Object Classifiers in a Single CNN”
“Rethinking Generalization Requires Revisiting Old Ideas: Statistical Mechanics Approaches and Complex Learning Behavior”, Martin & Mahoney 2017
“There’s No Fire Alarm for Artificial General Intelligence”, Yudkowsky 2017
“WebVision Database: Visual Learning and Understanding from Web Data”, Li et al 2017
“WebVision Database: Visual Learning and Understanding from Web Data”
“Revisiting Unreasonable Effectiveness of Data in Deep Learning Era”, Sun et al 2017
“Revisiting Unreasonable Effectiveness of Data in Deep Learning Era”
“Towards Deep Learning Models Resistant to Adversarial Attacks”, Madry et al 2017
“Towards Deep Learning Models Resistant to Adversarial Attacks”
“Learning to Learn from Noisy Web Videos”, Yeung et al 2017
“Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour”, Goyal et al 2017
“Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour”
“A Simple Neural Network Module for Relational Reasoning”, Santoro et al 2017
“Deep Learning Is Robust to Massive Label Noise”, Rolnick et al 2017
“Quo Vadis, Action Recognition? A New Model I3D and the Kinetics Dataset”, Carreira & Zisserman 2017
“Quo Vadis, Action Recognition? A New Model I3D and the Kinetics Dataset”
“WebVision Challenge: Visual Learning and Understanding With Web Data”, Li et al 2017
“WebVision Challenge: Visual Learning and Understanding With Web Data”
“Geometry of Optimization and Implicit Regularization in Deep Learning”, Neyshabur et al 2017
“Geometry of Optimization and Implicit Regularization in Deep Learning”
“On the Impossibility of Supersized Machines”, Garfinkel et al 2017
“Research Ideas”, Gwern 2017
“Parallel Multiscale Autoregressive Density Estimation”, Reed et al 2017
“Universal Representations:The Missing Link between Faces, Text, Planktons, and Cat Breeds”, Bilen & Vedaldi 2017
“Universal representations:The missing link between faces, text, planktons, and cat breeds”
“Estimation of Gap Between Current Language Models and Human Performance”, Shen et al 2017
“Estimation of Gap Between Current Language Models and Human Performance”
“Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles”, Lakshminarayanan et al 2016
“Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles”
“Understanding Deep Learning Requires Rethinking Generalization”, Zhang et al 2016
“Understanding deep learning requires rethinking generalization”
“Ra”, Constantin 2016
“The LAMBADA Dataset: Word Prediction Requiring a Broad Discourse Context”, Paperno et al 2016
“The LAMBADA dataset: Word prediction requiring a broad discourse context”
“Do Deep Convolutional Nets Really Need to Be Deep and Convolutional?”, Urban et al 2016
“Do Deep Convolutional Nets Really Need to be Deep and Convolutional?”
“PlaNet—Photo Geolocation With Convolutional Neural Networks”, Weyand et al 2016
“PlaNet—Photo Geolocation with Convolutional Neural Networks”
“Exploring the Limits of Language Modeling”, Jozefowicz et al 2016
“The Singularity: A Philosophical Analysis”, Chalmers 2016
“The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition”, Krause et al 2015
“The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition”
“Net2Net: Accelerating Learning via Knowledge Transfer”, Chen et al 2015
“Generative Concatenative Nets Jointly Learn to Write and Classify Reviews”, Lipton et al 2015
“Generative Concatenative Nets Jointly Learn to Write and Classify Reviews”
“Learning Visual Features from Large Weakly Supervised Data”, Arm et al 2015
“Learning Visual Features from Large Weakly Supervised Data”
“LSUN: Construction of a Large-scale Image Dataset Using Deep Learning With Humans in the Loop”, Yu et al 2015
“LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop”
“Clothing-1M: Learning from Massive Noisy Labeled Data for Image Classification”, Xiao et al 2015
“Clothing-1M: Learning from Massive Noisy Labeled Data for Image Classification”
“The Unreasonable Effectiveness of Recurrent Neural Networks”, Karpathy 2015
“The Unreasonable Effectiveness of Recurrent Neural Networks”
“YFCC100M: The New Data in Multimedia Research”, Thomee et al 2015
“Evolution of the Human Brain: From Matter to Mind”, Hofman 2015
“In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning”, Neyshabur et al 2014
“In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning”
“Technology Forecasting: The Garden of Forking Paths”, Gwern 2014
“Jumping NLP Curves: A Review of Natural Language Processing Research [Review Article]”, Cambria & White 2014
“Jumping NLP Curves: A Review of Natural Language Processing Research [Review Article]”
“Neural Networks, Manifolds, and Topology”, Olah 2014
“Computing’s Energy Problem (and What We Can Do about It)”, Horowitz 2014b
“N-gram Counts and Language Models from the Common Crawl”, Buck et al 2014
“One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling”, Chelba et al 2013
“One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling”
“Algorithmic Progress in Six Domains”, Grace 2013
“Large–Scale Machine Learning Revisited [slides]”, Bottou 2013
“Intelligence Explosion Microeconomics”, Yudkowsky 2013
“Scalable Modified Kneser-Ney Language Model Estimation”, Heafield et al 2013
“The Remarkable, yet Not Extraordinary, Human Brain As a Scaled-up Primate Brain and Its Associated Cost”, Herculano-Houzel 2012
“Advantages of Artificial Intelligences, Uploads, and Digital Minds”, Sotala 2012
“Advantages of Artificial Intelligences, Uploads, and Digital Minds”
“Recurrent Neural Network Based Language Model”, Mikolov et al 2010
“How Complex Are Individual Differences?”, Gwern 2010
“Understanding Sources of Inefficiency in General-purpose Chips”, Hameed et al 2010
“Understanding sources of inefficiency in general-purpose chips”
“The Teenies”, Legg 2009
“Tick, Tock, Tick, Tock… BING”, Legg 2009
“Halloween Nightmare Scenario, Early 2020’s”, Wood 2009
“The Unreasonable Effectiveness of Data”, Halevy et al 2009
“Economics Of The Singularity: Stuffed into Skyscrapers by the Billion, Brainy Bugbots Will Be the Knowledge Workers of the Future”, Hanson 2008
“Large Language Models in Machine Translation”, Brants et al 2007
“Cellular Scaling Rules for Primate Brains”, Herculano-Houzel et al 2007
“The Tradeoffs of Large-Scale Learning”, Bottou & Bousquet 2007
“Robot Predictions Evolution”, Moravec 2004
“Tree Induction vs. Logistic Regression: A Learning-Curve Analysis”, Perlich et al 2003
“Tree Induction vs. Logistic Regression: A Learning-Curve Analysis”
“Scaling to Very Very Large Corpora for Natural Language Disambiguation”, Banko & Brill 2001
“Scaling to Very Very Large Corpora for Natural Language Disambiguation”
“On Discriminative vs. Generative Classifiers: A Comparison of Logistic Regression and Naive Bayes”, Ng & Jordan 2001
“On Discriminative vs. Generative Classifiers: A comparison of logistic regression and naive Bayes”
“A Survey of Methods for Scaling Up Inductive Algorithms”, Provost & Kolluri 1999
“On The Effect of Data Set Size on Bias And Variance in Classification Learning”, Brain & Webb 1999
“On The Effect of Data Set Size on Bias And Variance in Classification Learning”
“The Effects of Training Set Size on Decision Tree Complexity”, Oates & Jensen 1997
“The Effects of Training Set Size on Decision Tree Complexity”
“Rigorous Learning Curve Bounds from Statistical Mechanics”, Haussler et al 1996
“Scaling up the Accuracy of Naive-Bayes Classifiers: a Decision-tree Hybrid”, Kohavi 1996
“Scaling up the accuracy of Naive-Bayes classifiers: a decision-tree hybrid”
“Reflections After Refereeing Papers for NIPS”, Breiman 1995
“Building a Large Annotated Corpus of English: The Penn Treebank”, Marcus et al 1993
“Building a Large Annotated Corpus of English: The Penn Treebank”
“Statistical Theory of Learning Curves under Entropic Loss Criterion”, Amari & Murata 1993
“Statistical Theory of Learning Curves under Entropic Loss Criterion”
“Learning Curves: Asymptotic Values and Rate of Convergence”, Cortes et al 1993
“Learning Curves: Asymptotic Values and Rate of Convergence”
“Exhaustive Learning”, Schwartz et al 1990
“Computing With Connections”, Sejnowski 1987
“The Role Of RAW POWER In INTELLIGENCE”, Moravec 1976
“Don’t Worry—It Can’t Happen”, Harrington 1940
“Homepage of Paul F. Christiano”, Christiano 2023
“Ilya Sutskever: Deep Learning | AI Podcast #94 With Lex Fridman”
“Ilya Sutskever: Deep Learning | AI Podcast #94 with Lex Fridman”
“A Universal Law of Robustness”
“A Law of Robustness and the Importance of Overparameterization in Deep Learning”
“A law of robustness and the importance of overparameterization in deep learning”
Sort By Magic
Annotations sorted by machine learning into inferred 'tags'. This provides an alternative way to browse: instead of by date order, one can browse in topic order. The 'sorted' list has been automatically clustered into multiple sections & auto-labeled for easier browsing.
Beginning with the newest annotation, it uses the embedding of each annotation to attempt to create a list of nearest-neighbor annotations, creating a progression of topics. For more details, see the link.
language-models
generalization
ai-research
multimodal
neuralnetworks
deeplearning
neuralscale
scalingup
Wikipedia
Miscellaneous
-
/doc/ai/nn/fully-connected/2023-bachmann-figure7-suprachinchilladatascalingformlpsoncifar100loss.png -
/doc/ai/nn/fully-connected/2023-bachmann-figure5-scalingofmlpsoncifar10andimagenet1k.png -
/doc/ai/nn/fully-connected/2023-bachmann-figure4-mlpsscalewellwithincreasingbatchsize.png -
/doc/ai/nn/fully-connected/2023-bachmann-figure1-mlpcomputescalingoncifar100.png -
/doc/ai/scaling/2023-eldan-figure23-scalinglawoftinystoriesgpttransformermodelswithtrainingflops.png -
/doc/ai/scaling/2022-zhu-figure9-webface260mcnnfacerecognitionscalingbyn.png -
/doc/ai/scaling/2021-10-11-xinzhiyuan-inspursource10gpt245b.html -
/doc/ai/scaling/2021-zhang-figure2-conformerpmodelworderrorscalingratesindatasetsize.png -
/doc/ai/scaling/2021-zhang-figure1a-conformermodelworderrorscalingindatasetsize.png -
/doc/ai/scaling/2021-hu-figure2-a-datascalingfinetuningperformanceonmscoco.png -
/doc/ai/scaling/2021-hu-figure1-lemontransformerscalingonmscocoimagecaptioning.png -
/doc/ai/scaling/2021-goyal-figure1-seerscalinginparameters.png -
/doc/ai/scaling/2020-rosset-turingnlg-nlpmodelparametercountovertime.png -
/doc/ai/scaling/2020-chrisdyer-aacl2020-machinetranslationscaling-ngramsvsrnns.jpg -
/doc/ai/scaling/2020-carlsmith-figure5-flopsbudgetestimates.png -
/doc/ai/scaling/2019-liu-table4-robertabenefitsfromscalingdatasets10xoverbert.png -
/doc/ai/scaling/2018-howard-figure3-datascalingofrnnpretrainingfortextclassification.png -
/doc/ai/scaling/2017-koehn-figure3-bleuscoreswithvaryingamountsoftrainingdata.png -
/doc/ai/scaling/2014-cambria-figure1-hypotheticalnlpprogresscurves.png -
/doc/ai/scaling/2012-bottou-figure13-2-sgdtrainingtimetestlossvsconjugategradients.png -
/doc/ai/scaling/2012-bottou-figure13-1-sgdtrainingtimetestlossvstron.png -
/doc/ai/scaling/2009-12-07-shanelegg-supercomputerlinpackoverpast50years.png -
https://blog.research.google/2018/06/scalable-deep-reinforcement-learning.html -
https://blog.research.google/2022/04/large-scale-matrix-factorization-on-tpus.html -
https://en.chessbase.com/post/komodo-8-the-smartphone-vs-desktop-challenge -
https://lilianweng.github.io/lil-log/2021/12/05/semi-supervised-learning.html -
https://nonint.com/2023/06/10/the-it-in-ai-models-is-the-dataset/ -
https://rootnodes.substack.com/p/why-didnt-deepmind-build-gpt3 -
https://towardsdatascience.com/neural-networks-are-fundamentally-bayesian-bee9a172fad8 -
https://twitter.com/andrewwhite01/status/1634728559506870274 -
https://twitter.com/olivercameron/status/1622802466470514688 -
https://venturebeat.com/business/openai-disbands-its-robotics-research-team/ -
https://www.lesswrong.com/posts/3daPPjWbjYNP6nbre/appendix-more-is-different-in-other-domains -
https://www.lesswrong.com/posts/FRv7ryoqtvSuqBxuT/understanding-deep-double-descent -
https://www.lesswrong.com/posts/KbRxdBCcJqwtbiPzm/whisper-s-wild-implications-1 -
https://www.lesswrong.com/posts/No5JpRCHzBrWA4jmS/q-and-a-with-shane-legg-on-risks-from-ai -
https://www.lesswrong.com/posts/Q3XaZTExzDpCLr4wu/efficiency-and-resource-use-scaling-parity -
https://www.lesswrong.com/posts/fnjKpBoWJXcSDwhZk/what-s-the-backward-forward-flop-ratio-for-nns -
https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality-is-the-tiger-and-agents-are-its-teeth -
https://www.lesswrong.com/posts/qdStMFDMrWAnTqNWL/gpt-4-predictions
Link Bibliography
-
https://arxiv.org/abs/2311.04145#alibaba: “I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models”, Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, Jingren Zhou -
https://arxiv.org/abs/2310.16764#deepmind: “ConvNets Match Vision Transformers at Scale”, Samuel L. Smith, Andrew Brock, Leonard Berrada, Soham De -
https://arxiv.org/abs/2310.09199#google: “PaLI-3 Vision Language Models: Smaller, Faster, Stronger”, -
https://arxiv.org/abs/2310.06694: “Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning”, Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, Danqi Chen -
https://arxiv.org/abs/2310.06213: “GeoLLM: Extracting Geospatial Knowledge from Large Language Models”, Rohin Manvi, Samar Khanna, Gengchen Mai, Marshall Burke, David Lobell, Stefano Ermon -
https://arxiv.org/abs/2310.03214#google: “FreshLLMs: Refreshing Large Language Models With Search Engine Augmentation”, -
https://arxiv.org/abs/2308.11596#facebook: “SeamlessM4T: Massively Multilingual & Multimodal Machine Translation”, -
https://arxiv.org/abs/2308.03958#deepmind: “Simple Synthetic Data Reduces Sycophancy in Large Language Models”, Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, Quoc V. Le -
aunn: “Absolute Unit NNs: Regression-Based MLPs for Everything”, Gwern -
https://openai.com/blog/introducing-superalignment: “Introducing Superalignment”, Jan Leike, Ilya Sutskever -
https://www.youtube.com/watch?v=lfXxzAVtdpU&t=1763s: “Gödel, Escher, Bach Author Douglas Hofstadter on the State of AI Today § What about AI Terrifies You?”, Douglas Hofstadter, Amy Jo Kim -
https://arxiv.org/abs/2306.13575: “Scaling MLPs: A Tale of Inductive Bias”, Gregor Bachmann, Sotiris Anagnostidis, Thomas Hofmann -
https://arxiv.org/abs/2305.15717: “The False Promise of Imitating Proprietary LLMs”, Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, Dawn Song -
https://arxiv.org/abs/2305.11863: “Scaling Laws for Language Encoding Models in FMRI”, Richard Antonello, Aditya Vaidya, Alexander G. Huth -
https://www.cnbc.com/2023/05/16/googles-palm-2-uses-nearly-five-times-more-text-data-than-predecessor.html: “Google's Newest A.I. Model Uses Nearly 5× More Text Data for Training Than Its Predecessor”, Jennifer Elias -
https://arxiv.org/abs/2305.07759#microsoft: “TinyStories: How Small Can Language Models Be and Still Speak Coherent English?”, Ronen Eldan, Yuanzhi Li -
https://arxiv.org/abs/2305.05665#facebook: “ImageBind: One Embedding Space To Bind Them All”, Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Arm, Joulin, Ishan Misra -
https://arxiv.org/abs/2304.07193#facebook: “DINOv2: Learning Robust Visual Features without Supervision”, -
https://arxiv.org/abs/2303.15343#google: “Sigmoid Loss for Language Image Pre-Training”, Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, Lucas Beyer -
https://arxiv.org/abs/2304.02015#alibaba: “How Well Do Large Language Models Perform in Arithmetic Tasks?”, Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang -
https://jameswphillips.substack.com/p/securing-liberal-democratic-control: “Securing Liberal Democratic Control of AGI through UK Leadership”, James W. Phillips -
https://arxiv.org/abs/2302.05442#google: “Scaling Vision Transformers to 22 Billion Parameters”, -
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4335945: “Large Language Models As Fiduciaries: A Case Study Toward Robustly Communicating With Artificial Intelligence Through Legal Standards”, John Nay -
https://arxiv.org/abs/2301.09515#nvidia: “StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-to-Image Synthesis”, Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, Timo Aila -
https://arxiv.org/abs/2301.07088#bytedance: “MUG: Vision Learners Meet Web Image-Text Pairs”, Bingchen Zhao, Quan Cui, Hao Wu, Osamu Yoshie, Cheng Yang -
https://arxiv.org/abs/2301.04408: “GPT-3 As Knowledge Worker: A Zero-Shot Evaluation of (AI)CPA Capabilities”, Jillian Bommarito, Michael Bommarito, Daniel Martin Katz, Jessica Katz -
https://arxiv.org/abs/2301.03728#facebook: “Scaling Laws for Generative Mixed-Modal Language Models”, -
https://arxiv.org/abs/2301.02111#microsoft: “VALL-E: Neural Codec Language Models Are Zero-Shot Text to Speech Synthesizers”, -
https://arxiv.org/abs/2212.14402: “GPT-3 Takes the Bar Exam”, Michael Bommarito II, Daniel Martin Katz -
https://arxiv.org/abs/2212.09741: “One Embedder, Any Task: Instruction-Finetuned Text Embeddings (INSTRUCTOR)”, -
https://arxiv.org/abs/2212.07143: “Reproducible Scaling Laws for Contrastive Language-image Learning”, -
https://arxiv.org/abs/2212.05051: “VindLU: A Recipe for Effective Video-and-Language Pretraining”, Feng Cheng, Xizi Wang, Jie Lei, David Crandall, Mohit Bansal, Gedas Bertasius -
https://arxiv.org/abs/2212.04979#google: “Video-Text Modeling With Zero-Shot Transfer from Contrastive Captioners”, Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, Jiahui Yu -
https://ai.facebook.com/blog/multiray-large-scale-AI-models/: “MultiRay: Optimizing Efficiency for Large-scale AI Models”, Nikhil Gupta, Michael Gschwind, Don Husa, Christopher Dewan, Madian Khabsa -
https://arxiv.org/abs/2211.09085#facebook: “Galactica: A Large Language Model for Science”, -
https://arxiv.org/abs/2211.07636#baai: “EVA: Exploring the Limits of Masked Visual Representation Learning at Scale”, Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, Yue Cao -
https://www.youtube.com/watch?v=Q-TJFyUoenc&t=2444s: “Increments Podcast: #45—4 Central Fallacies of AI Research (with Melanie Mitchell)”, Melanie Mitchell, Benny Chugg -
https://arxiv.org/abs/2210.13673#nvidia: “Evaluating Parameter Efficient Learning for Generation”, -
https://arxiv.org/abs/2210.11416#google: “FLAN: Scaling Instruction-Finetuned Language Models”, -
https://arxiv.org/abs/2210.10341#microsoft: “BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining”, Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, Tie-Yan Liu -
https://arxiv.org/abs/2210.06423#microsoft: “Foundation Transformers”, -
https://arxiv.org/abs/2210.03350#allen: “Self-Ask: Measuring and Narrowing the Compositionality Gap in Language Models (Bamboogle)”, Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, Mike Lewis -
https://arxiv.org/abs/2210.02441: “Ask Me Anything (AMA): A Simple Strategy for Prompting Language Models”, -
https://arxiv.org/abs/2210.02414#baai: “GLM-130B: An Open Bilingual Pre-trained Model”, -
https://arxiv.org/abs/2208.05516: “Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP”, Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, Ludwig Schmidt -
https://arxiv.org/abs/2207.06991: “PIXEL: Language Modelling With Pixels”, Phillip Rust, Jonas F. Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de Lhoneux, Desmond Elliott -
https://arxiv.org/abs/2207.05221#anthropic: “Language Models (Mostly) Know What They Know”, -
https://arxiv.org/abs/2206.15472: “On-Device Training Under 256KB Memory”, Ji Lin, Ligeng Zhu, Wei-Ming Chen, Wei-Chen Wang, Chuang Gan, Song Han -
https://arxiv.org/abs/2206.14486: “Beyond Neural Scaling Laws: Beating Power Law Scaling via Data Pruning”, Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, Ari S. Morcos -
https://arxiv.org/abs/2206.04658#nvidia: “BigVGAN: A Universal Neural Vocoder With Large-Scale Training”, Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, Sungroh Yoon -
https://arxiv.org/abs/2206.01685: “Toward a Realistic Model of Speech Processing in the Brain With Self-supervised Learning”, -
https://arxiv.org/abs/2205.14204#google: “M3AE: Multimodal Masked Autoencoders Learn Transferable Representations”, Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurams, Sergey Levine, Pieter Abbeel -
https://arxiv.org/abs/2205.10625#google: “Least-to-Most Prompting Enables Complex Reasoning in Large Language Models”, -
https://arxiv.org/abs/2205.09073#google: “Dialog Inpainting: Turning Documents into Dialogues”, Zhuyun Dai, Arun Tejasvi Chaganty, Vincent Zhao, Aida Amini, Qazi Mamunur Rashid, Mike Green, Kelvin Guu -
https://arxiv.org/abs/2205.05131#google: “Unifying Language Learning Paradigms”, -
https://arxiv.org/abs/2205.04596#google: “When Does Dough Become a Bagel? Analyzing the Remaining Mistakes on ImageNet”, Vijay Vasudevan, Benjamin Caine, Raphael Gontijo-Lopes, Sara Fridovich-Keil, Rebecca Roelofs -
https://arxiv.org/abs/2205.03983#google: “Building Machine Translation Systems for the Next Thousand Languages”, -
https://arxiv.org/abs/2205.01917#google: “CoCa: Contrastive Captioners Are Image-Text Foundation Models”, Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, Yonghui Wu -
https://arxiv.org/abs/2205.01397: “Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)”, Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, Ludwig Schmidt -
https://arxiv.org/abs/2204.14198#deepmind: “Flamingo: a Visual Language Model for Few-Shot Learning”, -
https://arxiv.org/abs/2204.10149: “WebFace260M: A Benchmark for Million-Scale Deep Face Recognition”, -
https://www.lesswrong.com/posts/SbAgRYo8tkHwhd9Qx/deepmind-the-podcast-excerpts-on-agi: “DeepMind: The Podcast—Excerpts on AGI”, William Kiely -
https://arxiv.org/abs/2203.15556#deepmind: “Chinchilla: Training Compute-Optimal Large Language Models”, -
https://arxiv.org/abs/2203.11171#google: “Self-Consistency Improves Chain-of-Thought Reasoning in Language Models”, Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Denny Zhou -
https://arxiv.org/abs/2203.03466: “Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer”, -
https://arxiv.org/abs/2203.00854: “FastFold: Reducing AlphaFold Training Time from 11 Days to 67 Hours”, Shenggan Cheng, Ruidong Wu, Zhongming Yu, Binrui Li, Xiwen Zhang, Jian Peng, Yang You -
https://arxiv.org/abs/2202.12211#google: “Self-Distilled StyleGAN: Towards Generation from Internet Photos”, Ron Mokady, Michal Yarom, Omer Tov, Oran Lang, Daniel Cohen-Or, Tali Dekel, Michal Irani, Inbar Mosseri -
https://www.nature.com/articles/s42003-022-03036-1: “Brains and Algorithms Partially Converge in Natural Language Processing”, Charlotte Caucheteux, Jean-Rémi King -
https://arxiv.org/abs/2202.06767#huawei: “Wukong: 100 Million Large-scale Chinese Cross-modal Pre-training Dataset and A Foundation Framework”, -
https://arxiv.org/abs/2202.03052#alibaba: “OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework”, -
https://arxiv.org/abs/2202.02317#allen: “Webly Supervised Concept Expansion for General Purpose Vision Models”, Amita Kamath, Christopher Clark, Tanmay Gupta, Eric Kolve, Derek Hoiem, Aniruddha Kembhavi -
https://arxiv.org/abs/2201.11990#microsoftnvidia: “Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model”, -
https://arxiv.org/abs/2201.11473#microsoft: “Reasoning Like Program Executors”, Xinyu Pi, Qian Liu, Bei Chen, Morteza Ziyadi, Zeqi Lin, Yan Gao, Qiang Fu, Jian-Guang Lou, Weizhu Chen -
https://arxiv.org/abs/2201.10005#openai: “Text and Code Embeddings by Contrastive Pre-Training”, -
https://arxiv.org/abs/2201.08371#facebook: “SWAG: Revisiting Weakly Supervised Pre-Training of Visual Perception Models”, -
https://arxiv.org/abs/2201.06910: “ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalization”, Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Yanggang Wang, Haiyu Li, Zhilin Yang -
https://royalsocietypublishing.org/doi/10.1098/rstb.2020.0529: “The Evolution of Quantitative Sensitivity”, -
https://arxiv.org/abs/2112.07381#samsung: “You Only Need One Model for Open-domain Question Answering”, Haejun Lee, Akhil Kedia, Jongwon Lee, Ashwin Paranjape, Christopher D. Manning, Kyoung-Gu Woo -
https://arxiv.org/abs/2112.04426#deepmind: “Improving Language Models by Retrieving from Trillions of Tokens”, -
https://arxiv.org/abs/2111.12763#google: “Sparse Is Enough in Scaling Transformers”, -
https://arxiv.org/abs/2111.12233#microsoft: “LEMON: Scaling Up Vision-Language Pre-training for Image Captioning”, Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, Lijuan Wang -
https://arxiv.org/abs/2111.11904#microsoft: “Can Pre-trained Language Models Be Used to Resolve Textual and Semantic Merge Conflicts?”, Jialu Zhang, Todd Mytkowicz, Mike Kaufman, Ruzica Piskac, Shuvendu K. Lahiri -
https://arxiv.org/abs/2111.11432#microsoft: “Florence: A New Foundation Model for Computer Vision”, -
https://arxiv.org/abs/2111.11133: “L-Verse: Bidirectional Generation Between Image and Text”, -
https://arxiv.org/abs/2111.10050#google: “BASIC: Combined Scaling for Open-Vocabulary Image Classification”, -
https://arxiv.org/abs/2111.09883: “Swin Transformer V2: Scaling Up Capacity and Resolution”, -
https://arxiv.org/abs/2111.08267: “Solving Probability and Statistics Problems by Program Synthesis”, Leonard Tang, Elizabeth Ke, Nikhil Singh, Nakul Verma, Iddo Drori -
https://arxiv.org/abs/2111.06377#facebook: “MAE: Masked Autoencoders Are Scalable Vision Learners”, Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, Ross Girshick -
https://arxiv.org/abs/2111.02114#laion: “LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs”, -
https://arxiv.org/abs/2110.14168#openai: “Training Verifiers to Solve Math Word Problems”, Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, John Schulman -
https://arxiv.org/abs/2110.06990: “Scaling Laws for the Few-Shot Adaptation of Pre-trained Image Classifiers”, Gabriele Prato, Simon Guiroy, Ethan Caballero, Irina Rish, Sarath Chandar -
https://arxiv.org/abs/2110.02095#google: “Exploring the Limits of Large Scale Pre-training”, Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, Hanie Sedghi -
https://arxiv.org/abs/2109.10686#google: “Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers”, -
https://arxiv.org/abs/2109.07958: “TruthfulQA: Measuring How Models Mimic Human Falsehoods”, Stephanie Lin, Jacob Hilton, Owain Evans -
https://arxiv.org/abs/2109.02593#allen: “General-Purpose Question-Answering With Macaw”, Oyvind Tafjord, Peter Clark -
https://arxiv.org/abs/2108.08810#google: “Do Vision Transformers See Like Convolutional Neural Networks?”, Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, Alexey Dosovitskiy -
https://arxiv.org/abs/2108.07686: “Scaling Laws for Deep Learning”, Jonathan S. Rosenfeld -
https://arxiv.org/abs/2107.02137#baidu: “ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation”, -
https://arxiv.org/abs/2107.01294#allen: “Scarecrow: A Framework for Scrutinizing Machine Text”, Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A. Smith, Yejin Choi -
https://arxiv.org/abs/2106.07411: “Partial Success in Closing the Gap between Human and Machine Vision”, -
https://arxiv.org/abs/2106.09488#amazon: “Scaling Laws for Acoustic Models”, Jasha Droppo, Oguz Elibol -
https://arxiv.org/abs/2106.05237#google: “Knowledge Distillation: A Good Teacher Is Patient and Consistent”, Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, Alexander Kolesnikov -
https://arxiv.org/abs/2106.04803#google: “CoAtNet: Marrying Convolution and Attention for All Data Sizes”, Zihang Dai, Hanxiao Liu, Quoc V. Le, Mingxing Tan -
https://arxiv.org/abs/2106.04560#google: “Scaling Vision Transformers”, Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, Lucas Beyer -
https://arxiv.org/abs/2106.03004#google: “Exploring the Limits of Out-of-Distribution Detection”, Stanislav Fort, Jie Ren, Balaji Lakshminarayanan -
https://arxiv.org/abs/2106.00116: “Effect of Pre-Training Scale on Intra/Inter-Domain Full and Few-Shot Transfer Learning for Natural and Medical X-Ray Chest Images”, Mehdi Cherti, Jenia Jitsev -
https://arxiv.org/abs/2105.12806: “A Universal Law of Robustness via Isoperimetry”, Sébastien Bubeck, Mark Sellke -
https://m.koreaherald.com/view.php?ud=20210525000824#naver: “Naver Unveils First ‘hyperscale’ AI Platform”, Kang Jae-eun -
https://venturebeat.com/ai/google-details-new-ai-accelerator-chips/: “Google Details New AI Accelerator Chips”, Kyle Wiggers -
https://arxiv.org/abs/2105.01601#google: “MLP-Mixer: An All-MLP Architecture for Vision”, -
https://arxiv.org/abs/2105.00572#facebook: “XLM-R XL: Larger-Scale Transformers for Multilingual Masked Language Modeling”, Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, Alexis Conneau -
scaling: “Machine Learning Scaling”, Gwern -
fc: “Fully-Connected Neural Nets”, Gwern -
https://arxiv.org/abs/2104.08691#google: “The Power of Scale for Parameter-Efficient Prompt Tuning”, Brian Lester, Rami Al-Rfou, Noah Constant -
https://arxiv.org/abs/2103.14586#google: “Understanding Robustness of Transformers for Image Classification”, Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li, Thomas Unterthiner, Andreas Veit -
https://arxiv.org/abs/2103.13009#allen: “UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark”, Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi -
https://arxiv.org/abs/2103.10957#deepmind: “Efficient Visual Pretraining With Contrastive Detection”, Olivier J. Hénaff, Skanda Koppula, Jean-Baptiste Alayrac, Aaron van den Oord, Oriol Vinyals, João Carreira -
https://arxiv.org/abs/2103.07579#google: “Revisiting ResNets: Improved Training and Scaling Strategies”, -
https://ai.facebook.com/blog/learning-from-videos-to-understand-the-world/: “Learning from Videos to Understand the World”, Geoffrey Zweig, Polina Kuznetsova, Michael Auli, Francois Fagan -
https://arxiv.org/abs/2103.01988#facebook: “SEER: Self-supervised Pretraining of Visual Features in the Wild”, -
https://arxiv.org/abs/2102.09672#openai: “Improved Denoising Diffusion Probabilistic Models”, Alex Nichol, Prafulla Dhariwal -
https://arxiv.org/abs/2102.06171#deepmind: “NFNet: High-Performance Large-Scale Image Recognition Without Normalization”, Andrew Brock, Soham De, Samuel L. Smith, Karen Simonyan -
https://arxiv.org/abs/2102.05918#google: “ALIGN: Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision”, -
https://arxiv.org/abs/2102.02888#microsoft: “1-bit Adam: Communication Efficient Large-Scale Training With Adam’s Convergence Speed”, -
https://arxiv.org/abs/2102.01951#scaling&org=deepmind: “Mind the Gap: Assessing Temporal Generalization in Neural Language Models § Scaling”, -
https://cdn.openai.com/papers/Learning_Transferable_Visual_Models_From_Natural_Language_Supervision.pdf: “CLIP: Learning Transferable Visual Models From Natural Language Supervision”, -
https://arxiv.org/abs/2003.10580#google: “Meta Pseudo Labels”, Hieu Pham, Zihang Dai, Qizhe Xie, Minh-Thang Luong, Quoc V. Le -
https://www.alignmentforum.org/posts/k2SNji3jXaLGhBeYP/extrapolating-gpt-n-performance: “Extrapolating GPT-N Performance”, Lukas Finnveden -
https://arxiv.org/abs/2011.10650#openai: “Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images”, Rewon Child -
https://arxiv.org/abs/2010.14701#openai: “Scaling Laws for Autoregressive Generative Modeling”, -
https://arxiv.org/abs/2010.14571#google: “Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus”, Isaac Caswell, Theresa Breiner, Daan van Esch, Ankur Bapna -
https://arxiv.org/abs/2010.10504#google: “Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition”, Yu Zhang, James Qin, Daniel S. Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang, Quoc V. Le, Yonghui Wu -
https://ai.meta.com/blog/introducing-many-to-many-multilingual-machine-translation/: “The First AI Model That Translates 100 Languages without Relying on English Data”, Angela Fan -
https://arxiv.org/abs/2010.11929#google: “Vision Transformer: An Image Is Worth 16×16 Words: Transformers for Image Recognition at Scale”, -
https://www.openphilanthropy.org/research/new-report-on-how-much-computational-power-it-takes-to-match-the-human-brain/: “New Report on How Much Computational Power It Takes to Match the Human Brain”, Joseph Carlsmith -
https://arxiv.org/abs/2009.03393#openai: “Generative Language Modeling for Automated Theorem Proving”, Stanislas Polu, Ilya Sutskever -
https://arxiv.org/abs/2008.09037: “Accuracy and Performance Comparison of Video Action Recognition Approaches”, -
https://www.lesswrong.com/posts/Wnqua6eQkewL3bqsF/matt-botvinick-on-the-spontaneous-emergence-of-learning: “Matt Botvinick on the Spontaneous Emergence of Learning Algorithms”, Adam Scholl -
https://arxiv.org/abs/2008.02217: “Hopfield Networks Is All You Need”, -
https://arxiv.org/abs/2007.06225: “ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Deep Learning and High Performance Computing”, -
https://arxiv.org/abs/2007.03898#nvidia: “NVAE: A Deep Hierarchical Variational Autoencoder”, Arash Vahdat, Jan Kautz -
gpt-3: “GPT-3 Creative Fiction”, Gwern -
https://arxiv.org/abs/2006.10621: “On the Predictability of Pruning Across Scales”, Jonathan S. Rosenfeld, Jonathan Frankle, Michael Carbin, Nir Shavit -
https://openai.com/research/image-gpt: “Image GPT (iGPT): We Find That, Just As a Large Transformer Model Trained on Language Can Generate Coherent Text, the Same Exact Model Trained on Pixel Sequences Can Generate Coherent Image Completions and Samples”, Mark Chen, Alec Radford, Ilya Sutskever -
https://arxiv.org/abs/2006.09882#facebook: “SwAV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments”, Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, Armand Joulin -
2020-chen-2.pdf#openai: “IGPT: Generative Pretraining from Pixels”, Mark Chen, Alec Radford, Rewon Child, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, Ilya Sutskever -
scaling-hypothesis: “The Scaling Hypothesis”, Gwern -
https://www.microsoft.com/en-us/research/blog/zero-2-deepspeed-shattering-barriers-of-deep-learning-speed-scale/: “ZeRO-2 & DeepSpeed: Shattering Barriers of Deep Learning Speed & Scale”, DeepSpeed Team -
https://openai.com/research/jukebox: “Jukebox: We’re Introducing Jukebox, a Neural Net That Generates Music, including Rudimentary Singing, As Raw Audio in a Variety of Genres and Artist Styles. We’re Releasing the Model Weights and Code, along With a Tool to Explore the Generated Samples.”, Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, Ilya Sutskever -
https://ai.meta.com/blog/state-of-the-art-open-source-chatbot/: “Blender: A State-of-the-art Open Source Chatbot”, Stephen Roller, Jason Weston, Emily Dinan -
https://arxiv.org/abs/2004.10802: “Scaling Laws from the Data Manifold Dimension”, Utkarsh Sharma, Jared Kaplan -
https://arxiv.org/abs/2004.07159#alibaba: “PALM: Pre-training an Autoencoding & Autoregressive Language Model for Context-conditioned Generation”, Bin Bi, Chenliang Li, Chen Wu, Ming Yan, Wei Wang, Songfang Huang, Fei Huang, Luo Si -
https://www.technologyreview.com/2020/02/17/844721/ai-openai-moonshot-elon-musk-sam-altman-greg-brockman-messy-secretive-reality/: “The Messy, Secretive Reality behind OpenAI’s Bid to save the World: The AI Moonshot Was Founded in the Spirit of Transparency. This Is the inside Story of How Competitive Pressure Eroded That Idealism”, Karen Hao -
https://arxiv.org/abs/2002.05709#google: “A Simple Framework for Contrastive Learning of Visual Representations”, Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton -
https://www.microsoft.com/en-us/research/blog/turing-nlg-a-17-billion-parameter-language-model-by-microsoft/: “Turing-NLG: A 17-billion-parameter Language Model by Microsoft”, Corby Rosset -
https://blog.research.google/2020/01/towards-conversational-agent-that-can.html: “Towards a Conversational Agent That Can Chat About…Anything”, Daniel Adiwardana, Thang Luong -
https://arxiv.org/abs/2001.08361#openai: “Scaling Laws for Neural Language Models”, -
https://openai.com/research/deep-double-descent: “Deep Double Descent: We Show That the Double Descent Phenomenon Occurs in CNNs, ResNets, and Transformers: Performance First Improves, Then Gets Worse, and Then Improves Again With Increasing Model Size, Data Size, or Training Time. This Effect Is Often Avoided through Careful Regularization. While This Behavior Appears to Be Fairly Universal, We Don’t yet Fully Understand Why It Happens, and View Further Study of This Phenomenon As an Important Research Direction.”, Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, Ilya Sutskever -
https://arxiv.org/abs/1911.05722#facebook: “Momentum Contrast for Unsupervised Visual Representation Learning”, Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, Ross Girshick -
https://arxiv.org/abs/1911.04252#google: “Self-training With Noisy Student Improves ImageNet Classification”, Qizhe Xie, Minh-Thang Luong, Eduard Hovy, Quoc V. Le -
https://arxiv.org/abs/1911.02116#facebook: “Unsupervised Cross-lingual Representation Learning at Scale”, -
https://arxiv.org/abs/1910.02054#microsoft: “ZeRO: Memory Optimizations Toward Training Trillion Parameter Models”, Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He -
https://arxiv.org/abs/1909.11740: “UNITER: UNiversal Image-TExt Representation Learning”, Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, Jingjing Liu -
https://arxiv.org/abs/1909.05858#salesforce: “CTRL: A Conditional Transformer Language Model For Controllable Generation”, Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, Richard Socher (Salesforce) -
https://nv-adlr.github.io/MegatronLM: “MegatronLM: Training Billion+ Parameter Language Models Using GPU Model Parallelism”, NVID I. A. ADLR -
https://arxiv.org/abs/1907.11692#facebook: “RoBERTa: A Robustly Optimized BERT Pretraining Approach”, -
https://arxiv.org/abs/1906.06669: “One Epoch Is All You Need”, Aran Komatsuzaki -
https://david-abel.github.io/notes/icml_2019.pdf: “ICML 2019 Notes”, David Abel -
https://arxiv.org/abs/1905.11946#google: “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks”, Mingxing Tan, Quoc V. Le -
https://arxiv.org/abs/1905.03197: “UniLM: Unified Language Model Pre-training for Natural Language Understanding and Generation”, Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon -
https://arxiv.org/abs/1905.00546#facebook: “Billion-scale Semi-supervised Learning for Image Classification”, I. Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, Dhruv Mahajan -
https://openai.com/research/better-language-models: “Better Language Models and Their Implications”, Alec Radford, Jeffrey Wu, Dario Amodei, Daniela Amodei, Jack Clark, Miles Brundage, Ilya Sutskever -
https://melaniemitchell.me/aibook/: “Artificial Intelligence: A Guide for Thinking Humans § Prologue: Terrified”, Melanie Mitchell -
https://openai.com/research/how-ai-training-scales: “How AI Training Scales”, Sam McCandlish, Jared Kaplan, Dario Amodei -
https://slatestarcodex.com/2018/11/26/is-science-slowing-down-2/: “Is Science Slowing Down?”, Scott Alexander -
https://arxiv.org/abs/1808.01097: “CurriculumNet: Weakly Supervised Learning from Large-Scale Web Images”, Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang, Dengke Dong, Matthew R. Scott, Dinglong Huang -
https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf#page=5: “GPT-1: Improving Language Understanding by Generative Pre-Training § Model Specifications”, Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever -
https://arxiv.org/abs/1805.00932#facebook: “Exploring the Limits of Weakly Supervised Pretraining”, -
https://arxiv.org/abs/1801.06146: “ULMFiT: Universal Language Model Fine-tuning for Text Classification”, Jeremy Howard, Sebastian Ruder -
https://arxiv.org/abs/1706.06083: “Towards Deep Learning Models Resistant to Adversarial Attacks”, Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, Adrian Vladu -
https://arxiv.org/abs/1706.01427#deepmind: “A Simple Neural Network Module for Relational Reasoning”, -
https://arxiv.org/abs/1705.07750#deepmind: “Quo Vadis, Action Recognition? A New Model I3D and the Kinetics Dataset”, Joao Carreira, Andrew Zisserman -
https://arxiv.org/abs/1705.05640: “WebVision Challenge: Visual Learning and Understanding With Web Data”, Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, Jesse Berent, Abhinav Gupta, Rahul Sukthankar, Luc Van Gool -
idea: “Research Ideas”, Gwern -
https://arxiv.org/abs/1511.06789#google: “The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition”, -
https://arxiv.org/abs/1511.02251#facebook: “Learning Visual Features from Large Weakly Supervised Data”, Arm, Joulin, Laurens van der Maaten, Allan Jabri, Nicolas Vasilache -
https://openaccess.thecvf.com/content_cvpr_2015/papers/Xiao_Learning_From_Massive_2015_CVPR_paper.pdf#baidu: “Clothing-1M: Learning from Massive Noisy Labeled Data for Image Classification”, Tong Xiao, Tian Xia, Yi Yang, Chang Huang, Xiaogang Wang -
forking-path: “Technology Forecasting: The Garden of Forking Paths”, Gwern -
http://www.lrec-conf.org/proceedings/lrec2014/pdf/1097_Paper.pdf: “N-gram Counts and Language Models from the Common Crawl”, Christian Buck, Kenneth Heafield, Bas van Ooyen -
https://aclanthology.org/P13-2121.pdf: “Scalable Modified Kneser-Ney Language Model Estimation”, Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, Philipp Koehn -
2010-mikolov.pdf: “Recurrent Neural Network Based Language Model”, Tomas Mikolov, Martin Karafiat, Lukas Burget, Jan Cernocky, Sanjeev Khudanpur -
difference: “How Complex Are Individual Differences?”, Gwern -
2010-hameed.pdf: “Understanding Sources of Inefficiency in General-purpose Chips”, -
https://dw2blog.com/2009/11/02/halloween-nightmare-scenario-early-2020s/: “Halloween Nightmare Scenario, Early 2020’s”, David Wood -
https://web.archive.org/web/20230718144747/https://frc.ri.cmu.edu/~hpm/project.archive/robot.papers/2004/Predictions.html: “Robot Predictions Evolution”, Hans Moravec -
2003-perlich.pdf: “Tree Induction vs. Logistic Regression: A Learning-Curve Analysis”, Claudia Perlich, Foster Provost, Jeffrey S. Simonoff -
https://web.archive.org/web/20230710000944/https://frc.ri.cmu.edu/~hpm/project.archive/general.articles/1975/Raw.Power.html: “The Role Of RAW POWER In INTELLIGENCE”, Hans Moravec -
https://paulfchristiano.com/: “Homepage of Paul F. Christiano”, Paul F. Christiano