“‘AI Scaling’ Tag”,2019-08-31 (CNN, Transformer, AlphaGo, MARL, AI safety)⁠:

Bibliography for tag ai/scaling, most recent first: 21 related tags, 642 annotations, & 93 links (parent).

[page summary]

• See Also
• Gwern
  • “What Do You Do After ‘Winning’ an AI Arms Race?”, Gwern2024
  • “What Do We Mean by ‘Diminishing Returns’ in Scaling?”, Gwern2024
  • “Research Ideas”, Gwern2017
  • “Absolute Unit NNs: Regression-Based MLPs for Everything”, Gwern2023
  • “GPT-3 Creative Fiction”, Gwern2020
  • “GANs Didn’t Fail, They Were Abandoned”, Gwern2022
  • “The Scaling Hypothesis”, Gwern2020
  • “ML Scaling Subreddit”, Gwern2020
  • “WBE and DRL: a Middle Way of Imitation Learning from the Human Brain”, Gwern2018
  • “Computer Optimization: Your Computer Is Faster Than You Think”, Gwern2021
  • “Fully-Connected Neural Nets”, Gwern2021
  • “Machine Learning Scaling”, Gwern2021
  • “Technology Forecasting: The Garden of Forking Paths”, Gwern2014
• Links
  • “Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?”, Jeong et al 2024
  • “ABBYY’s Bitter Lesson: How Linguists Lost the Last Battle for NLP”, Skorinkin2024
  • “CT Foundation: Taking Medical Imaging Embeddings 3D”, Kiraly & Traverse2024
  • “Inference Scaling for Long-Context Retrieval Augmented Generation”, Yue et al 2024
  • “Strategic Insights from Simulation Gaming of AI Race Dynamics”, Gruetzemacher et al 2024
  • “How Feature Learning Can Improve Neural Scaling Laws”, Bordelon et al 2024
  • “Dwarkesh Podcast Progress Update”, Patel2024
  • “Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?”, Ren et al 2024
  • “Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process”, Ye et al 2024
  • “Future Events As Backdoor Triggers: Investigating Temporal Vulnerabilities in LLMs”, Price et al 2024
  • “Resolving Discrepancies in Compute-Optimal Scaling of Language Models”, Porian et al 2024
  • “Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?”, Lee et al 2024
  • “Probing the Decision Boundaries of In-Context Learning in Large Language Models”, Zhao et al 2024
  • “How Do Large Language Models Acquire Factual Knowledge During Pretraining?”, Chang et al 2024
  • “Explore the Limits of Omni-Modal Pretraining at Scale”, Zhang et al 2024
  • “Self-Consuming Generative Models With Curated Data Provably Optimize Human Preferences”, Ferbach et al 2024
  • “Beyond Model Collapse: Scaling Up With Synthesized Data Requires Reinforcement”, Feng et al 2024
  • “Attention As a Hypernetwork”, Schug et al 2024
  • “Training Compute-Optimal Protein Language Models”, Cheng et al 2024
  • “AI Will Become Mathematicians’ ‘Co-Pilot’: Fields Medalist Terence Tao Explains How Proof Checkers and AI Programs Are Dramatically Changing Mathematics”, Drösser & Tao2024
  • “The Scaling Law in Stellar Light Curves”, Pan et al 2024
  • “AstroPT: Scaling Large Observation Models for Astronomy”, Smith et al 2024
  • “XLSTM: Extended Long Short-Term Memory”, Beck et al 2024
  • “Position: Understanding LLMs Requires More Than Statistical Generalization”, Reizinger et al 2024
  • “GSM1k: A Careful Examination of Large Language Model Performance on Grade School Arithmetic”, Zhang et al 2024
  • “CatLIP: CLIP-Level Visual Recognition Accuracy With 2.7× Faster Pre-Training on Web-Scale Image-Text Data”, Mehta et al 2024
  • “Test-Time Augmentation to Solve ARC”, Cole2024
  • “Chinchilla Scaling: A Replication Attempt”, Besiroglu et al 2024
  • “Scaling (Down) CLIP: A Comprehensive Analysis of Data, Architecture, and Training Strategies”, Li et al 2024
  • “Why Do Small Language Models Underperform? Studying Language Model Saturation via the Softmax Bottleneck”, Godey et al 2024
  • “Language Imbalance Can Boost Cross-Lingual Generalization”, Schäfer et al 2024
  • “CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs’ (Lack Of) Multicultural Knowledge”, Chiu et al 2024
  • “Conformer-1: Robust ASR via Large-Scale Semi-Supervised Bootstrapping”, Zhang et al 2024
  • “MiniCPM: Unveiling the Potential of Small Language Models With Scalable Training Strategies”, Hu et al 2024
  • “Visual Autoregressive Modeling (VAR): Scalable Image Generation via Next-Scale Prediction”, Tian et al 2024
  • “Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data”, Gerstgrasser et al 2024
  • “Long-Form Factuality in Large Language Models”, Wei et al 2024
  • “Mechanistic Design and Scaling of Hybrid Architectures”, Poli et al 2024
  • “8 Google Employees Invented Modern AI. Here’s the Inside Story: They Met by Chance, Got Hooked on an Idea, and Wrote the Transformers Paper—The Most Consequential Tech Breakthrough in Recent History”, Levy2024
  • “Inflection-2.5: Meet the World’s Best Personal AI”, Inflection2024
  • “Actions Speak Louder Than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations (HSTU)”, Zhai et al 2024
  • “When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method”, Zhang et al 2024
  • “Investigating Continual Pretraining in Large Language Models: Insights and Implications”, Yıldız et al 2024
  • “The Era of 1-Bit LLMs: All Large Language Models Are in 1.58 Bits”, Ma et al 2024
  • “StructLM: Towards Building Generalist Models for Structured Knowledge Grounding”, Zhuang et al 2024
  • “How to Train Data-Efficient LLMs”, Sachdeva et al 2024
  • “Weaver: Foundation Models for Creative Writing”, Wang et al 2024
  • “Arrows of Time for Large Language Models”, Papadopoulos et al 2024
  • “Can AI Assistants Know What They Don’t Know?”, Cheng et al 2024
  • “I Am a Strange Dataset: Metalinguistic Tests for Language Models”, Thrush et al 2024
  • “TF-T2V: A Recipe for Scaling up Text-To-Video Generation With Text-Free Videos”, Wang et al 2023
  • “Generative Multimodal Models Are In-Context Learners”, Sun et al 2023
  • “Zoology: Measuring and Improving Recall in Efficient Language Models”, Arora et al 2023
  • “Seamless: Multilingual Expressive and Streaming Speech Translation”, Communication et al 2023
  • “Scaling Transformer Neural Networks for Skillful and Reliable Medium-Range Weather Forecasting”, Nguyen et al 2023
  • “Instruction-Tuning Aligns LLMs to the Human Brain”, Aw et al 2023
  • “Mamba: Linear-Time Sequence Modeling With Selective State Spaces”, Gu & Dao2023
  • “Sequential Modeling Enables Scalable Learning for Large Vision Models”, Bai et al 2023
  • “UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition”, Ding et al 2023
  • “In Search of the Long-Tail: Systematic Generation of Long-Tail Inferential Knowledge via Logical Rule Guided Search”, Li et al 2023
  • “First Tragedy, Then Parse: History Repeats Itself in the New Era of Large Language Models”, Saphra et al 2023
  • “I2VGen-XL: High-Quality Image-To-Video Synthesis via Cascaded Diffusion Models”, Zhang et al 2023
  • “A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models”, Eisape et al 2023
  • “Sam Altman Accepts the 2023 Hawking Fellowship Award § Is There Another Breakthrough That’s Needed to Reach AGI?”, Altman2023
  • “ConvNets Match Vision Transformers at Scale”, Smith et al 2023
  • “PaLI-3 Vision Language Models: Smaller, Faster, Stronger”, Chen et al 2023
  • “GeoLLM: Extracting Geospatial Knowledge from Large Language Models”, Manvi et al 2023
  • “Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition”, Chen et al 2023
  • “Sheared LLaMA: Accelerating Language Model Pre-Training via Structured Pruning”, Xia et al 2023
  • “FreshLLMs: Refreshing Large Language Models With Search Engine Augmentation”, Vu et al 2023
  • “Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors”, Amos et al 2023
  • “MTOB: A Benchmark for Learning to Translate a New Language from One Grammar Book”, Tanzer et al 2023
  • “Intriguing Properties of Generative Classifiers”, Jaini et al 2023
  • “Taken out of Context: On Measuring Situational Awareness in LLMs”, Berglund et al 2023
  • “SeamlessM4T: Massively Multilingual & Multimodal Machine Translation”, Communication et al 2023
  • “Simple Synthetic Data Reduces Sycophancy in Large Language Models”, Wei et al 2023
  • “LLaMA-2: Open Foundation and Fine-Tuned Chat Models”, Touvron et al 2023
  • “Measuring Faithfulness in Chain-Of-Thought Reasoning”, Lanham et al 2023
  • “Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration”, Wang et al 2023
  • “Introducing Superalignment”, Leike & Sutskever2023
  • “Gödel, Escher, Bach Author Douglas Hofstadter on the State of AI Today § What about AI Terrifies You?”, Hofstadter & Kim2023
  • “Pretraining Task Diversity and the Emergence of Non-Bayesian In-Context Learning for Regression”, Raventós et al 2023
  • “Beyond Scale: the Diversity Coefficient As a Data Quality Metric Demonstrates LLMs Are Pre-Trained on Formally Diverse Data”, Lee et al 2023
  • “Scaling MLPs: A Tale of Inductive Bias”, Bachmann et al 2023
  • “Understanding Social Reasoning in Language Models With Language Models”, Gandhi et al 2023
  • “Image Captioners Are Scalable Vision Learners Too”, Tschannen et al 2023
  • “PaLI-X: On Scaling up a Multilingual Vision and Language Model”, Chen et al 2023
  • “The False Promise of Imitating Proprietary LLMs”, Gudibande et al 2023
  • “Scaling Data-Constrained Language Models”, Muennighoff et al 2023
  • “Scaling Laws for Language Encoding Models in FMRI”, Antonello et al 2023
  • “LIMA: Less Is More for Alignment”, Zhou et al 2023
  • “Google’s Newest AI Model Uses Nearly 5× More Text Data for Training Than Its Predecessor”, Elias2023
  • “TorToise: Better Speech Synthesis through Scaling”, Betker2023
  • “TinyStories: How Small Can Language Models Be and Still Speak Coherent English?”, Eldan & Li2023
  • “ImageBind: One Embedding Space To Bind Them All”, Girdhar et al 2023
  • “Finding Neurons in a Haystack: Case Studies With Sparse Probing”, Gurnee et al 2023
  • “Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4”, Chang et al 2023
  • “Google’s DeepMind-Brain Merger: Tech Giant Regroups for AI Battle”, Murgia2023
  • “CLaMP: Contrastive Language-Music Pre-Training for Cross-Modal Symbolic Music Information Retrieval”, Wu et al 2023
  • “Emergent and Predictable Memorization in Large Language Models”, Biderman et al 2023
  • “Power Law Trends in Speedrunning and Machine Learning”, Erdil & Sevilla2023
  • “Even The Politicians Thought the Open Letter Made No Sense In The Senate Hearing on AI Today’s Hearing on Ai Covered Ai Regulation and Challenges, and the Infamous Open Letter, Which Nearly Everyone in the Room Thought Was Unwise”, Gorrell2023
  • “DINOv2: Learning Robust Visual Features without Supervision”, Oquab et al 2023
  • “Segment Anything”, Kirillov et al 2023
  • “Humans in Humans Out: On GPT Converging Toward Common Sense in Both Success and Failure”, Koralus & Wang-Maścianica2023
  • “Sigmoid Loss for Language Image Pre-Training”, Zhai et al 2023
  • “How Well Do Large Language Models Perform in Arithmetic Tasks?”, Yuan et al 2023
  • “GPT-4 Technical Report”, OpenAI 2023
  • “Securing Liberal Democratic Control of AGI through UK Leadership”, Phillips2023
  • “GigaGAN: Scaling up GANs for Text-To-Image Synthesis”, Kang et al 2023
  • “Language Is Not All You Need: Aligning Perception With Language Models (Kosmos-1)”, Huang et al 2023
  • “Why Didn’t DeepMind Build GPT-3?”, Godwin2023
  • “Scaling Vision Transformers to 22 Billion Parameters”, Dehghani et al 2023
  • “John Carmack’s ‘Different Path’ to Artificial General Intelligence”, Carmack2023
  • “Large Language Models As Fiduciaries: A Case Study Toward Robustly Communicating With Artificial Intelligence Through Legal Standards”, Nay2023
  • “ClimaX: A Foundation Model for Weather and Climate”, Nguyen et al 2023
  • “StyleGAN-T: Unlocking the Power of GANs for Fast Large-Scale Text-To-Image Synthesis”, Sauer et al 2023
  • “MUG: Vision Learners Meet Web Image-Text Pairs”, Zhao et al 2023
  • “GPT-3 As Knowledge Worker: A Zero-Shot Evaluation of AI CPA Capabilities”, Bommarito et al 2023
  • “Scaling Laws for Generative Mixed-Modal Language Models”, Aghajanyan et al 2023
  • “VALL-E: Neural Codec Language Models Are Zero-Shot Text to Speech Synthesizers”, Wang et al 2023
  • “GPT-3 Takes the Bar Exam”, II & Katz2022
  • “Cramming: Training a Language Model on a Single GPU in One Day”, Geiping & Goldstein2022
  • “Evolutionary-Scale Prediction of Atomic Level Protein Structure With a Language Model”, Lin et al 2022
  • “Discovering Language Model Behaviors With Model-Written Evaluations”, Perez et al 2022
  • “One Embedder, Any Task: Instruction-Finetuned Text Embeddings (INSTRUCTOR)”, Su et al 2022
  • “Reproducible Scaling Laws for Contrastive Language-Image Learning”, Cherti et al 2022
  • “ERNIE-Code: Beyond English-Centric Cross-Lingual Pretraining for Programming Languages”, Chai et al 2022
  • “VideoCoCa: Video-Text Modeling With Zero-Shot Transfer from Contrastive Captioners”, Yan et al 2022
  • “VindLU: A Recipe for Effective Video-And-Language Pretraining”, Cheng et al 2022
  • “Whisper: Robust Speech Recognition via Large-Scale Weak Supervision”, Radford et al 2022
  • “Scaling Language-Image Pre-Training via Masking”, Li et al 2022
  • “MultiRay: Optimizing Efficiency for Large-Scale AI Models”, Gupta et al 2022
  • “Galactica: A Large Language Model for Science”, Taylor et al 2022
  • “Large Language Models Struggle to Learn Long-Tail Knowledge”, Kandpal et al 2022
  • “EVA: Exploring the Limits of Masked Visual Representation Learning at Scale”, Fang et al 2022
  • “MMDialog: A Large-Scale Multi-Turn Dialogue Dataset Towards Multi-Modal Open-Domain Conversation”, Feng et al 2022
  • “Adversarial Policies Beat Superhuman Go AIs”, Wang et al 2022
  • “Increments Podcast: #45—4 Central Fallacies of AI Research (with Melanie Mitchell)”, Mitchell & Chugg2022
  • “A Solvable Model of Neural Scaling Laws”, Maloney et al 2022
  • “Will We Run out of Data? An Analysis of the Limits of Scaling Datasets in Machine Learning”, Villalobos et al 2022
  • “Evaluating Parameter Efficient Learning for Generation”, Xu et al 2022
  • “FLAN: Scaling Instruction-Finetuned Language Models”, Chung et al 2022
  • “BioGPT: Generative Pre-Trained Transformer for Biomedical Text Generation and Mining”, Luo et al 2022
  • “Vision-Language Pre-Training: Basics, Recent Advances, and Future Trends”, Gan et al 2022
  • “Foundation Transformers”, Wang et al 2022
  • “Self-Ask: Measuring and Narrowing the Compositionality Gap in Language Models (Bamboogle)”, Press et al 2022
  • “GLM-130B: An Open Bilingual Pre-Trained Model”, Zeng et al 2022
  • “Ask Me Anything (AMA): A Simple Strategy for Prompting Language Models”, Arora et al 2022
  • “Do Current Multi-Task Optimization Methods in Deep Learning Even Help?”, Xin et al 2022
  • “Monolith: Real Time Recommendation System With Collisionless Embedding Table”, Liu et al 2022
  • “Machine Reading, Fast and Slow: When Do Models “Understand” Language?”, Choudhury et al 2022
  • “PaLI: A Jointly-Scaled Multilingual Language-Image Model”, Chen et al 2022
  • “Using Large Language Models to Simulate Multiple Humans”, Aher et al 2022
  • “Understanding Scaling Laws for Recommendation Models”, Ardalani et al 2022
  • “LLM.int8(): 8-Bit Matrix Multiplication for Transformers at Scale”, Dettmers et al 2022
  • “Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIP”, Nguyen et al 2022
  • “Efficient Training of Language Models to Fill in the Middle”, Bavarian et al 2022
  • “Why Do Tree-Based Models Still Outperform Deep Learning on Tabular Data?”, Grinsztajn et al 2022
  • “PIXEL: Language Modeling With Pixels”, Rust et al 2022
  • “High-Performing Neural Network Models of Visual Cortex Benefit from High Latent Dimensionality”, Elmoznino & Bonner2022
  • “Exploring Length Generalization in Large Language Models”, Anil et al 2022
  • “Language Models (Mostly) Know What They Know”, Kadavath et al 2022
  • “On-Device Training Under 256KB Memory”, Lin et al 2022
  • “Beyond Neural Scaling Laws: Beating Power Law Scaling via Data Pruning”, Sorscher et al 2022
  • “ProGen2: Exploring the Boundaries of Protein Language Models”, Nijkamp et al 2022
  • “RST: ReStructured Pre-Training”, Yuan & Liu2022
  • “Limitations of the NTK for Understanding Generalization in Deep Learning”, Vyas et al 2022
  • “Modeling Transformative AI Risks (MTAIR) Project—Summary Report”, Clarke et al 2022
  • “BigVGAN: A Universal Neural Vocoder With Large-Scale Training”, Lee et al 2022
  • “An Improved One Millisecond Mobile Backbone”, Vasu et al 2022
  • “A Neural Corpus Indexer for Document Retrieval”, Wang et al 2022
  • “Toward a Realistic Model of Speech Processing in the Brain With Self-Supervised Learning”, Millet et al 2022
  • “Teaching Models to Express Their Uncertainty in Words”, Lin et al 2022
  • “Why Robust Generalization in Deep Learning Is Difficult: Perspective of Expressive Power”, Li et al 2022
  • “M3AE: Multimodal Masked Autoencoders Learn Transferable Representations”, Geng et al 2022
  • “InstructDial: Improving Zero and Few-Shot Generalization in Dialogue through Instruction Tuning”, Gupta et al 2022
  • “Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models”, Tirumala et al 2022
  • “Least-To-Most Prompting Enables Complex Reasoning in Large Language Models”, Zhou et al 2022
  • “Continual Pre-Training Mitigates Forgetting in Language and Vision”, Cossu et al 2022
  • “Dialog Inpainting: Turning Documents into Dialogues”, Dai et al 2022
  • “Unifying Language Learning Paradigms”, Tay et al 2022
  • “Building Machine Translation Systems for the Next Thousand Languages”, Bapna et al 2022
  • “When Does Dough Become a Bagel? Analyzing the Remaining Mistakes on ImageNet”, Vasudevan et al 2022
  • “CoCa: Contrastive Captioners Are Image-Text Foundation Models”, Yu et al 2022
  • “Data Determines Distributional Robustness in Contrastive Language Image Pre-Training (CLIP)”, Fang et al 2022
  • “Continual Learning With Foundation Models: An Empirical Study of Latent Replay”, Ostapenko et al 2022
  • “Flamingo: a Visual Language Model for Few-Shot Learning”, Alayrac et al 2022
  • “WebFace260M: A Benchmark for Million-Scale Deep Face Recognition”, Zhu et al 2022
  • “What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?”, Wang et al 2022
  • “DeepMind: The Podcast—Excerpts on AGI”, Kiely2022
  • “Can Language Models Learn from Explanations in Context?”, Lampinen et al 2022
  • “Chinchilla: Training Compute-Optimal Large Language Models”, Hoffmann et al 2022
  • “A Roadmap for Big Model”, Yuan et al 2022
  • “A Conversational Paradigm for Program Synthesis”, Nijkamp et al 2022
  • “Self-Consistency Improves Chain-Of-Thought Reasoning in Language Models”, Wang et al 2022
  • “Effect of Scale on Catastrophic Forgetting in Neural Networks”, Ramasesh et al 2022
  • “Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer”, Yang et al 2022
  • “FastFold: Reducing AlphaFold Training Time from 11 Days to 67 Hours”, Cheng et al 2022
  • “Variational Autoencoders Without the Variation”, Daly et al 2022
  • “Performance Reserves in Brain-Imaging-Based Phenotype Prediction”, Schulz et al 2022
  • “Self-Distilled StyleGAN: Towards Generation from Internet Photos”, Mokady et al 2022
  • “UnifiedQA-V2: Stronger Generalization via Broader Cross-Format Training”, Khashabi et al 2022
  • “Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervision”, Goyal et al 2022
  • “Brains and Algorithms Partially Converge in Natural Language Processing”, Caucheteux & King2022
  • “Quantifying Memorization Across Neural Language Models”, Carlini et al 2022
  • “Wukong: 100 Million Large-Scale Chinese Cross-Modal Pre-Training Dataset and A Foundation Framework”, Gu et al 2022
  • “OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-To-Sequence Learning Framework”, Wang et al 2022
  • “Data Scaling Laws in NMT: The Effect of Noise and Architecture”, Bansal et al 2022
  • “Webly Supervised Concept Expansion for General Purpose Vision Models”, Kamath et al 2022
  • “StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets”, Sauer et al 2022
  • “Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model”, Smith et al 2022
  • “Reasoning Like Program Executors”, Pi et al 2022
  • “Text and Code Embeddings by Contrastive Pre-Training”, Neelakantan et al 2022
  • “LaMDA: Language Models for Dialog Applications”, Thoppilan et al 2022
  • “SWAG: Revisiting Weakly Supervised Pre-Training of Visual Perception Models”, Singh et al 2022
  • “CM3: A Causal Masked Multimodal Model of the Internet”, Aghajanyan et al 2022
  • “ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalization”, Xu et al 2022
  • “A High-Dimensional Sphere Spilling out of a High-Dimensional Cube despite Exponentially Many Constraints”, Fort2022
  • “ConvNeXt: A ConvNet for the 2020s”, Liu et al 2022
  • “The Defeat of the Winograd Schema Challenge”, Kocijan et al 2022
  • “Robust Self-Supervised Audio-Visual Speech Recognition”, Shi et al 2022
  • “AV-HuBERT: Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction”, Shi et al 2022
  • “Self-Supervised Learning from 100 Million Medical Images”, Ghesu et al 2022
  • “The Evolution of Quantitative Sensitivity”, Bryer et al 2021
  • “ERNIE 3.0 Titan: Exploring Larger-Scale Knowledge Enhanced Pre-Training for Language Understanding and Generation”, Wang et al 2021
  • “XGLM: Few-Shot Learning With Multilingual Language Models”, Lin et al 2021
  • “An Empirical Investigation of the Role of Pre-Training in Lifelong Learning”, Mehta et al 2021
  • “Few-Shot Instruction Prompts for Pretrained Language Models to Detect Social Biases”, Prabhumoye et al 2021
  • “Knowledge-Rich Self-Supervised Entity Linking”, Zhang et al 2021
  • “You Only Need One Model for Open-Domain Question Answering”, Lee et al 2021
  • “EBERT: Epigenomic Language Models Powered by Cerebras”, Trotter et al 2021
  • “MAGMA—Multimodal Augmentation of Generative Models through Adapter-Based Finetuning”, Eichenberg et al 2021
  • “Improving Language Models by Retrieving from Trillions of Tokens”, Borgeaud et al 2021
  • “MLP Architectures for Vision-And-Language Modeling: An Empirical Study”, Nie et al 2021
  • “LEMON: Scaling Up Vision-Language Pre-Training for Image Captioning”, Hu et al 2021
  • “Sparse Is Enough in Scaling Transformers”, Jaszczur et al 2021
  • “Can Pre-Trained Language Models Be Used to Resolve Textual and Semantic Merge Conflicts?”, Zhang et al 2021
  • “ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning”, Aribandi et al 2021
  • “L-Verse: Bidirectional Generation Between Image and Text”, Kim et al 2021
  • “RedCaps: Web-Curated Image-Text Data Created by the People, for the People”, Desai et al 2021
  • “Florence: A New Foundation Model for Computer Vision”, Yuan et al 2021
  • “BASIC: Combined Scaling for Open-Vocabulary Image Classification”, Pham et al 2021
  • “Swin Transformer V2: Scaling Up Capacity and Resolution”, Liu et al 2021
  • “XLS-R: Self-Supervised Cross-Lingual Speech Representation Learning at Scale”, Babu et al 2021
  • “Solving Linear Algebra by Program Synthesis”, Drori & Verma2021
  • “Covariate Shift in High-Dimensional Random Feature Regression”, Tripuraneni et al 2021
  • “Solving Probability and Statistics Problems by Program Synthesis”, Tang et al 2021
  • “Few-Shot Self-Rationalization With Natural Language Prompts”, Marasović et al 2021
  • “INTERN: A New Learning Paradigm Towards General Vision”, Shao et al 2021
  • “Scaling Law for Recommendation Models: Towards General-Purpose User Representations”, Shin et al 2021
  • “MAE: Masked Autoencoders Are Scalable Vision Learners”, He et al 2021
  • “Persia: An Open, Hybrid System Scaling Deep Learning-Based Recommenders up to 100 Trillion Parameters”, Lian et al 2021
  • “Scaling ASR Improves Zero and Few Shot Learning”, Xiao et al 2021
  • “Turing-Universal Learners With Optimal Scaling Laws”, Nakkiran2021
  • “LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs”, Schuhmann et al 2021
  • “Training Verifiers to Solve Math Word Problems”, Cobbe et al 2021
  • “Wide Neural Networks Forget Less Catastrophically”, Mirzadeh et al 2021
  • “When in Doubt, Summon the Titans: Efficient Inference With Large Models”, Rawat et al 2021
  • “The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail”, Bowman2021
  • “Symbolic Knowledge Distillation: from General Language Models to Commonsense Models”, West et al 2021
  • “LFPT5: A Unified Framework for Lifelong Few-Shot Language Learning Based on Prompt Tuning of T5”, Qin & Joty2021
  • “Scaling Laws for the Few-Shot Adaptation of Pre-Trained Image Classifiers”, Prato et al 2021
  • “Unsupervised Neural Machine Translation With Generative Language Models Only”, Han et al 2021
  • “Yuan 1.0: Large-Scale Pre-Trained Language Model in Zero-Shot and Few-Shot Learning”, Wu et al 2021
  • “Universal Paralinguistic Speech Representations Using Self-Supervised Conformers”, Shor et al 2021
  • “M6–10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining”, Lin et al 2021
  • “A Few More Examples May Be Worth Billions of Parameters”, Kirstain et al 2021
  • “Exploring the Limits of Large Scale Pre-Training”, Abnar et al 2021
  • “Show Your Work: Scratchpads for Intermediate Computation With Language Models”, Nye et al 2021
  • “Mining for Strong Gravitational Lenses With Self-Supervised Learning”, Stein et al 2021
  • “Stochastic Training Is Not Necessary for Generalization”, Geiping et al 2021
  • “Evaluating Machine Accuracy on ImageNet”, Shankar et al 2021
  • “BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition”, Zhang et al 2021
  • “Scale Efficiently: Insights from Pre-Training and Fine-Tuning Transformers”, Tay et al 2021
  • “Scaling Laws for Neural Machine Translation”, Ghorbani et al 2021
  • “What Changes Can Large-Scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-Scale Korean Generative Pretrained Transformers”, Kim et al 2021
  • “A Recipe For Arbitrary Text Style Transfer With Large Language Models”, Reif et al 2021
  • “TruthfulQA: Measuring How Models Mimic Human Falsehoods”, Lin et al 2021
  • “A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning”, Dar et al 2021
  • “General-Purpose Question-Answering With Macaw”, Tafjord & Clark2021
  • “An Empirical Exploration in Quality Filtering of Text Data”, Gao2021
  • “A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP”, Zhao et al 2021
  • “Want To Reduce Labeling Cost? GPT-3 Can Help”, Wang et al 2021
  • “Data and Parameter Scaling Laws for Neural Machine Translation”, Gordon et al 2021
  • “Do Vision Transformers See Like Convolutional Neural Networks?”, Raghu et al 2021
  • “Modeling Protein Using Large-Scale Pretrain Language Model”, Xiao et al 2021
  • “Scaling Laws for Deep Learning”, Rosenfeld2021
  • “Billion-Scale Pretraining With Vision Transformers for Multi-Task Visual Representations”, Beal et al 2021
  • “Facebook AI WMT21 News Translation Task Submission”, Tran et al 2021
  • “EVA: An Open-Domain Chinese Dialogue System With Large-Scale Generative Pre-Training”, Zhou et al 2021
  • “The History of Speech Recognition to the Year 2030”, Hannun2021
  • “The History of Speech Recognition to the Year 2030”, Hannun2021
  • “A Field Guide to Federated Optimization”, Wang et al 2021
  • “HTLM: Hyper-Text Pre-Training and Prompting of Language Models”, Aghajanyan et al 2021
  • “Brain-Like Functional Specialization Emerges Spontaneously in Deep Neural Networks”, Dobs et al 2021
  • “ERNIE 3.0: Large-Scale Knowledge Enhanced Pre-Training for Language Understanding and Generation”, Sun et al 2021
  • “Scarecrow: A Framework for Scrutinizing Machine Text”, Dou et al 2021
  • “The Dimpled Manifold Model of Adversarial Examples in Machine Learning”, Shamir et al 2021
  • “Revisiting the Calibration of Modern Neural Networks”, Minderer et al 2021
  • “Partial Success in Closing the Gap between Human and Machine Vision”, Geirhos et al 2021
  • “HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units”, Hsu et al 2021
  • “Scaling Laws for Acoustic Models”, Droppo & Elibol2021
  • “CoAtNet: Marrying Convolution and Attention for All Data Sizes”, Dai et al 2021
  • “Scaling Vision Transformers”, Zhai et al 2021
  • “Exploring the Limits of Out-Of-Distribution Detection”, Fort et al 2021
  • “Effect of Pre-Training Scale on Intra/Inter-Domain Full and Few-Shot Transfer Learning for Natural and Medical X-Ray Chest Images”, Cherti & Jitsev2021
  • “A Universal Law of Robustness via Isoperimetry”, Bubeck & Sellke2021
  • “Naver Unveils First ‘Hyperscale’ AI Platform”, Jae-eun2021
  • “Unsupervised Speech Recognition”, Baevski et al 2021
  • “One4all User Representation for Recommender Systems in E-Commerce”, Shin et al 2021
  • “RecPipe: Co-Designing Models and Hardware to Jointly Optimize Recommendation Quality and Performance”, Gupta et al 2021
  • “Google Details New AI Accelerator Chips”, Wiggers2021
  • “MLP-Mixer: An All-MLP Architecture for Vision”, Tolstikhin et al 2021
  • “XLM-R XL: Larger-Scale Transformers for Multilingual Masked Language Modeling”, Goyal et al 2021
  • “Scaling End-To-End Models for Large-Scale Multilingual ASR”, Li et al 2021
  • “DINO: Emerging Properties in Self-Supervised Vision Transformers”, Caron et al 2021
  • “What Are Bayesian Neural Network Posteriors Really Like?”, Izmailov et al 2021
  • “[Ali Released PLUG: 27 Billion Parameters, the Largest Pre-Trained Language Model in the Chinese Community]”, Yuying2021
  • “The Power of Scale for Parameter-Efficient Prompt Tuning”, Lester et al 2021
  • “Revealing Persona Biases in Dialogue Systems”, Sheng et al 2021
  • “CrossFit: A Few-Shot Learning Challenge for Cross-Task Generalization in NLP”, Ye et al 2021
  • “Probing Across Time: What Does RoBERTa Know and When?”, Liu et al 2021
  • “Memorization versus Generalization in Pre-Trained Language Models”, Tänzer et al 2021
  • “Large-Scale Self-Supervised and Semi-Supervised Learning for Speech Translation”, Wang et al 2021
  • “Scaling Laws for Language Transfer Learning”, Kim2021
  • “Adapting Language Models for Zero-Shot Learning by Meta-Tuning on Dataset and Prompt Collections”, Zhong et al 2021
  • “SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network”, Chan et al 2021
  • “Understanding Robustness of Transformers for Image Classification”, Bhojanapalli et al 2021
  • “UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark”, Lourie et al 2021
  • “Controllable Generation from Pre-Trained Language Models via Inverse Prompting”, Zou et al 2021
  • “The Shape of Learning Curves: a Review”, Viering & Loog2021
  • “Efficient Visual Pretraining With Contrastive Detection”, Hénaff et al 2021
  • “Revisiting ResNets: Improved Training and Scaling Strategies”, Bello et al 2021
  • “Learning from Videos to Understand the World”, Zweig et al 2021
  • “WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training”, Huo et al 2021
  • “Fast and Accurate Model Scaling”, Dollár et al 2021
  • “Pretrained Transformers As Universal Computation Engines”, Lu et al 2021
  • “Greedy Hierarchical Variational Autoencoders (GHVAEs) for Large-Scale Video Prediction”, Wu et al 2021
  • “Measuring Mathematical Problem Solving With the MATH Dataset”, Hendrycks et al 2021
  • “A Law of Robustness for Two-Layers Neural Networks”, Bubeck et al 2021
  • “SEER: Self-Supervised Pretraining of Visual Features in the Wild”, Goyal et al 2021
  • “M6: A Chinese Multimodal Pretrainer”, Lin et al 2021
  • “Zero-Shot Text-To-Image Generation”, Ramesh et al 2021
  • “Improved Denoising Diffusion Probabilistic Models”, Nichol & Dhariwal2021
  • “Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts”, Changpinyo et al 2021
  • “A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes”, Nado et al 2021
  • “Explaining Neural Scaling Laws”, Bahri et al 2021
  • “ALIGN: Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision”, Jia et al 2021
  • “NFNet: High-Performance Large-Scale Image Recognition Without Normalization”, Brock et al 2021
  • “Learning Curve Theory”, Hutter2021
  • “1-Bit Adam: Communication Efficient Large-Scale Training With Adam’s Convergence Speed”, Tang et al 2021
  • “Mind the Gap: Assessing Temporal Generalization in Neural Language Models § Scaling”, Lazaridou et al 2021
  • “Scaling Laws for Transfer”, Hernandez et al 2021
  • “Automatic Curation of Large-Scale Datasets for Audio-Visual Representation Learning”, Lee et al 2021
  • “Muppet: Massive Multi-Task Representations With Pre-Finetuning”, Aghajanyan et al 2021
  • “Language Processing in Brains and Deep Neural Networks: Computational Convergence and Its Limits”, Caucheteux & King2021
  • “Meta Pseudo Labels”, Pham et al 2021
  • “CLIP: Learning Transferable Visual Models From Natural Language Supervision”, Radford et al 2021
  • “VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation”, Wang et al 2021
  • “CDLM: Cross-Document Language Modeling”, Caciularu et al 2021
  • “VinVL: Revisiting Visual Representations in Vision-Language Models”, Zhang et al 2021
  • “Parameter Count vs Training Dataset Size (1952^–₆₉2021)”, Adlam2021
  • “Process for Adapting Language Models to Society (PALMS) With Values-Targeted Datasets”, Solaiman & Dennison2021
  • “Extrapolating GPT-N Performance”, Finnveden2020
  • “Biological Structure and Function Emerge from Scaling Unsupervised Learning to 250 Million Protein Sequences”, Rives et al 2020
  • “CPM: A Large-Scale Generative Chinese Pre-Trained Language Model”, Zhang et al 2020
  • “Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images”, Child2020
  • “When Do You Need Billions of Words of Pretraining Data?”, Zhang et al 2020
  • “Scaling Laws for Autoregressive Generative Modeling”, Henighan et al 2020
  • “Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus”, Caswell et al 2020
  • “MT5: A Massively Multilingual Pre-Trained Text-To-Text Transformer”, Xue et al 2020
  • “Beyond English-Centric Multilingual Machine Translation”, Fan et al 2020
  • “Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition”, Zhang et al 2020
  • “Towards End-To-End In-Image Neural Machine Translation”, Mansimov et al 2020
  • “The First AI Model That Translates 100 Languages without Relying on English Data”, Fan2020
  • “WinoGrande: An Adversarial Winograd Schema Challenge at Scale”, Sakaguchi et al 2020
  • “The Deep Bootstrap Framework: Good Online Learners Are Good Offline Generalizers”, Nakkiran et al 2020
  • “Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)”, Warstadt et al 2020
  • “The Neural Architecture of Language: Integrative Reverse-Engineering Converges on a Model for Predictive Processing”, Schrimpf et al 2020
  • “Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples”, Gowal et al 2020
  • “Fast Stencil-Code Computation on a Wafer-Scale Processor”, Rocki et al 2020
  • “Vision Transformer: An Image Is Worth 16×16 Words: Transformers for Image Recognition at Scale”, Dosovitskiy et al 2020
  • “Small Data, Big Decisions: Model Selection in the Small-Data Regime”, Bornschein et al 2020
  • “New Report on How Much Computational Power It Takes to Match the Human Brain”, Carlsmith2020
  • “Generative Language Modeling for Automated Theorem Proving”, Polu & Sutskever2020
  • “GrokNet: Unified Computer Vision Model Trunk and Embeddings For Commerce”, Bell et al 2020
  • “Accuracy and Performance Comparison of Video Action Recognition Approaches”, Hutchinson et al 2020
  • “Generative Models Are Unsupervised Predictors of Page Quality: A Colossal-Scale Study”, Bahri et al 2020
  • “Matt Botvinick on the Spontaneous Emergence of Learning Algorithms”, Scholl2020
  • “Self-Supervised Learning through the Eyes of a Child”, Orhan et al 2020
  • “On Robustness and Transferability of Convolutional Neural Networks”, Djolonga et al 2020
  • “Hopfield Networks Is All You Need”, Ramsauer et al 2020
  • “ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Deep Learning and High Performance Computing”, Elnaggar et al 2020
  • “NVAE: A Deep Hierarchical Variational Autoencoder”, Vahdat & Kautz2020
  • “Measuring Robustness to Natural Distribution Shifts in Image Classification”, Taori et al 2020
  • “Is SGD a Bayesian Sampler? Well, Almost”, Mingard et al 2020
  • “Unsupervised Cross-Lingual Representation Learning for Speech Recognition”, Conneau et al 2020
  • “Logarithmic Pruning Is All You Need”, Orseau et al 2020
  • “Wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations”, Baevski et al 2020
  • “Denoising Diffusion Probabilistic Models”, Ho et al 2020
  • “On the Predictability of Pruning Across Scales”, Rosenfeld et al 2020
  • “IGPT: Generative Pretraining from Pixels”, Chen et al 2020
  • “SwAV: Unsupervised Learning of Visual Features by Contrasting Cluster Assignments”, Caron et al 2020
  • “SimCLRv2: Big Self-Supervised Models Are Strong Semi-Supervised Learners”, Chen et al 2020
  • “Image GPT (iGPT): We Find That, Just As a Large Transformer Model Trained on Language Can Generate Coherent Text, the Same Exact Model Trained on Pixel Sequences Can Generate Coherent Image Completions and Samples”, Chen et al 2020
  • “Are We Done With ImageNet?”, Beyer et al 2020
  • “OpenAI API”, Brockman et al 2020
  • “Object Segmentation Without Labels With Large-Scale Generative Models”, Voynov et al 2020
  • “How Big Should My Language Model Be?”, Scao2020
  • “GPT-3 Paper § Figure F.1: Four Uncurated Completions from a Context Suggesting the Model Compose a Poem in the Style of Wallace Stevens With the Title ‘Shadows on the Way’”, GPT-3 2020 (page 48)
  • “Danny Hernandez on Forecasting and the Drivers of AI Progress”, Koehler et al 2020
  • “Powered by AI: Advancing Product Understanding and Building New Shopping Experiences”, Berg et al 2020
  • “ZeRO-2 & DeepSpeed: Shattering Barriers of Deep Learning Speed & Scale”, Team2020
  • “Measuring the Algorithmic Efficiency of Neural Networks”, Hernandez & Brown2020
  • “Pushing the Limit of Molecular Dynamics With ab Initio Accuracy to 100 Million Atoms With Machine Learning”, Jia et al 2020
  • “Jukebox: We’re Introducing Jukebox, a Neural Net That Generates Music, including Rudimentary Singing, As Raw Audio in a Variety of Genres and Artist Styles. We’re Releasing the Model Weights and Code, along With a Tool to Explore the Generated Samples.”, Dhariwal et al 2020
  • “Blender: A State-Of-The-Art Open Source Chatbot”, Roller et al 2020
  • “A Review of Winograd Schema Challenge Datasets and Approaches”, Kocijan et al 2020
  • “Scaling Laws from the Data Manifold Dimension”, Sharma & Kaplan2020
  • “DynamicEmbedding: Extending TensorFlow for Colossal-Scale Applications”, Zeng et al 2020
  • “PALM: Pre-Training an Autoencoding & Autoregressive Language Model for Context-Conditioned Generation”, Bi et al 2020
  • “Deep Learning Training in Facebook Data Centers: Design of Scale-Up and Scale-Out Systems”, Naumov et al 2020
  • “TTTTTackling WinoGrande Schemas”, Lin et al 2020
  • “A Metric Learning Reality Check”, Musgrave et al 2020
  • “Zoom In: An Introduction to Circuits—By Studying the Connections between Neurons, We Can Find Meaningful Algorithms in the Weights of Neural Networks”, Olah et al 2020
  • “Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited”, Maddox et al 2020
  • “Rethinking Bias-Variance Trade-Off for Generalization of Neural Networks”, Yang et al 2020
  • “Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers”, Li et al 2020
  • “The Messy, Secretive Reality behind OpenAI’s Bid to save the World: The AI Moonshot Was Founded in the Spirit of Transparency. This Is the inside Story of How Competitive Pressure Eroded That Idealism”, Hao2020
  • “The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence”, Marcus2020
  • “A Simple Framework for Contrastive Learning of Visual Representations”, Chen et al 2020
  • “How Much Knowledge Can You Pack Into the Parameters of a Language Model?”, Roberts et al 2020
  • “Turing-NLG: A 17-Billion-Parameter Language Model by Microsoft”, Rosset2020
  • “Quasi-Equivalence of Width and Depth of Neural Networks”, Fan et al 2020
  • “Impact of ImageNet Model Selection on Domain Adaptation”, Zhang & Davison2020
  • “Direct Fit to Nature: An Evolutionary Perspective on Biological and Artificial Neural Networks”, Hasson et al 2020
  • “Towards a Conversational Agent That Can Chat About…Anything”, Adiwardana & Luong2020
  • “Towards a Human-Like Open-Domain Chatbot”, Adiwardana et al 2020
  • “Scaling Laws for Neural Language Models”, Kaplan et al 2020
  • “Scaling Laws for Neural Language Models: Figure 15: Far beyond the Model Sizes We Study Empirically, We Find a Contradiction between Our Equations § Pg17”, Kaplan2020 (page 17 org openai)
  • “The Importance of Deconstruction”, Weinberger2020
  • “Big Transfer (BiT): General Visual Representation Learning”, Kolesnikov et al 2019
  • “12-In-1: Multi-Task Vision and Language Representation Learning”, Lu et al 2019
  • “Deep Double Descent: We Show That the Double Descent Phenomenon Occurs in CNNs, ResNets, and Transformers: Performance First Improves, Then Gets Worse, and Then Improves Again With Increasing Model Size, Data Size, or Training Time”, Nakkiran et al 2019
  • “Deep Double Descent: Where Bigger Models and More Data Hurt”, Nakkiran et al 2019
  • “What’s Hidden in a Randomly Weighted Neural Network?”, Ramanujan et al 2019
  • “Understanding the Generalization of ‘Lottery Tickets’ in Neural Networks”, Morcos & Tian2019
  • “The Deep Learning Revolution and Its Implications for Computer Architecture and Chip Design”, Dean2019
  • “Momentum Contrast for Unsupervised Visual Representation Learning”, He et al 2019
  • “SimpleShot: Revisiting Nearest-Neighbor Classification for Few-Shot Learning”, Wang et al 2019
  • “Self-Training With Noisy Student Improves ImageNet Classification”, Xie et al 2019
  • “CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB”, Schwenk et al 2019
  • “CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs”, El-Kishky et al 2019
  • “XLM-R: State-Of-The-Art Cross-Lingual Understanding through Self-Supervision”, FAIR 2019
  • “High Fidelity Video Prediction With Large Stochastic Recurrent Neural Networks”, Villegas et al 2019
  • “Unsupervised Cross-Lingual Representation Learning at Scale”, Conneau et al 2019
  • “T5: Exploring the Limits of Transfer Learning With a Unified Text-To-Text Transformer”, Raffel et al 2019
  • “ZeRO: Memory Optimizations Toward Training Trillion Parameter Models”, Rajbhandari et al 2019
  • “Environmental Drivers of Systematicity and Generalization in a Situated Agent”, Hill et al 2019
  • “A Constructive Prediction of the Generalization Error Across Scales”, Rosenfeld et al 2019
  • “Large-Scale Pretraining for Neural Machine Translation With Tens of Billions of Sentence Pairs”, Meng et al 2019
  • “UNITER: UNiversal Image-TExt Representation Learning”, Chen et al 2019
  • “Exascale Deep Learning for Scientific Inverse Problems”, Laanait et al 2019
  • “Simple, Scalable Adaptation for Neural Machine Translation”, Bapna et al 2019
  • “CTRL: A Conditional Transformer Language Model For Controllable Generation”, Keskar et al 2019
  • “Show Your Work: Improved Reporting of Experimental Results”, Dodge et al 2019
  • “MegatronLM: Training Billion+ Parameter Language Models Using GPU Model Parallelism”, ADLR 2019
  • “RoBERTa: A Robustly Optimized BERT Pretraining Approach”, Liu et al 2019
  • “Robustness Properties of Facebook’s ResNeXt WSL Models”, Orhan2019
  • “Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges”, Arivazhagan et al 2019
  • “Large Scale Adversarial Representation Learning”, Donahue & Simonyan2019
  • “One Epoch Is All You Need”, Komatsuzaki2019
  • “Does Learning Require Memorization? A Short Tale about a Long Tail”, Feldman2019
  • “Intriguing Properties of Adversarial Training at Scale”, Xie & Yuille2019
  • “Scaling Autoregressive Video Models”, Weissenborn et al 2019
  • “A Mathematical Theory of Semantic Development in Deep Neural Networks”, Saxe et al 2019
  • “Adversarially Robust Generalization Just Requires More Unlabeled Data”, Zhai et al 2019
  • “ICML 2019 Notes”, Abel2019
  • “Are Labels Required for Improving Adversarial Robustness?”, Uesato et al 2019
  • “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks”, Tan & Le2019
  • “SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers”, Fedorov et al 2019
  • “Asymptotic Learning Curves of Kernel Methods: Empirical Data versus Teacher-Student Paradigm”, Spigler et al 2019
  • “UniLM: Unified Language Model Pre-Training for Natural Language Understanding and Generation”, Dong et al 2019
  • “Adversarial Examples Are Not Bugs, They Are Features”, Ilyas et al 2019
  • “Billion-Scale Semi-Supervised Learning for Image Classification”, Yalniz et al 2019
  • “VideoBERT: A Joint Model for Video and Language Representation Learning”, Sun et al 2019
  • “Benchmarking Neural Network Robustness to Common Corruptions and Perturbations”, Hendrycks & Dietterich2019
  • “Surprises in High-Dimensional Ridgeless Least Squares Interpolation”, Hastie et al 2019
  • “The Bitter Lesson”, Sutton2019
  • “GPT-2 As Step Toward General Intelligence”, Alexander2019
  • “Deep Learning Hardware: Past, Present, & Future”, LeCun2019
  • “Language Models Are Unsupervised Multitask Learners”, Radford et al 2019
  • “Better Language Models and Their Implications”, Radford et al 2019
  • “Do ImageNet Classifiers Generalize to ImageNet?”, Recht et al 2019
  • “Cross-Lingual Language Model Pretraining”, Lample & Conneau2019
  • “Artificial Intelligence: A Guide for Thinking Humans § Prologue: Terrified”, Mitchell2019
  • “High Fidelity Video Prediction With Large Stochastic Recurrent Neural Networks: Videos”, Villegas et al 2019
  • “Reconciling Modern Machine Learning Practice and the Bias-Variance Trade-Off”, Belkin et al 2018
  • “Nocaps: Novel Object Captioning at Scale”, Agrawal et al 2018
  • “On Lazy Training in Differentiable Programming”, Chizat et al 2018
  • “How AI Training Scales”, McCandlish et al 2018
  • “Is Science Slowing Down?”, Alexander2018
  • “Large Scale GAN Training for High Fidelity Natural Image Synthesis”, Brock et al 2018
  • “BigGAN: Large Scale GAN Training For High Fidelity Natural Image Synthesis § 5.2 Additional Evaluation On JFT-300M”, Brock et al 2018 (page 8 org deepmind)
  • “Measurement Invariance Explains the Universal Law of Generalization for Psychological Perception”, Frank2018
  • “CurriculumNet: Weakly Supervised Learning from Large-Scale Web Images”, Guo et al 2018
  • “Large-Scale Visual Speech Recognition”, Shillingford et al 2018
  • “Troubling Trends in Machine Learning Scholarship”, Lipton & Steinhardt2018
  • “Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations”, Hendrycks & Dietterich2018
  • “Neural Scene Representation and Rendering”, Eslami et al 2018
  • “GPT-1: Improving Language Understanding With Unsupervised Learning”, OpenAI 2018
  • “GPT-1: Improving Language Understanding by Generative Pre-Training”, Radford et al 2018
  • “GPT-1: Improving Language Understanding by Generative Pre-Training § Model Specifications”, Radford et al 2018 (page 5)
  • “Do CIFAR-10 Classifiers Generalize to CIFAR-10?”, Recht et al 2018
  • “Deep Learning Generalizes Because the Parameter-Function Map Is Biased towards Simple Functions”, Valle-Pérez et al 2018
  • “Google DeepMind Founder and Leader in Artificial Intelligence Returns to Hamilton”, Tantau2018
  • “Exploring the Limits of Weakly Supervised Pretraining”, Mahajan et al 2018
  • “One Big Net For Everything”, Schmidhuber2018
  • “Sensitivity and Generalization in Neural Networks: an Empirical Study”, Novak et al 2018
  • “ULMFiT: Universal Language Model Fine-Tuning for Text Classification”, Howard & Ruder2018
  • “GPipe: Easy Scaling With Micro-Batch Pipeline Parallelism § Pg4”, Huang2018 (page 4 org google)
  • “Deep Image Reconstruction from Human Brain Activity”, Shen et al 2017
  • “Deep Learning Scaling Is Predictable, Empirically”, Hestness et al 2017
  • “Are GANs Created Equal? A Large-Scale Study”, Lucic et al 2017
  • “Knowledge Concentration: Learning 100K Object Classifiers in a Single CNN”, Gao et al 2017
  • “Rethinking Generalization Requires Revisiting Old Ideas: Statistical Mechanics Approaches and Complex Learning Behavior”, Martin & Mahoney2017
  • “There’s No Fire Alarm for Artificial General Intelligence”, Yudkowsky2017
  • “WebVision Database: Visual Learning and Understanding from Web Data”, Li et al 2017
  • “Revisiting Unreasonable Effectiveness of Data in Deep Learning Era”, Sun et al 2017
  • “Towards Deep Learning Models Resistant to Adversarial Attacks”, Madry et al 2017
  • “Gradient Diversity: a Key Ingredient for Scalable Distributed Learning”, Yin et al 2017
  • “Learning to Learn from Noisy Web Videos”, Yeung et al 2017
  • “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour”, Goyal et al 2017
  • “A Simple Neural Network Module for Relational Reasoning”, Santoro et al 2017
  • “Deep Learning Is Robust to Massive Label Noise”, Rolnick et al 2017
  • “Quo Vadis, Action Recognition? A New Model I3D and the Kinetics Dataset”, Carreira & Zisserman2017
  • “WebVision Challenge: Visual Learning and Understanding With Web Data”, Li et al 2017
  • “Geometry of Optimization and Implicit Regularization in Deep Learning”, Neyshabur et al 2017
  • “On the Impossibility of Supersized Machines”, Garfinkel et al 2017
  • “Parallel Multiscale Autoregressive Density Estimation”, Reed et al 2017
  • “Universal Representations: The Missing Link between Faces, Text, Planktons, and Cat Breeds”, Bilen & Vedaldi2017
  • “Estimation of Gap Between Current Language Models and Human Performance”, Shen et al 2017
  • “Simple and Scalable Predictive Uncertainty Estimation Using Deep Ensembles”, Lakshminarayanan et al 2016
  • “Understanding Deep Learning Requires Rethinking Generalization”, Zhang et al 2016
  • “Why Does Deep and Cheap Learning Work so Well?”, Lin et al 2016
  • “The LAMBADA Dataset: Word Prediction Requiring a Broad Discourse Context”, Paperno et al 2016
  • “Residual Networks Behave Like Ensembles of Relatively Shallow Networks”, Veit et al 2016
  • “Do Deep Convolutional Nets Really Need to Be Deep and Convolutional?”, Urban et al 2016
  • “PlaNet—Photo Geolocation With Convolutional Neural Networks”, Weyand et al 2016
  • “Exploring the Limits of Language Modeling”, Jozefowicz et al 2016
  • “The Singularity: A Philosophical Analysis”, Chalmers2016
  • “Microsoft Researchers Win ImageNet Computer Vision Challenge”, Linn2015
  • “The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition”, Krause et al 2015
  • “Net2Net: Accelerating Learning via Knowledge Transfer”, Chen et al 2015
  • “Generative Concatenative Nets Jointly Learn to Write and Classify Reviews”, Lipton et al 2015
  • “Learning Visual Features from Large Weakly Supervised Data”, Joulin et al 2015
  • “LSUN: Construction of a Large-Scale Image Dataset Using Deep Learning With Humans in the Loop”, Yu et al 2015
  • “Clothing-1M: Learning from Massive Noisy Labeled Data for Image Classification”, Xiao et al 2015
  • “The Unreasonable Effectiveness of Recurrent Neural Networks”, Karpathy2015
  • “LSTM: A Search Space Odyssey”, Greff et al 2015
  • “YFCC100M: The New Data in Multimedia Research”, Thomee et al 2015
  • “Machine Intelligence, Part 1”, Altman2015
  • “Evolution of the Human Brain: From Matter to Mind”, Hofman2015
  • “In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning”, Neyshabur et al 2014
  • “Jumping NLP Curves: A Review of Natural Language Processing Research [Review Article]”, Cambria & White2014
  • “Neural Networks, Manifolds, and Topology”, Olah2014
  • “Computing’s Energy Problem (and What We Can Do about It)”, Horowitz2014b
  • “N-Gram Counts and Language Models from the Common Crawl”, Buck et al 2014
  • “Evolution of the Human Brain: When Bigger Is Better”, Hofman2014
  • “One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling”, Chelba et al 2013
  • “Algorithmic Progress in Six Domains”, Grace2013
  • “Large–Scale Machine Learning Revisited [Slides]”, Bottou2013
  • “Intelligence Explosion Microeconomics”, Yudkowsky2013
  • “Scalable Modified Kneser-Ney Language Model Estimation”, Heafield et al 2013
  • “The Remarkable, yet Not Extraordinary, Human Brain As a Scaled-Up Primate Brain and Its Associated Cost”, Herculano-Houzel2012
  • “Advantages of Artificial Intelligences, Uploads, and Digital Minds”, Sotala2012
  • “Recurrent Neural Network Based Language Model”, Mikolov et al 2010
  • “Understanding Sources of Inefficiency in General-Purpose Chips”, Hameed et al 2010
  • “The Teenies”, Legg2009
  • “Tick, Tock, Tick, Tock… BING”, Legg2009
  • “Halloween Nightmare Scenario, Early 2020’s”, Wood2009
  • “The Unreasonable Effectiveness of Data”, Halevy et al 2009
  • “Economics Of The Singularity: Stuffed into Skyscrapers by the Billion, Brainy Bugbots Will Be the Knowledge Workers of the Future”, Hanson2008
  • “Large Language Models in Machine Translation”, Brants et al 2007
  • “The Tradeoffs of Large-Scale Learning”, Bottou & Bousquet2007
  • “Cellular Scaling Rules for Primate Brains”, Herculano-Houzel et al 2007
  • “Robot Predictions Evolution”, Moravec2004
  • “Tree Induction vs. Logistic Regression: A Learning-Curve Analysis”, Perlich et al 2003
  • “Analytic and Algorithmic Solution of Random Satisfiability Problems”, Mezard et al 2002
  • “A Bit of Progress in Language Modeling”, Goodman2001
  • “Scaling to Very Very Large Corpora for Natural Language Disambiguation”, Banko & Brill2001
  • “On Discriminative vs. Generative Classifiers: A Comparison of Logistic Regression and Naive Bayes”, Ng & Jordan2001
  • “A Survey of Methods for Scaling Up Inductive Algorithms”, Provost & Kolluri1999
  • “On The Effect of Data Set Size on Bias And Variance in Classification Learning”, Brain & Webb1999
  • “The Anatomy of a Large-Scale Hypertextual Web Search Engine”, Brin & Page1998
  • “The Effects of Training Set Size on Decision Tree Complexity”, Oates & Jensen1997
  • “Rigorous Learning Curve Bounds from Statistical Mechanics”, Haussler et al 1996
  • “Scaling up the Accuracy of Naive-Bayes Classifiers: a Decision-Tree Hybrid”, Kohavi1996
  • “Reflections After Refereeing Papers for NIPS”, Breiman1995
  • “Building a Large Annotated Corpus of English: The Penn Treebank”, Marcus et al 1993
  • “Statistical Theory of Learning Curves under Entropic Loss Criterion”, Amari & Murata1993
  • “Learning Curves: Asymptotic Values and Rate of Convergence”, Cortes et al 1993
  • “Exhaustive Learning”, Schwartz et al 1990
  • “Computing With Connections”, Sejnowski1987
  • “Don’t Worry—It Can’t Happen”, Harrington1940
  • “Eric Michaud on Neural Quantum Interpretability”
  • “Billion-Scale Semi-Supervised Learning for State-Of-The-Art Image and Video Classification”
  • “No Physics? No Problem. AI Weather Forecasting Is Already Making Huge Strides.”
  • “Report Describes Apple’s ‘Organizational Dysfunction’ and ‘Lack of Ambition’ in AI”
  • “StyleGAN-2 512px Trained on Danbooru2019”
  • “Blake Bordelon”, Bordelon2024
  • “Google Workloads for Consumer Devices: Mitigating Data Movement Bottlenecks”
  • “Komodo 8: the Smartphone vs Desktop Challenge”
  • “Trading Off Compute in Training and Inference § Pruning”
  • “How Can We Make Robotics More like Generative Modeling?”
  • “Inverse-Scaling/prize: A Prize for Finding Tasks That Cause Large Language Models to Show Inverse Scaling”
  • “Scaling up StyleGAN-2”
  • “Semi Supervised Learning”
  • “Homepage of Paul F. Christiano”, Christiano2024
  • “Statistical Modeling: The Two Cultures”, Breiman2024
  • “Jared Kaplan”
  • “Safe Superintelligence Inc.”
  • “OpenAI Disbands Its Robotics Research Team”
  • “The Uneasy Relationship between Deep Learning and (classical) Statistics”
  • “Parameter Counts in Machine Learning”
  • “Can LLMs Learn from a Single Example?”
  • “Deciphering China’s AI Dream”
  • “Appendix: More Is Different In Other Domains”
  • “Understanding ‘Deep Double Descent’”
  • “How Much Compute Was Used to Train DeepMind’s Generally Capable Agents?”
  • “Why Neural Networks Generalise, and Why They Are (Kind Of) Bayesian”
  • “What’s the Backward-Forward FLOP Ratio for Neural Networks?”
  • “Optimality Is the Tiger, and Agents Are Its Teeth”
  • “What Next? A Dozen Information-Technology Research Goals: 3. Turing’s Vision of Machine Intelligence”
  • “Was Linguistic A.I. Created by Accident?”
  • “Ilya Sutskever: Deep Learning | AI Podcast #94 With Lex Fridman”
  • “A Universal Law of Robustness”
  • “Greg Brockman: OpenAI and AGI”, Brockman2024
  • “Season 1 Ep. 22 OpenAI’s Ilya Sutskever: The Man Who Made AI Work”
  • “A Law of Robustness and the Importance of Overparameterization in Deep Learning”
  • “WELM”
  • Sort By Magic
    • philosophy
    • learning-dynamics
    • scaling-laws
  • Wikipedia
• Miscellaneous
• Bibliography