“Context on the NVIDIA ChatGPT Opportunity—and Ramifications of Large Language Model Enthusiasm”, Stanley2023

“Context on the NVIDIA ChatGPT opportunity—and ramifications of large language model enthusiasm”⁠, Morgan Stanley2023-02-10 (GPT-4⁠, GPT-5⁠, AI scaling/economics; backlinks; similar; bibliography)

“Microsoft and OpenAI Extend Partnership”, Microsoft2023

“Microsoft and OpenAI extend partnership”⁠, Microsoft2023-01-23 (AI scaling/economics; backlinks; similar; bibliography)

“Efficiently Scaling Transformer Inference”, Pope Et Al 2022

“Efficiently Scaling Transformer Inference”⁠, Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek2022-11-09 (reduced-precision NNs⁠, self-attention⁠, PaLM; similar; bibliography)

“Reserve Capacity of NVIDIA HGX H100s on CoreWeave Now: Available at Scale in Q1 2023 Starting at $2.23/​hr”, CoreWeave2022

“Petals: Collaborative Inference and Fine-tuning of Large Models”, Borzunov Et Al 2022

“Petals: Collaborative Inference and Fine-tuning of Large Models”⁠, Alexander Borzunov, Dmitry Baranchuk, Tim Dettmers, Max Ryabinin, Younes Belkada, Artem Chumachenko, Pavel Samygin2022-09-02 (GPT; similar)

“Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training”, You Et Al 2022

“Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training”⁠, Jie You, Jae-Won Chung, Mosharaf Chowdhury2022-08-12 (neural net⁠, Bayes; similar)

“Is Integer Arithmetic Enough for Deep Learning Training?”, Ghaffari Et Al 2022

“Is Integer Arithmetic Enough for Deep Learning Training?”⁠, Alireza Ghaffari, Marzieh S. Tahaei, Mohammadreza Tayaranian, Masoud Asgharian, Vahid Partovi Nia2022-07-18 (reduced-precision NNs; similar)

“Efficient NLP Inference at the Edge via Elastic Pipelining”, Guo Et Al 2022

“Efficient NLP Inference at the Edge via Elastic Pipelining”⁠, Liwei Guo, Wonkyo Choe, Felix Xiaozhu Lin2022-07-11 (similar)

“Training Transformers Together”, Borzunov Et Al 2022

“Training Transformers Together”⁠, Alexander Borzunov, Max Ryabinin, Tim Dettmers, Quentin Lhoest, Lucile Saulnier, Michael Diskin, Yacine Jernite2022-07-07 (DALL·E 2; backlinks; similar)

“Tutel: Adaptive Mixture-of-Experts at Scale”, Hwang Et Al 2022

“Tutel: Adaptive Mixture-of-Experts at Scale”⁠, Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose2022-06-07 (MoE NN; similar; bibliography)

“8-bit Numerical Formats for Deep Neural Networks”, Noune Et Al 2022

“8-bit Numerical Formats for Deep Neural Networks”⁠, Badreddine Noune, Philip Jones, Daniel Justus, Dominic Masters, Carlo Luschi2022-06-06 (reduced-precision NNs; similar)

“ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers”, Yao Et Al 2022

“ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers”⁠, Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, Yuxiong He2022-06-04 (knowledge distillation⁠, reduced-precision NNs⁠, GPT; similar; bibliography)

“FlashAttention: Fast and Memory-Efficient Exact Attention With IO-Awareness”, Dao Et Al 2022

“FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness”⁠, Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré2022-05-27 (self-attention⁠, GPT; backlinks; similar; bibliography)

“A Low-latency Communication Design for Brain Simulations”, Du2022

“A Low-latency Communication Design for Brain Simulations”⁠, Xin Du2022-05-14 (neuroscience; similar)

“Reducing Activation Recomputation in Large Transformer Models”, Korthikanti Et Al 2022

“Reducing Activation Recomputation in Large Transformer Models”⁠, Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, Bryan Catanzaro2022-05-10 (similar)

“What Language Model to Train If You Have One Million GPU Hours?”, Scao Et Al 2022

“What Language Model to Train if You Have One Million GPU Hours?”⁠, Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M. Saiful Bari, Stella Biderman2022-04-11 (GPT; similar)

“Monarch: Expressive Structured Matrices for Efficient and Accurate Training”, Dao Et Al 2022

“Monarch: Expressive Structured Matrices for Efficient and Accurate Training”⁠, Tri Dao, Beidi Chen, Nimit Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao2022-04-01 (NN sparsity⁠, CS/algorithm; similar; bibliography)

“Pathways: Asynchronous Distributed Dataflow for ML”, Barham Et Al 2022

“Pathways: Asynchronous Distributed Dataflow for ML”⁠, Paul Barham, Aakanksha Chowdhery, Jeff Dean, Sanjay Ghemawat, Steven Hand, Dan Hurt, Michael Isard, Hyeontaek Lim2022-03-23 (T5 Transformer; similar)

“Singularity: Planet-Scale, Preemptive and Elastic Scheduling of AI Workloads”, Shukla Et Al 2022

“Singularity: Planet-Scale, Preemptive and Elastic Scheduling of AI Workloads”⁠, Dharma Shukla, Muthian Sivathanu, Srinidhi Viswanatha, Bhargav Gulavani, Rimma Nehme, Amey Agrawal, Chen Chen2022-02-16 (similar)

“Maximizing Communication Efficiency for Large-scale Training via 0/​1 Adam”, Lu Et Al 2022

“Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam”⁠, Yucheng Lu, Conglong Li, Minjia Zhang, Christopher De Sa, Yuxiong He2022-02-12 (reduced-precision NNs; similar; bibliography)

“Introducing the AI Research SuperCluster—Meta’s Cutting-edge AI Supercomputer for AI Research”, Lee & Sengupta2022

“Introducing the AI Research SuperCluster—Meta’s cutting-edge AI supercomputer for AI research”⁠, Kevin Lee, Shubho Sengupta2022-01-24 (similar; bibliography)

“Is Programmable Overhead Worth The Cost? How Much Do We Pay for a System to Be Programmable? It Depends upon Who You Ask”, Bailey2022

“Is Programmable Overhead Worth The Cost? How much do we pay for a system to be programmable? It depends upon who you ask”⁠, Brian Bailey2022-01-13 (reduced-precision NNs⁠, CS/hardware; backlinks; similar; bibliography)

“Spiking Neural Networks and Their Applications: A Review”, Yamazaki Et Al 2022

“Spiking Neural Networks and Their Applications: A Review”⁠, Kashu Yamazaki, Viet-Khoa Vo-Ho, Darshan Bulsara, Ngan Le2022 (NN sparsity⁠, neuroscience; similar)

“On the Working Memory of Humans and Great Apes: Strikingly Similar or Remarkably Different?”, Read Et Al 2021

“On the Working Memory of Humans and Great Apes: Strikingly Similar or Remarkably Different?”⁠, Dwight W. Read, Héctor M. Manrique, Michael J. Walker2021-12-14 (DNB⁠, IQ⁠, neuroscience; similar)

“SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient”, Ryabinin Et Al 2021

“SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient”⁠, Max Ryabinin, Tim Dettmers, Michael Diskin, Alexander Borzunov2021-11-23 (backlinks; similar)

“M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining”, Lin Et Al 2021

“M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining”⁠, Junyang Lin, An Yang, Jinze Bai, Chang Zhou, Le Jiang, Xianyan Jia, Ang Wang, Jie Zhang, Yong Li, Wei Lin2021-11-17 (GPT; similar)

“Sustainable AI: Environmental Implications, Challenges and Opportunities”, Wu Et Al 2021

“Sustainable AI: Environmental Implications, Challenges and Opportunities”⁠, Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang2021-10-30 (AI scaling/economics; similar)

“China Has Already Reached Exascale—On Two Separate Systems”, Hemsoth2021

“China Has Already Reached Exascale—On Two Separate Systems”⁠, Nicole Hemsoth2021-10-26 (backlinks; similar)

“The Efficiency Misnomer”, Dehghani Et Al 2021

“The Efficiency Misnomer”⁠, Mostafa Dehghani, Anurag Arnab, Lucas Beyer, Ashish Vaswani, Yi Tay2021-10-25 (CNN⁠, Transformer NN; similar)

“Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning”, Rudin Et Al 2021

“Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning”⁠, Nikita Rudin, David Hoeller, Philipp Reist, Marco Hutter2021-09-24 (robotics⁠, RL scaling; similar)

“WarpDrive: Extremely Fast End-to-End Deep Multi-Agent Reinforcement Learning on a GPU”, Lan Et Al 2021

“WarpDrive: Extremely Fast End-to-End Deep Multi-Agent Reinforcement Learning on a GPU”⁠, Tian Lan, Sunil Srinivasa, Stephan Zheng2021-08-31 (MARL⁠, RL scaling; similar)

“Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning”, Makoviychuk Et Al 2021

“Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning”⁠, Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller2021-08-24 (robotics⁠, RL scaling; similar)

“PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management”, Fang Et Al 2021

“PatrickStar: Parallel Training of Pre-trained Models via Chunk-based Memory Management”⁠, Jiarui Fang, Yang Yu, Zilin Zhu, Shenggui Li, Yang You, Jie Zhou2021-08-12 (similar)

“Demonstration of Decentralized, Physics-Driven Learning”, Dillavou Et Al 2021

“Demonstration of Decentralized, Physics-Driven Learning”⁠, Sam Dillavou, Menachem Stern, Andrea J. Liu, Douglas J. Durian2021-07-31 (similar)

“Chimera: Efficiently Training Large-Scale Neural Networks With Bidirectional Pipelines”, Li & Hoefler2021

“Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines”⁠, Shigang Li, Torsten Hoefler2021-07-14 (backlinks; similar)

“First-Generation Inference Accelerator Deployment at Facebook”, Anderson Et Al 2021

“First-Generation Inference Accelerator Deployment at Facebook”⁠, Michael Anderson, Benny Chen, Stephen Chen, Summer Deng, Jordan Fix, Michael Gschwind, Aravind Kalaiah2021-07-08 (AI scaling/economics; similar)

“Single-chip Photonic Deep Neural Network for Instantaneous Image Classification”, Ashtiani Et Al 2021

“Single-chip photonic deep neural network for instantaneous image classification”⁠, Farshid Ashtiani, Alexander J. Geers, Firooz Aflatouni2021-06-19 (similar)

“Distributed Deep Learning in Open Collaborations”, Diskin Et Al 2021

“Distributed Deep Learning in Open Collaborations”⁠, Michael Diskin, Alexey Bukhtiyarov, Max Ryabinin, Lucile Saulnier, Quentin Lhoest, Anton Sinitsin, Dmitry Popov2021-06-18 (backlinks; similar; bibliography)

“Ten Lessons From Three Generations Shaped Google’s TPUv4i”, Jouppi Et Al 2021

“Ten Lessons From Three Generations Shaped Google’s TPUv4i”⁠, Norman P. Jouppi, Doe Hyun Yoon, Matthew Ashcraft, Mark Gottscho, Thomas B. Jablin, George Kurian, James Laudon2021-06-14 (RNN⁠, reduced-precision NNs; similar; bibliography)

“2.5-dimensional Distributed Model Training”, Wang Et Al 2021

“2.5-dimensional distributed model training”⁠, Boxiang Wang, Qifan Xu, Zhengda Bian, Yang You2021-05-30 (similar)

“Maximizing 3-D Parallelism in Distributed Training for Huge Neural Networks”, Bian Et Al 2021

“Maximizing 3-D Parallelism in Distributed Training for Huge Neural Networks”⁠, Zhengda Bian, Qifan Xu, Boxiang Wang, Yang You2021-05-30 (Transformer NN; similar)

“A Full-stack Accelerator Search Technique for Vision Applications”, Zhang Et Al 2021

“A Full-stack Accelerator Search Technique for Vision Applications”⁠, Dan Zhang, Safeen Huda, Ebrahim Songhori, Quoc Le, Anna Goldie, Azalia Mirhoseini2021-05-26 (meta-learning; similar)

“ChinAI #141: The PanGu Origin Story: Notes from an Informative Zhihu Thread on PanGu”, Ding2021

“ChinAI #141: The PanGu Origin Story: Notes from an informative Zhihu Thread on PanGu”⁠, Jeffrey Ding2021-05-17 (similar; bibliography)

“GSPMD: General and Scalable Parallelization for ML Computation Graphs”, Xu Et Al 2021

“GSPMD: General and Scalable Parallelization for ML Computation Graphs”⁠, Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin2021-05-10 (similar)

“PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models With Auto-parallel Computation”, Zeng Et Al 2021

“PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation”⁠, Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang, Xin Jiang, ZhenZhang Yang, Kaisheng Wang2021-04-26 (similar)

“ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning”, Rajbhandari Et Al 2021

“ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning”⁠, Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, Yuxiong He2021-04-16 (similar)

“How to Train BERT With an Academic Budget”, Izsak Et Al 2021

“How to Train BERT with an Academic Budget”⁠, Peter Izsak, Moshe Berchansky, Omer Levy2021-04-15 (experience curves; similar)

“Podracer Architectures for Scalable Reinforcement Learning”, Hessel Et Al 2021

“Podracer architectures for scalable Reinforcement Learning”⁠, Matteo Hessel, Manuel Kroiss, Aidan Clark, Iurii Kemaev, John Quan, Thomas Keck, Fabio Viola, Hado van Hasselt2021-04-13 (meta-learning⁠, model-free RL⁠, MuZero⁠, RL scaling; similar; bibliography)

“An Efficient 2D Method for Training Super-Large Deep Learning Models”, Xu Et Al 2021

“An Efficient 2D Method for Training Super-Large Deep Learning Models”⁠, Qifan Xu, Shenggui Li, Chaoyu Gong, Yang You2021-04-12 (similar)

“High-performance, Distributed Training of Large-scale Deep Learning Recommendation Models (DLRMs)”, Mudigere Et Al 2021

“High-performance, Distributed Training of Large-scale Deep Learning Recommendation Models (DLRMs)”⁠, Dheevatsa Mudigere, Yuchen Hao, Jianyu Huang, Andrew Tulloch, Srinivas Sridharan, Xing Liu, Mustafa Ozdal2021-04-12 (reduced-precision NNs; similar)

“Efficient Large-Scale Language Model Training on GPU Clusters”, Narayanan Et Al 2021

“Efficient Large-Scale Language Model Training on GPU Clusters”⁠, Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay An, Korthikanti2021-04-09 (similar)

“Large Batch Simulation for Deep Reinforcement Learning”, Shacklett Et Al 2021

“Large Batch Simulation for Deep Reinforcement Learning”⁠, Brennan Shacklett, Erik Wijmans, Aleksei Petrenko, Manolis Savva, Dhruv Batra, Vladlen Koltun, Kayvon Fatahalian2021-03-12 (model-free RL⁠, RL scaling; backlinks; similar)

“Warehouse-Scale Video Acceleration (Argos): Co-design and Deployment in the Wild”, Ranganathan Et Al 2021

“Warehouse-Scale Video Acceleration (Argos): Co-design and Deployment in the Wild”⁠, Parthasarathy Ranganathan, Daniel Stodolsky, Jeff Calow, Jeremy Dorfman, Marisabel Guevara, Clinton Wills Smullen IV2021-02-27 (CS/hardware; similar)

“TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models”, Li Et Al 2021

“TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models”⁠, Zhuohan Li, Siyuan Zhuang, Shiyuan Guo, Danyang Zhuo, Hao Zhang, Dawn Song, Ion Stoica2021-02-16 (backlinks; similar; bibliography)

“PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformers”, He Et Al 2021

“PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformers”⁠, Chaoyang He, Shen Li, Mahdi Soltanolkotabi, Salman Avestimehr2021-02-05 (similar; bibliography)

“ZeRO-Offload: Democratizing Billion-Scale Model Training”, Ren Et Al 2021

“ZeRO-Offload: Democratizing Billion-Scale Model Training”⁠, Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li2021-01-18 (similar)

“The Design Process for Google’s Training Chips: TPUv2 and TPUv3”, Norrie Et Al 2021

“The Design Process for Google's Training Chips: TPUv2 and TPUv3”⁠, Thomas Norrie, Nishant Patil, Doe Hyun Yoon, George Kurian, Sheng Li, James Laudon, Cliff Young, Norman Jouppi2021 (similar)

“Hardware Beyond Backpropagation: a Photonic Co-Processor for Direct Feedback Alignment”, Launay Et Al 2020

“Hardware Beyond Backpropagation: a Photonic Co-Processor for Direct Feedback Alignment”⁠, Julien Launay, Iacopo Poli, Kilian Müller, Gustave Pariente, Igor Carron, Laurent Daudet, Florent Krzakala2020-12-11 (similar; bibliography)

“Parallel Training of Deep Networks With Local Updates”, Laskin Et Al 2020

“Parallel Training of Deep Networks with Local Updates”⁠, Michael Laskin, Luke Metz, Seth Nabarrao, Mark Saroufim, Badreddine Noune, Carlo Luschi, Jascha Sohl-Dickstein2020-12-07 (similar)

“Exploring the Limits of Concurrency in ML Training on Google TPUs”, Kumar Et Al 2020

“Exploring the limits of Concurrency in ML Training on Google TPUs”⁠, Sameer Kumar, James Bradbury, Cliff Young, Yu Emma Wang, Anselm Levskaya, Blake Hechtman, Dehao Chen2020-11-07 (similar)

“BytePS: A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/​CPU Clusters”, Jiang Et Al 2020

“BytePS: A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters”⁠, Yimin Jiang, Yibo Zhu, Chang Lan, Bairen Yi, Yong Cui, Chuanxiong Guo2020-11-04 (similar; bibliography)

“Training EfficientNets at Supercomputer Scale: 83% ImageNet Top-1 Accuracy in One Hour”, Wongpanich Et Al 2020

“Training EfficientNets at Supercomputer Scale: 83% ImageNet Top-1 Accuracy in One Hour”⁠, Arissa Wongpanich, Hieu Pham, James Demmel, Mingxing Tan, Quoc Le, Yang You, Sameer Kumar2020-10-30 (similar; bibliography)

“Matrix Engines for High Performance Computing:A Paragon of Performance or Grasping at Straws?”, Domke Et Al 2020

“Matrix Engines for High Performance Computing:A Paragon of Performance or Grasping at Straws?”⁠, Jens Domke, Emil Vatai, Aleksandr Drozd, Peng Chen, Yosuke Oyama, Lingqi Zhang, Shweta Salaria, Daichi Mukunoki2020-10-27 (similar)

“Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures”, Launay Et Al 2020b

“Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures”⁠, Julien Launay, Iacopo Poli, François Boniface, Florent Krzakala2020-10-22 (similar; bibliography)

“L2L: Training Large Neural Networks With Constant Memory Using a New Execution Algorithm”, Pudipeddi Et Al 2020

“L2L: Training Large Neural Networks with Constant Memory using a New Execution Algorithm”⁠, Bharadwaj Pudipeddi, Maral Mesmakhosroshahi, Jinwen Xi, Sujeeth Bharadwaj2020-10-16 (reduced-precision NNs⁠, GPT; similar)

“Interlocking Backpropagation: Improving Depthwise Model-parallelism”, Gomez Et Al 2020

“Interlocking Backpropagation: Improving depthwise model-parallelism”⁠, Aidan N. Gomez, Oscar Key, Kuba Perlin, Stephen Gou, Nick Frosst, Jeff Dean, Yarin Gal2020-10-08 (similar)

“DeepSpeed: Extreme-scale Model Training for Everyone”, Team Et Al 2020

“DeepSpeed: Extreme-scale model training for everyone”⁠, DeepSpeed Team, Rangan Majumder, Junhua Wang2020-09-10 (Transformer NN; backlinks; similar; bibliography)

“Measuring Hardware Overhang”, Hippke 2020

“Measuring hardware overhang”⁠, hippke2020-08-05 (CS/algorithm⁠, CS/hardware⁠, experience curves⁠, AI chess⁠, RL scaling; backlinks; similar)

“The Node Is Nonsense: There Are Better Ways to Measure Progress Than the Old Moore’s Law Metric”, Moore2020

“The Node Is Nonsense: There are better ways to measure progress than the old Moore’s law metric”⁠, Samuel K. Moore2020-07-28 (similar)

“Are We in an AI Overhang?”, Jones2020

“Are we in an AI overhang?”⁠, Andy Jones2020-07-27 (backlinks; similar)

“HOBFLOPS CNNs: Hardware Optimized Bitslice-Parallel Floating-Point Operations for Convolutional Neural Networks”, Garland & Gregg2020

“HOBFLOPS CNNs: Hardware Optimized Bitslice-Parallel Floating-Point Operations for Convolutional Neural Networks”⁠, James Garland, David Gregg2020-07-11 (reduced-precision NNs; similar)

“The Computational Limits of Deep Learning”, Thompson Et Al 2020

“The Computational Limits of Deep Learning”⁠, Neil C. Thompson, Kristjan Greenewald, Keeheon Lee, Gabriel F. Manso2020-07-10 (backlinks; similar)

“Data Movement Is All You Need: A Case Study on Optimizing Transformers”, Ivanov Et Al 2020

“Data Movement Is All You Need: A Case Study on Optimizing Transformers”⁠, Andrei Ivanov, Nikoli Dryden, Tal Ben-Nun, Shigang Li, Torsten Hoefler2020-06-30 (Transformer NN; similar)

“PyTorch Distributed: Experiences on Accelerating Data Parallel Training”, Li Et Al 2020

“PyTorch Distributed: Experiences on Accelerating Data Parallel Training”⁠, Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith2020-06-28 (similar)

“Japanese Supercomputer Is Crowned World’s Speediest: In the Race for the Most Powerful Computers, Fugaku, a Japanese Supercomputer, Recently Beat American and Chinese Machines”, Clark2020

“Japanese Supercomputer Is Crowned World’s Speediest: In the race for the most powerful computers, Fugaku, a Japanese supercomputer, recently beat American and Chinese machines”⁠, Don Clark2020-06-22 (Japan; backlinks; similar)

“Sample Factory: Egocentric 3D Control from Pixels at 100,000 FPS With Asynchronous Reinforcement Learning”, Petrenko Et Al 2020

“Sample Factory: Egocentric 3D Control from Pixels at 100,000 FPS with Asynchronous Reinforcement Learning”⁠, Aleksei Petrenko, Zhehui Huang, Tushar Kumar, Gaurav Sukhatme, Vladlen Koltun2020-06-21 (robotics⁠, RL scaling; similar)

“PipeDream-2BW: Memory-Efficient Pipeline-Parallel DNN Training”, Narayanan Et Al 2020

“PipeDream-2BW: Memory-Efficient Pipeline-Parallel DNN Training”⁠, Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, Matei Zaharia2020-06-16 (Transformer NN; similar)

“There’s Plenty of Room at the Top: What Will Drive Computer Performance After Moore’s Law?”, Leiserson Et Al 2020

“There’s plenty of room at the Top: What will drive computer performance after Moore’s law?”⁠, Charles E. Leiserson, Neil C. Thompson, Joel S. Emer, Bradley C. Kuszmaul, Butler W. Lampson, Daniel Sanchez2020-06-05 (CS/hardware; backlinks; similar)

“A Domain-specific Supercomputer for Training Deep Neural Networks”, Jouppi Et Al 2020

“A domain-specific supercomputer for training deep neural networks”⁠, Norman P. Jouppi, Doe Hyun Yoon, George Kurian, Sheng Li, Nishant Patil, James Laudon, Cliff Young, David Patterson2020-06-01 (similar)

“Microsoft Announces New Supercomputer, Lays out Vision for Future AI Work”, Langston2020

“Microsoft announces new supercomputer, lays out vision for future AI work”⁠, Jennifer Langston2020-05-19 (backlinks; similar; bibliography)

“AI and Efficiency: We’re Releasing an Analysis Showing That Since 2012 the Amount of Compute Needed to Train a Neural Net to the Same Performance on ImageNet Classification Has Been Decreasing by a Factor of 2 Every 16 Months”, Hernandez & Brown2020

“AI and Efficiency: We’re releasing an analysis showing that since 2012 the amount of compute needed to train a neural net to the same performance on ImageNet classification has been decreasing by a factor of 2 every 16 months”⁠, Danny Hernandez, Tom Brown2020-05-05 (CNN⁠, AI scaling/economics⁠, experience curves⁠, forecasting; backlinks; similar)

“Computation in the Human Cerebral Cortex Uses Less Than 0.2 Watts yet This Great Expense Is Optimal When considering Communication Costs”, Levy & Calvert2020

“Computation in the human cerebral cortex uses less than 0.2 watts yet this great expense is optimal when considering communication costs”⁠, William B. Levy, Victoria G. Calvert2020-04-25 (CS/algorithm⁠, neuroscience; similar)

“Startup Tenstorrent Shows AI Is Changing Computing and vice Versa: Tenstorrent Is One of the Rush of AI Chip Makers Founded in 2016 and Finally Showing Product. The New Wave of Chips Represent a Substantial Departure from How Traditional Computer Chips Work, but Also Point to Ways That Neural Network Design May Change in the Years to Come”, Ray2020

“Startup Tenstorrent shows AI is changing computing and vice versa: Tenstorrent is one of the rush of AI chip makers founded in 2016 and finally showing product. The new wave of chips represent a substantial departure from how traditional computer chips work, but also point to ways that neural network design may change in the years to come”⁠, Tiernan Ray2020-04-10 (similar; bibliography)

“AI Chips: What They Are and Why They Matter-An AI Chips Reference”, Khan & Mann2020

“AI Chips: What They Are and Why They Matter-An AI Chips Reference”⁠, Saif M. Khan, Alexander Mann2020-04 (backlinks; similar; bibliography)

“Pipelined Backpropagation at Scale: Training Large Models without Batches”, Kosson Et Al 2020

“Pipelined Backpropagation at Scale: Training Large Models without Batches”⁠, Atli Kosson, Vitaliy Chiley, Abhinav Venigalla, Joel Hestness, Urs Köster2020-03-25 (similar)

“2019 Recent Trends in GPU Price per FLOPS”, Bergal2020

“2019 recent trends in GPU price per FLOPS”⁠, Asya Bergal2020-03-25 (experience curves; backlinks; similar)

“Ultrafast Machine Vision With 2D Material Neural Network Image Sensors”, Mennel Et Al 2020

“Towards Spike-based Machine Intelligence With Neuromorphic Computing”, Roy Et Al 2019

“Towards spike-based machine intelligence with neuromorphic computing”⁠, Kaushik Roy, Akhilesh Jaiswal, Priyadarshini Panda2019-11-27 (similar)

“Checkmate: Breaking the Memory Wall With Optimal Tensor Rematerialization”, Jain Et Al 2019

“Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization”⁠, Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Kurt Keutzer, Ion Stoica, Joseph E. Gonzalez2019-10-07 (similar)

“Training Kinetics in 15 Minutes: Large-scale Distributed Training on Videos”, Lin Et Al 2019

“Training Kinetics in 15 Minutes: Large-scale Distributed Training on Videos”⁠, Ji Lin, Chuang Gan, Song Han2019-10-01 (similar)

“Energy and Policy Considerations for Deep Learning in NLP”, Strubell Et Al 2019

“Energy and Policy Considerations for Deep Learning in NLP”⁠, Emma Strubell, Ananya Ganesh, Andrew McCallum2019-06-05 (Transformer NN; backlinks; similar)

“Large Batch Optimization for Deep Learning: Training BERT in 76 Minutes”, You Et Al 2019

“Large Batch Optimization for Deep Learning: Training BERT in 76 minutes”⁠, Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel2019-04-01 (Transformer NN; similar; bibliography)

“GAP: Generalizable Approximate Graph Partitioning Framework”, Nazi Et Al 2019

“GAP: Generalizable Approximate Graph Partitioning Framework”⁠, Azade Nazi, Will Hang, Anna Goldie, Sujith Ravi, Azalia Mirhoseini2019-03-02 (CS/algorithm; similar)

“An Empirical Model of Large-Batch Training”, McCandlish Et Al 2018

“An Empirical Model of Large-Batch Training”⁠, Sam McCandlish, Jared Kaplan, Dario Amodei, Open A. I. Dota Team2018-12-14 (autoencoder NN⁠, OA5⁠, RL scaling; similar)

“Bayesian Layers: A Module for Neural Network Uncertainty”, Tran Et Al 2018

“Bayesian Layers: A Module for Neural Network Uncertainty”⁠, Dustin Tran, Michael W. Dusenberry, Mark van der Wilk, Danijar Hafner2018-12-10 (RNN⁠, Transformer NN⁠, model-based RL⁠, RL scaling⁠, Bayes; similar)

“GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism”, Huang Et Al 2018

“GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism”⁠, Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Mia Xu Chen, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam2018-11-16 (similar)

“Measuring the Effects of Data Parallelism on Neural Network Training”, Shallue Et Al 2018

“Measuring the Effects of Data Parallelism on Neural Network Training”⁠, Christopher J. Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, George E. Dahl2018-11-08 (similar)

“Mesh-TensorFlow: Deep Learning for Supercomputers”, Shazeer Et Al 2018

“Mesh-TensorFlow: Deep Learning for Supercomputers”⁠, Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins2018-11-05 (similar; bibliography)

“There Is Plenty of Time at the Bottom: the Economics, Risk and Ethics of Time Compression”, Sandberg2018

“There is plenty of time at the bottom: the economics, risk and ethics of time compression”⁠, Anders Sandberg2018-10-30 (AI safety; backlinks; similar)

“Highly Scalable Deep Learning Training System With Mixed-Precision: Training ImageNet in 4 Minutes”, Jia Et Al 2018

“Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in 4 Minutes”⁠, Xianyan Jia, Shutao Song, Wei He, Yangzihao Wang, Haidong Rong, Feihu Zhou, Liqiang Xie, Zhenyu Guo, Yuanzhou Yang2018-07-30 (reduced-precision NNs; similar)

“AI and Compute”, Amodei Et Al 2018

“AI and Compute”⁠, Dario Amodei, Danny Hernandez, Girish Sastry, Jack Clark, Greg Brockman, Ilya Sutskever2018-05-26 (AI scaling/economics⁠, RL scaling; backlinks; similar; bibliography)

“Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions”, Vasilache Et Al 2018

“Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions”⁠, Nicolas Vasilache, Oleksandr Zinenko, Theodoros Theodoridis, Priya Goyal, Zachary DeVito, William S. Moses2018-02-13 (CS/algorithm; backlinks; similar)

“Loihi: A Neuromorphic Manycore Processor With On-Chip Learning”, Davies Et Al 2018

“Loihi: A Neuromorphic Manycore Processor with On-Chip Learning”⁠, Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou2018-01-16 (similar)

“Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training”, Lin Et Al 2017

“Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training”⁠, Yujun Lin, Song Han, Huizi Mao, Yu Wang, William J. Dally2017-12-05 (reduced-precision NNs; similar; bibliography)

“Mixed Precision Training”, Micikevicius Et Al 2017

“Mixed Precision Training”⁠, Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg2017-10-10 (GAN⁠, RNN⁠, reduced-precision NNs; similar)

“On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima”, Keskar Et Al 2016

“On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima”⁠, Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, Ping Tak Peter Tang2016-09-15 (neural net; similar)

“Training Deep Nets With Sublinear Memory Cost”, Chen Et Al 2016

“Training Deep Nets with Sublinear Memory Cost”⁠, Tianqi Chen, Bing Xu, Chiyuan Zhang, Carlos Guestrin2016-04-21 (RNN; backlinks; similar)

“GeePS: Scalable Deep Learning on Distributed GPUs With a GPU-specialized Parameter Server”, Cui Et Al 2016

“GeePS: scalable deep learning on distributed GPUs with a GPU-specialized parameter server”⁠, Henggang Cui, Hao Zhang, Gregory R. Ganger, Phillip B. Gibbons, Eric P. Xing2016-04-01 (similar)

“Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing”, Esser Et Al 2016

“Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing”⁠, Steven K. Esser, Paul A. Merolla, John V. Arthur, Andrew S. Cassidy, Rathinakumar Appuswamy, Alexander Andreopoulos2016-03-28 (CNN⁠, reduced-precision NNs; similar)

“Communication-Efficient Learning of Deep Networks from Decentralized Data”, McMahan Et Al 2016

“Communication-Efficient Learning of Deep Networks from Decentralized Data”⁠, H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Agüera y Arcas2016-02-17 (similar)

“Persistent RNNs: Stashing Recurrent Weights On-Chip”, Diamos Et Al 2016

“Persistent RNNs: Stashing Recurrent Weights On-Chip”⁠, Gregory Diamos, Shubho Sengupta, Bryan Catanzaro, Mike Chrzanowski, Adam Coates, Erich Elsen, Jesse Engel2016-01 (RNN; similar)

“Scaling Distributed Machine Learning With the Parameter Server”, Li Et Al 2014

“Scaling Distributed Machine Learning with the Parameter Server”⁠, Mu Li, David G. Andersen, Jun Woo Park, Alexander J. Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J. Shekita2014-10-06 (similar)

“Multi-column Deep Neural Network for Traffic Sign Classification”, Cireşan Et Al 2012

“Multi-column deep neural network for traffic sign classification”⁠, Dan Cireşan, Ueli Meier, Jonathan Masci, Jürgen Schmidhuber2012-08 (CNN; backlinks; similar)

“Slowing Moore’s Law: How It Could Happen”, Branwen2012

“Slowing Moore’s Law: How It Could Happen”⁠, Gwern Branwen2012-03-16 (AI scaling/economics⁠, terrorism⁠, CS/hardware⁠, economics; backlinks; similar; bibliography)

“Multi-column Deep Neural Networks for Image Classification”, Cireşan Et Al 2012

“Multi-column Deep Neural Networks for Image Classification”⁠, Dan Cireşan, Ueli Meier, Juergen Schmidhuber2012-02-13 (CNN; similar)

“HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent”, Niu Et Al 2011

“HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent”⁠, Feng Niu, Benjamin Recht, Christopher Re, Stephen J. Wright2011-06-28 (neural net; backlinks; similar)

“DanNet: Flexible, High Performance Convolutional Neural Networks for Image Classification”, Ciresan Et Al 2011

“DanNet: Flexible, High Performance Convolutional Neural Networks for Image Classification”⁠, Dan Claudiu Ciresan, Ueli Meier, Jonathan Masci, Luca Maria Gambardella, Jürgen Schmidhuber2011-02-01 (CNN; similar; bibliography)

“Goodbye 2010”, Legg2010

“Goodbye 2010”⁠, Shane Legg2010-10-10 (forecasting; backlinks; similar)

“Bandwidth Optimal All-reduce Algorithms for Clusters of Workstations”, Patarasuk & Yuan2009

“Bandwidth optimal all-reduce algorithms for clusters of workstations”⁠, Pitch Patarasuk, Xin Yuan2009-02-01 (similar; bibliography)

“Whole Brain Emulation: A Roadmap”

“Moore’s Law and the Technology S-Curve”, Bowden2004

“Ultimate Physical Limits to Computation”, Lloyd1999

“Ultimate physical limits to computation”⁠, Seth Lloyd1999-08-13 (CS/hardware⁠, science; backlinks; similar)

“When Will Computer Hardware Match the Human Brain?”, Moravec1998

“When will computer hardware match the human brain?”⁠, Hans Moravec1998 (AI chess⁠, robotics⁠, RL scaling; backlinks; similar)

“A Sociological Study of the Official History of the Perceptrons Controversy [1993]”, Olazaran1993

“A Sociological Study of the Official History of the Perceptrons Controversy [1993]”⁠, Mikel Olazaran1993-08 (neural net⁠, philosophy/epistemology; backlinks; similar; bibliography)

“Intelligence As an Emergent Behavior; Or, The Songs of Eden”, Hillis1988

“TensorFlow Research Cloud (TRC): Accelerate Your Cutting-edge Machine Learning Research With Free Cloud TPUs”, TRC 2023

“TensorFlow Research Cloud (TRC): Accelerate your cutting-edge machine learning research with free Cloud TPUs”⁠, TRC (anime AI⁠, StyleGAN⁠, GPT⁠, Google; backlinks; similar)

“48:44—Tesla Vision · 1:13:12—Planning and Control · 1:24:35—Manual Labeling · 1:28:11—Auto Labeling · 1:35:15—Simulation · 1:42:10—Hardware Integration · 1:45:40—Dojo”

Wikipedia

1. ASML Holding

2. Ampere (microarchitecture)

3. Extreme ultraviolet lithography

4. GPGPU

5. HBM2

6. HBM3

7. Hopper (microarchitecture)

8. Minifloat

9. Nvidia DGX

10. Nvidia Tesla

11. Spiking neural network

12. Summit (supercomputer)

13. TSMC

14. Tianhe-1A