“Learning Humanoid Locomotion With Transformers”, Radosavovic Et Al 2023

“Learning Humanoid Locomotion with Transformers”⁠, Ilija Radosavovic, Tete Xiao, Bike Zhang, Trevor Darrell, Jitendra Malik, Koushil Sreenath2023-03-06 (Decision Transformer⁠, robotics; similar)

“ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematics”, Azerbayev Et Al 2023

“ProofNet: Autoformalizing and Formally Proving Undergraduate-Level Mathematics”⁠, Zhangir Azerbayev, Bartosz Piotrowski, Hailey Schoelkopf, Edward W. Ayers, Dragomir Radev, Jeremy Avigad2023-02-24 (retrieval AI⁠, Codex⁠, dataset⁠, math; similar; bibliography)

“Scaling Vision Transformers to 22 Billion Parameters”, Dehghani Et Al 2023

“Scaling Vision Transformers to 22 Billion Parameters”⁠, Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner2023-02-10 (Transformer NN⁠, AI scaling⁠, video analysis; similar; bibliography)

“BMT: Binarized Neural Machine Translation”, Zhang Et Al 2023

“BMT: Binarized Neural Machine Translation”⁠, Yichi Zhang, Ankush Garg, Yuan Cao, Łukasz Lew, Behrooz Ghorbani, Zhiru Zhang, Orhan Firat2023-02-09 (reduced-precision NNs⁠, Transformer NN; similar; bibliography)

“Use GPT-3 Incorrectly: Reduce Costs 40× and Increase Speed by 5×”, Pullen2023

“Use GPT-3 incorrectly: reduce costs 40× and increase speed by 5×”⁠, Alistar Pullen2023-02-06 (GPT⁠, preference learning; backlinks; similar)

“TinyMIM: An Empirical Study of Distilling MIM Pre-trained Models”, Ren Et Al 2023

“TinyMIM: An Empirical Study of Distilling MIM Pre-trained Models”⁠, Sucheng Ren, Fangyun Wei, Zheng Zhang, Han Hu2023-01-03 (masked auto-encoder; similar; bibliography)

“Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints”, Komatsuzaki Et Al 2022

“Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpoints”⁠, Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie2022-12-09 (T5 Transformer⁠, MoE NN; similar; bibliography)

“MaskDistill: A Unified View of Masked Image Modeling”, Anonymous2022

“MaskDistill: A Unified View of Masked Image Modeling”⁠, Anonymous2022-11-17 (CLIP⁠, masked auto-encoder; similar; bibliography)

“Distilled DeepConsensus: Knowledge Distillation for Fast and Accurate DNA Sequence Correction”, Belyaeva Et Al 2022

“Distilled DeepConsensus: Knowledge distillation for fast and accurate DNA sequence correction”⁠, Anastasiya Belyaeva, Joel Shor, Daniel E. Cook, Kishwar Shafin, Daniel Liu, Armin Töpfer, Aaron M. Wenger2022-11-17 (Transformer NN⁠, gene sequencing; similar)

“Legged Locomotion in Challenging Terrains Using Egocentric Vision”, Agarwal Et Al 2022

“Legged Locomotion in Challenging Terrains using Egocentric Vision”⁠, Ananye Agarwal, Ashish Kumar, Jitendra Malik, Deepak Pathak2022-11-14 (RNN⁠, model-free RL⁠, robotics; similar; bibliography)

“EVA: Exploring the Limits of Masked Visual Representation Learning at Scale”, Fang Et Al 2022

“EVA: Exploring the Limits of Masked Visual Representation Learning at Scale”⁠, Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, Yue Cao2022-11-14 (masked auto-encoder⁠, AI scaling; similar; bibliography)

“EDiff-I: Text-to-Image Diffusion Models With an Ensemble of Expert Denoisers”, Balaji Et Al 2022

“eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers”⁠, Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila2022-11-02 (diffusion model⁠, CLIP⁠, T5 Transformer⁠, MoE NN; similar; bibliography)

“Fast DistilBERT on CPUs”, Shen Et Al 2022

“Fast DistilBERT on CPUs”⁠, Haihao Shen, Ofir Zafrir, Bo Dong, Hengyu Meng, Xinyu Ye, Zhe Wang, Yi Ding, Hanwen Chang, Guy Boudoukh2022-10-27 (reduced-precision NNs⁠, Transformer NN; similar)

“Large Language Models Can Self-Improve”, Huang Et Al 2022

“Large Language Models Can Self-Improve”⁠, Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, Jiawei Han2022-10-20 (ML dataset⁠, inner monologue (AI)⁠, PaLM; similar; bibliography)

“Exclusive Supermask Subnetwork Training for Continual Learning”, Yadav & Bansal2022

“Exclusive Supermask Subnetwork Training for Continual Learning”⁠, Prateek Yadav, Mohit Bansal2022-10-18 (similar)

“The Unreasonable Effectiveness of Fully-Connected Layers for Low-Data Regimes”, Kocsis Et Al 2022

“The Unreasonable Effectiveness of Fully-Connected Layers for Low-Data Regimes”⁠, Peter Kocsis, Peter Súkeník, Guillem Brasó, Matthias Nießner, Laura Leal-Taixé, Ismail Elezi2022-10-11 (CNN⁠, MLP NN⁠, active learning; similar)

“On Distillation of Guided Diffusion Models”, Meng Et Al 2022

“On Distillation of Guided Diffusion Models”⁠, Chenlin Meng, Ruiqi Gao, Diederik P. Kingma, Stefano Ermon, Jonathan Ho, Tim Salimans2022-10-06 (diffusion model; similar; bibliography)

“Human-level Atari 200× Faster”, Kapturowski Et Al 2022

“Human-level Atari 200× faster”⁠, Steven Kapturowski, Víctor Campos, Ray Jiang, Nemanja Rakićević, Hado van Hasselt, Charles Blundell, Adrià Puigdomènech Badia2022-09-15 (model-free RL⁠, RL scaling; similar; bibliography)

“On the Effectiveness of Compact Biomedical Transformers (✱BioBERT)”, Rohanian Et Al 2022

“On the Effectiveness of Compact Biomedical Transformers (✱BioBERT)”⁠, Omid Rohanian, Mohammadmahdi Nouriborji, Samaneh Kouchaki, David A. Clifton2022-09-07 (Transformer NN; similar)

“Neural Payoff Machines: Predicting Fair and Stable Payoff Allocations Among Team Members”, Cornelisse Et Al 2022

“Neural Payoff Machines: Predicting Fair and Stable Payoff Allocations Among Team Members”⁠, Daphne Cornelisse, Thomas Rood, Mateusz Malinowski, Yoram Bachrach, Tal Kachman2022-08-18 (economics⁠, MARL; similar)

“Re2G: Retrieve, Rerank, Generate”, Glass Et Al 2022

“Re2G: Retrieve, Rerank, Generate”⁠, Michael Glass, Gaetano Rossiello, Md Faisal Mahbub Chowdhury, Ankita Rajaram Naik, Pengshan Cai, Alfio Gliozzo2022-07-13 (retrieval AI⁠, Transformer NN; similar; bibliography)

“Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems”, FitzGerald Et Al 2022

“Alexa Teacher Model: Pretraining and Distilling Multi-Billion-Parameter Encoders for Natural Language Understanding Systems”⁠, Jack FitzGerald, Shankar Ananthakrishnan, Konstantine Arkoudas, Davide Bernardi, Abhishek Bhagia, Claudio Delli Bovi2022-06-15 (T5 Transformer; similar; bibliography)

“SBERT Studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features”, Opitz & Frank2022

“SBERT studies Meaning Representations: Decomposing Sentence Embeddings into Explainable Semantic Features”⁠, Juri Opitz, Anette Frank2022-06-14 (Transformer NN; similar)

“ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers”, Yao Et Al 2022

“ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers”⁠, Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, Yuxiong He2022-06-04 (reduced-precision NNs⁠, GPT⁠, AI scaling/hardware; similar; bibliography)

“Dataset Condensation via Efficient Synthetic-Data Parameterization”, Kim Et Al 2022

“Dataset Condensation via Efficient Synthetic-Data Parameterization”⁠, Jang-Hyun Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung-Woo Ha, Hyun Oh Song2022-05-30 (ML dataset; similar)

“Dialog Inpainting: Turning Documents into Dialogues”, Dai Et Al 2022

“Dialog Inpainting: Turning Documents into Dialogues”⁠, Zhuyun Dai, Arun Tejasvi Chaganty, Vincent Zhao, Aida Amini, Qazi Mamunur Rashid, Mike Green, Kelvin Guu2022-05-18 (ML dataset⁠, inner monologue (AI)⁠, AI scaling; similar; bibliography)

“Solving ImageNet: a Unified Scheme for Training Any Backbone to Top Results”, Ridnik Et Al 2022

“Solving ImageNet: a Unified Scheme for Training any Backbone to Top Results”⁠, Tal Ridnik, Hussam Lawen, Emanuel Ben-Baruch, Asaf Noy2022-04-07 (similar; bibliography)

“Knowledge Distillation: Bad Models Can Be Good Role Models”, Kaplun Et Al 2022

“Knowledge Distillation: Bad Models Can Be Good Role Models”⁠, Gal Kaplun, Eran Malach, Preetum Nakkiran, Shai Shalev-Shwartz2022-03-28 (similar)

“STaR: Bootstrapping Reasoning With Reasoning”, Zelikman Et Al 2022

“STaR: Bootstrapping Reasoning With Reasoning”⁠, Eric Zelikman, Yuhuai Wu, Noah D. Goodman2022-03-28 (ML dataset⁠, inner monologue (AI); backlinks; similar)

“PPCD-GAN: Progressive Pruning and Class-Aware Distillation for Large-Scale Conditional GANs Compression”, Vo Et Al 2022

“PPCD-GAN: Progressive Pruning and Class-Aware Distillation for Large-Scale Conditional GANs Compression”⁠, Duc Minh Vo, Akihiro Sugimoto, Hideki Nakayama2022-03-16 (BigGAN⁠, NN pruning; similar)

“Self-Distilled StyleGAN: Towards Generation from Internet Photos”, Mokady Et Al 2022

“Self-Distilled StyleGAN: Towards Generation from Internet Photos”⁠, Ron Mokady, Michal Yarom, Omer Tov, Oran Lang, Daniel Cohen-Or, Tali Dekel, Michal Irani, Inbar Mosseri2022-02-24 (ML dataset⁠, StyleGAN⁠, AI scaling; similar; bibliography)

“AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models”, Xu Et Al 2022

“AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models”⁠, Dongkuan Xu, Subhabrata Mukherjee, Xiaodong Liu, Debadeepta Dey, Wenhui Wang, Xiang Zhang, Ahmed Hassan Awadallah2022-01-29 (Transformer NN; similar)

“DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale”, Rajbhandari Et Al 2022

“DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale”⁠, Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley2022-01-14 (MoE NN; similar; bibliography)

“Microdosing: Knowledge Distillation for GAN Based Compression”, Helminger Et Al 2022

“Microdosing: Knowledge Distillation for GAN based Compression”⁠, Leonhard Helminger, Roberto Azevedo, Abdelaziz Djelouah, Markus Gross, Christopher Schroers2022-01-07 (GAN⁠, video generation; similar)

“ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation”, Wang Et Al 2021

“ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation”⁠, Shuohuan Wang, Yu Sun, Yang Xiang, Zhihua Wu, Siyu Ding, Weibao Gong, Shikun Feng, Junyuan Shang, Yanbin Zhao2021-12-23 (AI scaling; similar)

“Amortized Noisy Channel Neural Machine Translation”, Pang Et Al 2021

“Amortized Noisy Channel Neural Machine Translation”⁠, Richard Yuanzhe Pang, He He, Kyunghyun Cho2021-12-16 (GPT⁠, model-free RL; similar)

“Causal Distillation for Language Models”, Wu Et Al 2021

“Causal Distillation for Language Models”⁠, Zhengxuan Wu, Atticus Geiger, Josh Rozner, Elisa Kreiss, Hanson Lu, Thomas Icard, Christopher Potts, Noah D. Goodman2021-12-05 (similar)

“Extrapolating from a Single Image to a Thousand Classes Using Distillation”, Asano & Saeed2021

“Extrapolating from a Single Image to a Thousand Classes using Distillation”⁠, Yuki M. Asano, Aaqib Saeed2021-12-01 (similar)

“Prune Once for All: Sparse Pre-Trained Language Models”, Zafrir Et Al 2021

“Prune Once for All: Sparse Pre-Trained Language Models”⁠, Ofir Zafrir, Ariel Larey, Guy Boudoukh, Haihao Shen, Moshe Wasserblat2021-11-10 (reduced-precision NNs⁠, NN pruning; similar; bibliography)

“Training Verifiers to Solve Math Word Problems”, Cobbe Et Al 2021

“Training Verifiers to Solve Math Word Problems”⁠, Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, John Schulman2021-10-27 (ML dataset⁠, inner monologue (AI)⁠, AI scaling⁠, math; similar)

“Wav2CLIP: Learning Robust Audio Representations From CLIP”, Wu Et Al 2021

“Wav2CLIP: Learning Robust Audio Representations From CLIP”⁠, Ho-Hsiang Wu, Prem Seetharaman, Kundan Kumar, Juan Pablo Bello2021-10-21 (CLIP; similar)

“When in Doubt, Summon the Titans: Efficient Inference With Large Models”, Rawat Et Al 2021

“When in Doubt, Summon the Titans: Efficient Inference with Large Models”⁠, Ankit Singh Rawat, Manzil Zaheer, Aditya Krishna Menon, Amr Ahmed, Sanjiv Kumar2021-10-19 (AI scaling; similar)

“Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora”, Jin Et Al 2021

“Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora”⁠, Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, Xiang Ren2021-10-16 (Transformer NN; similar)

“Symbolic Knowledge Distillation: from General Language Models to Commonsense Models”, West Et Al 2021

“Symbolic Knowledge Distillation: from General Language Models to Commonsense Models”⁠, Peter West, Chandra Bhagavatula, Jack Hessel, Jena D. Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck2021-10-14 (AI scaling; similar)

“Language Modelling via Learning to Rank”, Frydenlund Et Al 2021

“Language Modelling via Learning to Rank”⁠, Arvid Frydenlund, Gagandeep Singh, Frank Rudzicz2021-10-13 (similar; bibliography)

“Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapes”, Lee Et Al 2021

“Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapes”⁠, Alex X. Lee, Coline Devin, Yuxiang Zhou, Thomas Lampe, Konstantinos Bousmalis, Jost Tobias Springenberg2021-10-12 (robotics; similar)

“Unsupervised Neural Machine Translation With Generative Language Models Only”, Han Et Al 2021

“Unsupervised Neural Machine Translation with Generative Language Models Only”⁠, Jesse Michael Han, Igor Babuschkin, Harrison Edwards, Arvind Neelakantan, Tao Xu, Stanislas Polu, Alex Ray2021-10-11 (inner monologue (AI)⁠, AI scaling; similar)

“Progressive Distillation for Fast Sampling of Diffusion Models”, Salimans & Ho2021

“Progressive Distillation for Fast Sampling of Diffusion Models”⁠, Tim Salimans, Jonathan Ho2021-10-05 (diffusion model; similar)

“OTTER: Data Efficient Language-Supervised Zero-Shot Recognition With Optimal Transport Distillation”, Wu Et Al 2021

“OTTER: Data Efficient Language-Supervised Zero-Shot Recognition with Optimal Transport Distillation”⁠, Bichen Wu, Ruizhe Cheng, Peizhao Zhang, Peter Vajda, Joseph E. Gonzalez2021-10-05 (CLIP; similar; bibliography)

“On the Interplay Between Sparsity, Naturalness, Intelligibility, and Prosody in Speech Synthesis”, Lai Et Al 2021

“On the Interplay Between Sparsity, Naturalness, Intelligibility, and Prosody in Speech Synthesis”⁠, Cheng-I Jeff Lai, Erica Cooper, Yang Zhang, Shiyu Chang, Kaizhi Qian, Yi-Lun Liao, Yung-Sung Chuang, Alexander H. Liu2021-10-04 (NN pruning; similar)

“Beyond Distillation: Task-level Mixture-of-Experts (TaskMoE) for Efficient Inference”, Kudugunta Et Al 2021

“Beyond Distillation: Task-level Mixture-of-Experts (TaskMoE) for Efficient Inference”⁠, Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh-Thang Luong, Orhan Firat2021-09-24 (MoE NN; similar)

“ZSD-YOLO: Zero-Shot YOLO Detection Using Vision-Language Knowledge Distillation”, Xie & Zheng2021

“ZSD-YOLO: Zero-Shot YOLO Detection using Vision-Language Knowledge Distillation”⁠, Johnathan Xie, Shuai Zheng2021-09-24 (CLIP; similar; bibliography)

“SPLADE V2: Sparse Lexical and Expansion Model for Information Retrieval”, Formal Et Al 2021

“SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval”⁠, Thibault Formal, Carlos Lassance, Benjamin Piwowarski, Stéphane Clinchant2021-09-21 (retrieval AI; similar)

“KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation”, Tahaei Et Al 2021

“KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation”⁠, Marzieh S. Tahaei, Ella Charlaix, Vahid Partovi Nia, Ali Ghodsi, Mehdi Rezagholizadeh2021-09-13 (Transformer NN; similar; bibliography)

“Multi-Task Self-Training for Learning General Representations”, Ghiasi Et Al 2021

“Multi-Task Self-Training for Learning General Representations”⁠, Golnaz Ghiasi, Barret Zoph, Ekin D. Cubuk, Quoc V. Le, Tsung-Yi Lin2021-08-25 (active learning⁠, RL scaling; similar)

“Dataset Distillation With Infinitely Wide Convolutional Networks”, Nguyen Et Al 2021

“Dataset Distillation with Infinitely Wide Convolutional Networks”⁠, Timothy Nguyen, Roman Novak, Lechao Xiao, Jaehoon Lee2021-07-27 (CNN⁠, active learning⁠, meta-learning; similar)

“Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better”, Menghani2021

“Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better”⁠, Gaurav Menghani2021-06-16 (reduced-precision NNs; similar)

“Knowledge Distillation: A Good Teacher Is Patient and Consistent”, Beyer Et Al 2021

“Knowledge distillation: A good teacher is patient and consistent”⁠, Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, Alexander Kolesnikov2021-06-09 (AI scaling; similar; bibliography)

“ResMLP: Feedforward Networks for Image Classification With Data-efficient Training”, Touvron Et Al 2021

“ResMLP: Feedforward networks for image classification with data-efficient training”⁠, Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard2021-05-07 (MLP NN; similar)

“DINO: Emerging Properties in Self-Supervised Vision Transformers”, Caron Et Al 2021

“DINO: Emerging Properties in Self-Supervised Vision Transformers”⁠, Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Arm, Joulin2021-04-29 (AI scaling; similar)

“Zero-Shot Detection via Vision and Language Knowledge Distillation”, Gu Et Al 2021

“Zero-Shot Detection via Vision and Language Knowledge Distillation”⁠, Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, Yin Cui2021-04-28 (CLIP; similar; bibliography)

“Data-Efficient Language-Supervised Zero-Shot Learning With Self-Distillation”, Cheng Et Al 2021

“Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillation”⁠, Ruizhe Cheng, Bichen Wu, Peizhao Zhang, Peter Vajda, Joseph E. Gonzalez2021-04-18 (CLIP; similar; bibliography)

“Efficient Transformers in Reinforcement Learning Using Actor-Learner Distillation”, Parisotto & Salakhutdinov2021

“Efficient Transformers in Reinforcement Learning using Actor-Learner Distillation”⁠, Emilio Parisotto, Ruslan Salakhutdinov2021-04-04 (RNN⁠, self-attention⁠, model-free RL; backlinks; similar)

“KiloNeRF: Speeding up Neural Radiance Fields With Thousands of Tiny MLPs”, Reiser Et Al 2021

“KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs”⁠, Christian Reiser, Songyou Peng, Yiyi Liao, Andreas Geiger2021-03-25 (MLP NN; backlinks; similar)

“China’s GPT-3? BAAI Introduces Superscale Intelligence Model ‘Wu Dao 1.0’: The Beijing Academy of Artificial Intelligence (BAAI) Releases Wu Dao 1.0, China’s First Large-scale Pretraining Model.”, Synced2021

“China’s GPT-3? BAAI Introduces Superscale Intelligence Model ‘Wu Dao 1.0’: The Beijing Academy of Artificial Intelligence (BAAI) releases Wu Dao 1.0, China’s first large-scale pretraining model.”⁠, Synced2021-03-23 (retrieval AI⁠, DALL·E 1⁠, MoE NN⁠, video generation; similar; bibliography)

“Distilling Large Language Models into Tiny and Effective Students Using PQRNN”, Kaliamoorthi Et Al 2021

“Distilling Large Language Models into Tiny and Effective Students using pQRNN”⁠, Prabhu Kaliamoorthi, Aditya Siddhant, Edward Li, Melvin Johnson2021-01-21 (RNN; similar)

“Training Data-efficient Image Transformers & Distillation through Attention”, Touvron Et Al 2020

“Training data-efficient image transformers & distillation through attention”⁠, Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, Hervé Jégou2020-12-23 (Transformer NN; similar; bibliography)

“Towards Playing Full MOBA Games With Deep Reinforcement Learning”, Ye Et Al 2020

“Towards Playing Full MOBA Games with Deep Reinforcement Learning”⁠, Deheng Ye, Guibin Chen, Wen Zhang, Sheng Chen, Bo Yuan, Bo Liu, Jia Chen, Zhao Liu, Fuhao Qiu, Hongsheng Yu2020-11-25 (RNN⁠, OA5⁠, MARL; similar; bibliography)

“A Primer in BERTology: What We Know about How BERT Works”, Rogers Et Al 2020

“A Primer in BERTology: What we know about how BERT works”⁠, Anna Rogers, Olga Kovaleva, Anna Rumshisky2020-11-09 (reduced-precision NNs⁠, NN pruning⁠, Transformer NN; similar)

“Dataset Meta-Learning from Kernel Ridge-Regression”, Nguyen Et Al 2020

“Dataset Meta-Learning from Kernel Ridge-Regression”⁠, Timothy Nguyen, Zhourong Chen, Jaehoon Lee2020-10-30 (active learning⁠, meta-learning; similar)

“TernaryBERT: Distillation-aware Ultra-low Bit BERT”, Zhang Et Al 2020

“TernaryBERT: Distillation-aware Ultra-low Bit BERT”⁠, Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, Qun Liu2020-09-27 (reduced-precision NNs⁠, Transformer NN; similar)

“SimCLRv2: Big Self-Supervised Models Are Strong Semi-Supervised Learners”, Chen Et Al 2020

“SimCLRv2: Big Self-Supervised Models are Strong Semi-Supervised Learners”⁠, Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, Geoffrey Hinton2020-06-17 (CNN⁠, AI scaling; similar)

“Movement Pruning: Adaptive Sparsity by Fine-Tuning”, Sanh Et Al 2020

“Movement Pruning: Adaptive Sparsity by Fine-Tuning”⁠, Victor Sanh, Thomas Wolf, Alexander M. Rush2020-05-15 (NN pruning; similar)

“General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference”, Du Et Al 2020

“General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference”⁠, Jingfei Du, Myle Ott, Haoran Li, Xing Zhou, Veselin Stoyanov2020-04-29 (reduced-precision NNs⁠, Transformer NN; similar)

“MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers”, Wang Et Al 2020

“MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers”⁠, Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, Ming Zhou2020-02-25 (Transformer NN; similar; bibliography)

“Towards a Conversational Agent That Can Chat About…Anything”, Adiwardana & Luong2020

“Towards a Conversational Agent that Can Chat About…Anything”⁠, Daniel Adiwardana, Thang Luong2020-01-28 (Transformer NN⁠, AI scaling; similar; bibliography)

“Understanding the Generalization Of ‘Lottery Tickets’ In Neural Networks”, Morcos & Tian2019

“Understanding the generalization of ‘lottery tickets’ in neural networks”⁠, Ari Morcos, Yuandong Tian2019-11-25 (MLP NN⁠, AI scaling; backlinks; similar)

“Self-training With Noisy Student Improves ImageNet Classification”, Xie Et Al 2019

“Self-training with Noisy Student improves ImageNet classification”⁠, Qizhe Xie, Minh-Thang Luong, Eduard Hovy, Quoc V. Le2019-11-11 (AI scaling; similar; bibliography)

“On Warm-Starting Neural Network Training”, Ash & Adams2019

“On Warm-Starting Neural Network Training”⁠, Jordan T. Ash, Ryan P. Adams2019-10-18 (active learning; similar)

“DistilBERT, a Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter”, Sanh Et Al 2019

“DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter”⁠, Victor Sanh, Lysandre Debut, Julien Chaumond, Thomas Wolf2019-10-02 (Transformer NN; backlinks; similar)

“TinyBERT: Distilling BERT for Natural Language Understanding”, Jiao Et Al 2019

“TinyBERT: Distilling BERT for Natural Language Understanding”⁠, Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, Qun Liu2019-09-23 (Transformer NN; backlinks; similar; bibliography)

“Smaller, Faster, Cheaper, Lighter: Introducing DistilGPT, a Distilled Version of GPT”, Sanh2019

“Smaller, faster, cheaper, lighter: Introducing DistilGPT, a distilled version of GPT”⁠, Victor Sanh2019-08-28 (GPT; similar)

“ICML 2019 Notes”, Abel2019

“ICML 2019 Notes”⁠, David Abel2019-06 (AI scaling⁠, RL exploration⁠, meta-learning⁠, MARL; similar; bibliography)

“Mask-Predict: Parallel Decoding of Conditional Masked Language Models”, Ghazvininejad Et Al 2019

“Mask-Predict: Parallel Decoding of Conditional Masked Language Models”⁠, Marjan Ghazvininejad, Omer Levy, Yinhan Liu, Luke Zettlemoyer2019-04-19 (Transformer NN; similar)

“Distilling Policy Distillation”, Czarnecki Et Al 2019

“Distilling Policy Distillation”⁠, Wojciech Marian Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant M. Jayakumar, Grzegorz Swirszcz, Max Jaderberg2019-02-06 (MARL; similar; bibliography)

“Compressing GANs Using Knowledge Distillation”, Aguinaldo Et Al 2019

“Compressing GANs using Knowledge Distillation”⁠, Angeline Aguinaldo, Ping-Yeh Chiang, Alex Gain, Ameya Patil, Kolten Pearson, Soheil Feizi2019-02-01 (GAN; similar)

“Neural Probabilistic Motor Primitives for Humanoid Control”, Merel Et Al 2018

“Neural probabilistic motor primitives for humanoid control”⁠, Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, Nicolas Heess2018-11-28 (autoencoder NN⁠, model-free RL⁠, robotics; similar)

“Dataset Distillation”, Wang Et Al 2018

“Dataset Distillation”⁠, Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, Alexei A. Efros2018-11-27 (active learning; backlinks; similar)

“Exploration by Random Network Distillation”, Burda Et Al 2018

“Exploration by Random Network Distillation”⁠, Yuri Burda, Harrison Edwards, Amos Storkey, Oleg Klimov2018-10-30 (RL exploration; similar)

“OCD: Optimal Completion Distillation for Sequence Learning”, Sabour Et Al 2018

“OCD: Optimal Completion Distillation for Sequence Learning”⁠, Sara Sabour, William Chan, Mohammad Norouzi2018-10-02 (backlinks; similar)

“Network Recasting: A Universal Method for Network Architecture Transformation”, Yu Et Al 2018

“Network Recasting: A Universal Method for Network Architecture Transformation”⁠, Joonsang Yu, Sungbum Kang, Kiyoung Choi2018-09-14 (similar)

“ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech”, Ping Et Al 2018

“ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech”⁠, Wei Ping, Kainan Peng, Jitong Chen2018-07-19 (similar)

“Self-Net: Lifelong Learning via Continual Self-Modeling”, Camp Et Al 2018

“Self-Net: Lifelong Learning via Continual Self-Modeling”⁠, Blake Camp, Jaya Krishna Mandivarapu, Rolando Estrada2018-05-25 (autoencoder NN; similar)

“Kickstarting Deep Reinforcement Learning”, Schmitt Et Al 2018

“Kickstarting Deep Reinforcement Learning”⁠, Simon Schmitt, Jonathan J. Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Wojciech M. Czarnecki2018-03-10 (meta-learning⁠, MARL; similar)

“Faster Gaze Prediction With Dense Networks and Fisher Pruning”, Theis Et Al 2018

“Faster gaze prediction with dense networks and Fisher pruning”⁠, Lucas Theis, Iryna Korshunova, Alykhan Tejani, Ferenc Huszár2018-01-17 (NN pruning; similar)

“Parallel WaveNet: Fast High-Fidelity Speech Synthesis”, Oord Et Al 2017

“Parallel WaveNet: Fast High-Fidelity Speech Synthesis”⁠, Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche2017-11-28 (similar)

“Knowledge Concentration: Learning 100K Object Classifiers in a Single CNN”, Gao Et Al 2017

“Knowledge Concentration: Learning 100K Object Classifiers in a Single CNN”⁠, Jiyang Gao, Zijian, Guo, Zhen Li, Ram Nevatia2017-11-21 (CNN⁠, AI scaling; similar)

“Policy Optimization by Genetic Distillation”, Gangwani & Peng2017

“Policy Optimization by Genetic Distillation”⁠, Tanmay Gangwani, Jian Peng2017-11-03 (RL exploration; backlinks; similar)

“N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning”, Ashok Et Al 2017

“N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning”⁠, Anubhav Ashok, Nicholas Rhinehart, Fares Beainy, Kris M. Kitani2017-09-18 (RNN; backlinks; similar)

“Training Shallow and Thin Networks for Acceleration via Knowledge Distillation With Conditional Adversarial Networks”, Xu Et Al 2017

“Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks”⁠, Zheng Xu, Yen-Chang Hsu, Jiawei Huang2017-09-02 (GAN; similar)

“Distral: Robust Multitask Reinforcement Learning”, Teh Et Al 2017

“Distral: Robust Multitask Reinforcement Learning”⁠, Yee Whye Teh, Victor Bapst, Wojciech Marian Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess2017-07-13 (RL exploration; similar)

“Biased Importance Sampling for Deep Neural Network Training”, Katharopoulos & Fleuret2017

“Biased Importance Sampling for Deep Neural Network Training”⁠, Angelos Katharopoulos, François Fleuret2017-05-31 (RNN; backlinks; similar)

“Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer”, Zagoruyko & Komodakis2016

“Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer”⁠, Sergey Zagoruyko, Nikos Komodakis2016-12-12 (Transformer NN; similar)

“Do Deep Convolutional Nets Really Need to Be Deep and Convolutional?”, Urban Et Al 2016

“Do Deep Convolutional Nets Really Need to be Deep and Convolutional?”⁠, Gregor Urban, Krzysztof J. Geras, Samira Ebrahimi Kahou, Ozlem Aslan, Shengjie Wang, Rich Caruana, Abdelrahman Mohamed2016-03-17 (CNN⁠, MLP NN⁠, AI scaling; backlinks; similar)

“Face Model Compression by Distilling Knowledge from Neurons”, Luo Et Al 2016

“Face Model Compression by Distilling Knowledge from Neurons”⁠, Ping Luo, Zhenyao Zhu, Ziwei Liu, Xiaogang Wang, Xiaoou Tang2016-03-05 (similar; bibliography)

“Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning”, Parisotto Et Al 2015

“Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning”⁠, Emilio Parisotto, Jimmy Lei Ba, Ruslan Salakhutdinov2015-11-19 (model-free RL; similar)

“Policy Distillation”, Rusu Et Al 2015

“Policy Distillation”⁠, Andrei A. Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu2015-11-19 (MARL; similar)

“Net2Net: Accelerating Learning via Knowledge Transfer”, Chen Et Al 2015

“Net2Net: Accelerating Learning via Knowledge Transfer”⁠, Tianqi Chen, Ian Goodfellow, Jonathon Shlens2015-11-18 (AI scaling⁠, OA5; backlinks; similar)

“Bayesian Dark Knowledge”, Korattikara Et Al 2015

“Bayesian Dark Knowledge”⁠, Anoop Korattikara, Vivek Rathod, Kevin Murphy, Max Welling2015-06-14 (Bayes; similar)

“Distilling the Knowledge in a Neural Network”, Hinton Et Al 2015

“Distilling the Knowledge in a Neural Network”⁠, Geoffrey Hinton, Oriol Vinyals, Jeff Dean2015-03-09 (MoE NN; similar)

“FitNets: Hints for Thin Deep Nets”, Romero Et Al 2014

“FitNets: Hints for Thin Deep Nets”⁠, Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, Yoshua Bengio2014-12-19 (similar)

“Do Deep Nets Really Need to Be Deep?”, Ba & Caruana2013

“Do Deep Nets Really Need to be Deep?”⁠, Lei Jimmy Ba, Rich Caruana2013-12-21 (MLP NN; backlinks; similar)

“Model Compression”, Bucila & Al 2006

“Learning Complex, Extended Sequences Using the Principle of History Compression”, Schmidhuber1992

“Learning Complex, Extended Sequences Using the Principle of History Compression”⁠, Juergen Schmidhuber1992 (RNN; similar)

“From Vision to Language: Semi-Supervised Learning in Action…at Scale”

Wikipedia

1. Knowledge distillation