- See Also
-
Links
- “CATS: Contextually-Aware Thresholding for Sparsity in Large Language Models”, Lee et al 2024
- “Exploring Concept Depth: How Large Language Models Acquire Knowledge at Different Layers?”, Jin et al 2024
- “ReFT: Representation Finetuning for Language Models”, Wu et al 2024
- “LTE: Training Neural Networks from Scratch With Parallel Low-Rank Adapters”, Huh et al 2024
- “Exponentially Faster Language Modelling”, Belcak & Wattenhofer 2023
- “DiLoCo: Distributed Low-Communication Training of Language Models”, Douillard et al 2023
- “Language Models Are Super Mario (DARE): Absorbing Abilities from Homologous Models As a Free Lunch”, Yu et al 2023
- “ProSG: Using Prompt Synthetic Gradients to Alleviate Prompt Forgetting of RNN-Like Language Models”, Luo et al 2023
- “The Impact of Depth and Width on Transformer Language Model Generalization”, Petty et al 2023
- “Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time”, Liu et al 2023
- “Fast Feedforward Networks”, Belcak & Wattenhofer 2023
- “Any Deep ReLU Network Is Shallow”, Villani & Schoots 2023
- “JaxPruner: A Concise Library for Sparsity Research”, Lee et al 2023
- “Reusing Deep Neural Network Models through Model Re-Engineering”, Qi et al 2023
- “MUX-PLMs: Pre-Training Language Models With Data Multiplexing”, Murahari et al 2023
- “DataMUX: Data Multiplexing for Neural Networks”, Murahari et al 2023
- “Deep Differentiable Logic Gate Networks”, Petersen et al 2022
- “The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers”, Li et al 2022
- “Noise Transforms Feed-Forward Networks into Sparse Coding Networks”, Anonymous 2022
- “Exploring Low Rank Training of Deep Neural Networks”, Kamalakara et al 2022
- “Monolith: Real Time Recommendation System With Collisionless Embedding Table”, Liu et al 2022
- “More ConvNets in the 2020s: Scaling up Kernels Beyond 51×51 Using Sparsity (SLaK)”, Liu et al 2022
- “Building Machine Translation Systems for the Next Thousand Languages”, Bapna et al 2022
- “Monarch: Expressive Structured Matrices for Efficient and Accurate Training”, Dao et al 2022
- “Efficient Language Modeling With Sparse All-MLP”, Yu et al 2022
- “NeuPL: Neural Population Learning”, Liu et al 2022
- “Datamodels: Predicting Predictions from Training Data”, Ilyas et al 2022
- “Spiking Neural Networks and Their Applications: A Review”, Yamazaki et al 2022
- “Persia: An Open, Hybrid System Scaling Deep Learning-Based Recommenders up to 100 Trillion Parameters”, Lian et al 2021
- “EvilModel: Hiding Malware Inside of Neural Network Models”, Wang et al 2021
- “LoRA: Low-Rank Adaptation of Large Language Models”, Hu et al 2021
- “On the Distribution, Sparsity, and Inference-Time Quantization of Attention Values in Transformers”, Ji et al 2021
- “The Neural Basis of Intelligence in Fine-Grained Cortical Topographies”, Feilong et al 2021
- “Clusterability in Neural Networks”, Filan et al 2021
- “Sparsity in Deep Learning: Pruning and Growth for Efficient Inference and Training in Neural Networks”, Hoefler et al 2021
- “Scaling down Deep Learning”, Greydanus 2020
- “Extreme Model Compression for On-Device Natural Language Understanding”, Sathyendra et al 2020
- “EventProp: Event-Based Backpropagation Can Compute Exact Gradients for Spiking Neural Networks”, Wunderlich & Pehle 2020
- “On Linear Identifiability of Learned Representations”, Roeder et al 2020
- “Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited”, Maddox et al 2020
- “Bayesian Deep Learning and a Probabilistic Perspective of Generalization”, Wilson & Izmailov 2020
- “Neural Arithmetic Units”, Madsen & Johansen 2020
- “Linear Mode Connectivity and the Lottery Ticket Hypothesis”, Frankle et al 2019
- “Learning to Seek: Autonomous Source Seeking With Deep Reinforcement Learning Onboard a Nano Drone Microcontroller”, Duisterhof et al 2019
- “Does Learning Require Memorization? A Short Tale about a Long Tail”, Feldman 2019
- “Weight Agnostic Neural Networks”, Gaier & Ha 2019
- “StyleNAS: An Empirical Study of Neural Architecture Search to Uncover Surprisingly Fast End-To-End Universal Style Transfer Networks”, An et al 2019
- “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks”, Tan & Le 2019
- “Superposition of Many Models into One”, Cheung et al 2019
- “Playing Atari With Six Neurons”, Cuccu et al 2018
- “Measuring the Intrinsic Dimension of Objective Landscapes”, Li et al 2018
- “SqueezeNext: Hardware-Aware Neural Network Design”, Gholami et al 2018
- “Wide Compression: Tensor Ring Nets”, Wang et al 2018
- “Intriguing Properties of Randomly Weighted Networks: Generalizing While Learning Next to Nothing”, Rosenfeld & Tsotsos 2018
- “Fix Your Classifier: the Marginal Value of Training the Last Weight Layer”, Hoffer et al 2018
- “Learning Compact Recurrent Neural Networks With Block-Term Tensor Decomposition”, Ye et al 2017
- “3D Semantic Segmentation With Submanifold Sparse Convolutional Networks”, Graham et al 2017
- “XUnit: Learning a Spatial Activation Function for Efficient Image Restoration”, Kligvasser et al 2017
- “Natural Language Processing With Small Feed-Forward Networks”, Botha et al 2017
- “ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices”, Zhang et al 2017
- “Submanifold Sparse Convolutional Networks”, Graham & Maaten 2017
- “Shake-Shake Regularization of 3-Branch Residual Networks”, Gastaldi 2017
- “Using the Output Embedding to Improve Language Models”, Press & Wolf 2016
- “Deep Residual Learning for Image Recognition”, He et al 2015
- “Tensorizing Neural Networks”, Novikov et al 2015
- “Eight Pairs of Descending Visual Neurons in the Dragonfly Give Wing Motor Centers Accurate Population Vector of Prey Direction”, Gonzalez-Bellido et al 2013
- “The Cat Is out of the Bag: Cortical Simulations With 109 Neurons, 1013 Synapses”, Ananthanarayanan et al 2009
- “On the Computational Power of Threshold Circuits With Sparse Activity”, Uchizawa et al 2006
- “Networks of Spiking Neurons: The Third Generation of Neural Network Models”, Maass 1997
- “Characteristics of Sparsely Encoded Associative Memory”, Amari 1989
- “[2110.08152] Kronecker Decomposition for GPT Compression”
- “Higher Accuracy on Vision Models With EfficientNet-Lite”
- “Delivering Real-Time AI in the Palm of Your Hand”
- “Sparsity-Aware Deep Learning Inference Runtime for CPUs”
- “Neuralmagic/sparseml: Libraries for Applying Sparsification Recipes to Neural Networks With a Few Lines of Code, Enabling Faster and Smaller Models”
- “Creating a 17 KB Style Transfer Model With Layer Pruning and Quantization”, Toole 2024
- “BERT-Large: Prune Once for DistilBERT Inference Performance”
- Sort By Magic
- Wikipedia
- Miscellaneous
- Link Bibliography
See Also
Links
“CATS: Contextually-Aware Thresholding for Sparsity in Large Language Models”, Lee et al 2024
CATS: Contextually-Aware Thresholding for Sparsity in Large Language Models
“Exploring Concept Depth: How Large Language Models Acquire Knowledge at Different Layers?”, Jin et al 2024
Exploring Concept Depth: How Large Language Models Acquire Knowledge at Different Layers?
“ReFT: Representation Finetuning for Language Models”, Wu et al 2024
“LTE: Training Neural Networks from Scratch With Parallel Low-Rank Adapters”, Huh et al 2024
LTE: Training Neural Networks from Scratch with Parallel Low-Rank Adapters
“Exponentially Faster Language Modelling”, Belcak & Wattenhofer 2023
“DiLoCo: Distributed Low-Communication Training of Language Models”, Douillard et al 2023
DiLoCo: Distributed Low-Communication Training of Language Models
“Language Models Are Super Mario (DARE): Absorbing Abilities from Homologous Models As a Free Lunch”, Yu et al 2023
Language Models are Super Mario (DARE): Absorbing Abilities from Homologous Models as a Free Lunch
“ProSG: Using Prompt Synthetic Gradients to Alleviate Prompt Forgetting of RNN-Like Language Models”, Luo et al 2023
ProSG: Using Prompt Synthetic Gradients to Alleviate Prompt Forgetting of RNN-like Language Models
“The Impact of Depth and Width on Transformer Language Model Generalization”, Petty et al 2023
The Impact of Depth and Width on Transformer Language Model Generalization
“Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time”, Liu et al 2023
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time
“Fast Feedforward Networks”, Belcak & Wattenhofer 2023
“Any Deep ReLU Network Is Shallow”, Villani & Schoots 2023
“JaxPruner: A Concise Library for Sparsity Research”, Lee et al 2023
“Reusing Deep Neural Network Models through Model Re-Engineering”, Qi et al 2023
Reusing Deep Neural Network Models through Model Re-engineering
“MUX-PLMs: Pre-Training Language Models With Data Multiplexing”, Murahari et al 2023
MUX-PLMs: Pre-training Language Models with Data Multiplexing
“DataMUX: Data Multiplexing for Neural Networks”, Murahari et al 2023
“Deep Differentiable Logic Gate Networks”, Petersen et al 2022
“The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers”, Li et al 2022
The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers
“Noise Transforms Feed-Forward Networks into Sparse Coding Networks”, Anonymous 2022
Noise Transforms Feed-Forward Networks into Sparse Coding Networks
“Exploring Low Rank Training of Deep Neural Networks”, Kamalakara et al 2022
“Monolith: Real Time Recommendation System With Collisionless Embedding Table”, Liu et al 2022
Monolith: Real Time Recommendation System With Collisionless Embedding Table
“More ConvNets in the 2020s: Scaling up Kernels Beyond 51×51 Using Sparsity (SLaK)”, Liu et al 2022
More ConvNets in the 2020s: Scaling up Kernels Beyond 51×51 using Sparsity (SLaK)
“Building Machine Translation Systems for the Next Thousand Languages”, Bapna et al 2022
Building Machine Translation Systems for the Next Thousand Languages
“Monarch: Expressive Structured Matrices for Efficient and Accurate Training”, Dao et al 2022
Monarch: Expressive Structured Matrices for Efficient and Accurate Training
“Efficient Language Modeling With Sparse All-MLP”, Yu et al 2022
“NeuPL: Neural Population Learning”, Liu et al 2022
“Datamodels: Predicting Predictions from Training Data”, Ilyas et al 2022
“Spiking Neural Networks and Their Applications: A Review”, Yamazaki et al 2022
“Persia: An Open, Hybrid System Scaling Deep Learning-Based Recommenders up to 100 Trillion Parameters”, Lian et al 2021
“EvilModel: Hiding Malware Inside of Neural Network Models”, Wang et al 2021
“LoRA: Low-Rank Adaptation of Large Language Models”, Hu et al 2021
“On the Distribution, Sparsity, and Inference-Time Quantization of Attention Values in Transformers”, Ji et al 2021
On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers
“The Neural Basis of Intelligence in Fine-Grained Cortical Topographies”, Feilong et al 2021
The neural basis of intelligence in fine-grained cortical topographies
“Clusterability in Neural Networks”, Filan et al 2021
“Sparsity in Deep Learning: Pruning and Growth for Efficient Inference and Training in Neural Networks”, Hoefler et al 2021
“Scaling down Deep Learning”, Greydanus 2020
“Extreme Model Compression for On-Device Natural Language Understanding”, Sathyendra et al 2020
Extreme Model Compression for On-device Natural Language Understanding
“EventProp: Event-Based Backpropagation Can Compute Exact Gradients for Spiking Neural Networks”, Wunderlich & Pehle 2020
EventProp: Event-Based Backpropagation can compute Exact Gradients for Spiking Neural Networks
“On Linear Identifiability of Learned Representations”, Roeder et al 2020
“Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited”, Maddox et al 2020
Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited
“Bayesian Deep Learning and a Probabilistic Perspective of Generalization”, Wilson & Izmailov 2020
Bayesian Deep Learning and a Probabilistic Perspective of Generalization
“Neural Arithmetic Units”, Madsen & Johansen 2020
“Linear Mode Connectivity and the Lottery Ticket Hypothesis”, Frankle et al 2019
“Learning to Seek: Autonomous Source Seeking With Deep Reinforcement Learning Onboard a Nano Drone Microcontroller”, Duisterhof et al 2019
“Does Learning Require Memorization? A Short Tale about a Long Tail”, Feldman 2019
Does Learning Require Memorization? A Short Tale about a Long Tail
“Weight Agnostic Neural Networks”, Gaier & Ha 2019
“StyleNAS: An Empirical Study of Neural Architecture Search to Uncover Surprisingly Fast End-To-End Universal Style Transfer Networks”, An et al 2019
“EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks”, Tan & Le 2019
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
“Superposition of Many Models into One”, Cheung et al 2019
“Playing Atari With Six Neurons”, Cuccu et al 2018
“Measuring the Intrinsic Dimension of Objective Landscapes”, Li et al 2018
“SqueezeNext: Hardware-Aware Neural Network Design”, Gholami et al 2018
“Wide Compression: Tensor Ring Nets”, Wang et al 2018
“Intriguing Properties of Randomly Weighted Networks: Generalizing While Learning Next to Nothing”, Rosenfeld & Tsotsos 2018
Intriguing Properties of Randomly Weighted Networks: Generalizing while Learning Next to Nothing
“Fix Your Classifier: the Marginal Value of Training the Last Weight Layer”, Hoffer et al 2018
Fix your classifier: the marginal value of training the last weight layer
“Learning Compact Recurrent Neural Networks With Block-Term Tensor Decomposition”, Ye et al 2017
Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition
“3D Semantic Segmentation With Submanifold Sparse Convolutional Networks”, Graham et al 2017
3D Semantic Segmentation with Submanifold Sparse Convolutional Networks
“XUnit: Learning a Spatial Activation Function for Efficient Image Restoration”, Kligvasser et al 2017
xUnit: Learning a Spatial Activation Function for Efficient Image Restoration
“Natural Language Processing With Small Feed-Forward Networks”, Botha et al 2017
Natural Language Processing with Small Feed-Forward Networks
“ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices”, Zhang et al 2017
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
“Submanifold Sparse Convolutional Networks”, Graham & Maaten 2017
“Shake-Shake Regularization of 3-Branch Residual Networks”, Gastaldi 2017
“Using the Output Embedding to Improve Language Models”, Press & Wolf 2016
“Deep Residual Learning for Image Recognition”, He et al 2015
“Tensorizing Neural Networks”, Novikov et al 2015
“Eight Pairs of Descending Visual Neurons in the Dragonfly Give Wing Motor Centers Accurate Population Vector of Prey Direction”, Gonzalez-Bellido et al 2013
“The Cat Is out of the Bag: Cortical Simulations With 109 Neurons, 1013 Synapses”, Ananthanarayanan et al 2009
The cat is out of the bag: cortical simulations with 109 neurons, 1013 synapses
“On the Computational Power of Threshold Circuits With Sparse Activity”, Uchizawa et al 2006
On the Computational Power of Threshold Circuits with Sparse Activity
“Networks of Spiking Neurons: The Third Generation of Neural Network Models”, Maass 1997
Networks of spiking neurons: The third generation of neural network models
“Characteristics of Sparsely Encoded Associative Memory”, Amari 1989
“[2110.08152] Kronecker Decomposition for GPT Compression”
“Higher Accuracy on Vision Models With EfficientNet-Lite”
“Delivering Real-Time AI in the Palm of Your Hand”
“Sparsity-Aware Deep Learning Inference Runtime for CPUs”
“Neuralmagic/sparseml: Libraries for Applying Sparsification Recipes to Neural Networks With a Few Lines of Code, Enabling Faster and Smaller Models”
“Creating a 17 KB Style Transfer Model With Layer Pruning and Quantization”, Toole 2024
Creating a 17 KB style transfer model with layer pruning and quantization
“BERT-Large: Prune Once for DistilBERT Inference Performance”
BERT-Large: Prune Once for DistilBERT Inference Performance:
Sort By Magic
Annotations sorted by machine learning into inferred 'tags'. This provides an alternative way to browse: instead of by date order, one can browse in topic order. The 'sorted' list has been automatically clustered into multiple sections & auto-labeled for easier browsing.
Beginning with the newest annotation, it uses the embedding of each annotation to attempt to create a list of nearest-neighbor annotations, creating a progression of topics. For more details, see the link.
language-efficiency
model-compression
spiking-neurons
Wikipedia
Miscellaneous
-
/doc/ai/nn/sparsity/2018-cheng.pdf:View PDF:
-
/doc/ai/nn/cnn/2017-rawat.pdf:View PDF:
-
https://ai.facebook.com/blog/a-highly-efficient-real-time-text-to-speech-system-deployed-on-cpus/ -
https://blog.research.google/2018/05/custom-on-device-ml-models.html: -
https://blog.research.google/2019/03/an-all-neural-on-device-speech.html: -
https://blog.research.google/2021/10/grammar-correction-as-you-type-on-pixel.html: -
https://blog.research.google/2021/12/training-machine-learning-models-more.html: -
https://blog.research.google/2022/03/auto-generated-summaries-in-google-docs.html: -
https://blog.research.google/2022/08/efficient-sequence-modeling-for-on.html: -
https://blog.roblox.com/2020/05/scaled-bert-serve-1-billion-daily-requests-cpus/: -
https://cprimozic.net/blog/growing-sparse-computational-graphs-with-rnns/: -
https://tech.pic-collage.com/distillation-of-clip-model-and-other-experiments-f8394b7321ce: -
https://www.quantamagazine.org/sparse-neural-networks-point-physicists-to-useful-data-20230608/: -
https://www.reddit.com/r/LocalLLaMA/comments/18luk10/wait_llama_and_falcon_are_also_moe/
Link Bibliography
-
https://arxiv.org/abs/2311.10770: “Exponentially Faster Language Modelling”, -
https://arxiv.org/abs/2310.17157: “Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time”, -
https://arxiv.org/abs/2308.14711: “Fast Feedforward Networks”, -
https://arxiv.org/abs/2302.12441: “MUX-PLMs: Pre-Training Language Models With Data Multiplexing”, -
https://arxiv.org/abs/2210.06313#google: “The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers”, -
https://arxiv.org/abs/2207.03620: “More ConvNets in the 2020s: Scaling up Kernels Beyond 51×51 Using Sparsity (SLaK)”, -
https://arxiv.org/abs/2205.03983#google: “Building Machine Translation Systems for the Next Thousand Languages”, -
https://arxiv.org/abs/2204.00595: “Monarch: Expressive Structured Matrices for Efficient and Accurate Training”, -
https://arxiv.org/abs/2203.06850: “Efficient Language Modeling With Sparse All-MLP”, -
https://arxiv.org/abs/2202.07415#deepmind: “NeuPL: Neural Population Learning”, -
https://arxiv.org/abs/2106.09685#microsoft: “LoRA: Low-Rank Adaptation of Large Language Models”, -
https://greydanus.github.io/2020/12/01/scaling-down/: “Scaling down Deep Learning”, -
https://arxiv.org/abs/1905.11946#google: “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks”, -
https://arxiv.org/abs/1803.10615: “SqueezeNext: Hardware-Aware Neural Network Design”,