--- title: '‘NN sparsity’ directory' description: "Bibliography for directory ai/nn/sparsity, most recent first: 7 related tags, 123 annotations, & 22 links ()." thumbnail: /doc/ai/nn/sparsity/2022-bapna-figure2-googletranslateneuralmachinetranslationscalingbylanguagecorpussize.jpg thumbnail-text: '' thumbnail-css: "outline" created: 2019-12-07 modified: 2026-07-19 status: in progress previous: /doc/ai/nn/sampling/index next: /doc/ai/nn/sparsity/knowledge-distillation/index confidence: log importance: 0 css-extension: dropcaps-not index: True backlink: False ... ::: {#manual-annotation .abstract-small .abstract-tag-directory} [\[page summary\]](/doc/ai/nn/sparsity/abstract "Transclude link for doc/ai/nn/sparsity/ tag-abstract page."){#gwern-doc-ai-nn-sparsity-abstract .link-annotated-partial .include-content-core .include-strict .link-page} ::: # See Also ::: {#see-alsos .directory-indexes .columns} - [Parent ('*`NN`{=html}*' tag)](/doc/ai/nn/index "Link to parent directory 'doc/ai/nn/' (ascending)"){#_ujmXzDuq .link-annotated .link-tag .directory-indexes-upwards rel="tag"} - [*`knowledge distillation`{=html}*](/doc/ai/nn/sparsity/knowledge-distillation/index "‘knowledge distillation’ directory"){#_Dc4dmVf_ .link-annotated .link-tag .directory-indexes-downwards rel="tag"} - [*`low-precision NN`{=html}*](/doc/ai/nn/sparsity/low-precision/index "‘low-precision NN’ directory"){#_Jey1dXgh .link-annotated .link-tag .directory-indexes-downwards rel="tag"} - [*`NN pruning`{=html}*](/doc/ai/nn/sparsity/pruning/index "‘NN pruning’ directory"){#_DKz_UUhy .link-annotated .link-tag .directory-indexes-downwards rel="tag"} - [*`sparse Transformer`{=html}*](/doc/ai/nn/transformer/attention/sparsity/index "‘sparse Transformer’ directory"){#_C9QXsKvk .link-annotated .link-tag .directory-indexes-sideways rel="tag"} - [*`MoE NN`{=html}*](/doc/ai/scaling/mixture-of-experts/index "‘MoE NN’ directory"){#_REiAxK2d .link-annotated .link-tag .directory-indexes-sideways rel="tag"} - [*`Portia spider`{=html}*](/doc/biology/portia/index "‘𝑃𝑜𝑟𝑡𝑖𝑎 spider’ directory"){#_Czl_djKh .link-annotated .link-tag .directory-indexes-sideways rel="tag"} - [*`compression`{=html}*](/doc/cs/algorithm/information/compression/index "‘compression’ directory"){#_xacO3S9W .link-annotated .link-tag .directory-indexes-sideways rel="tag"} ::: # Links {.display-pop-not} ## `“Navigating the Open-Source Model Ecosystem: An Empirical Study of Creator Practices in Artistic Image Generation”, Wei et al 2026`{=html} {#wei-et-al-2026-section} **[`Navigating the Open-Source Model Ecosystem: An Empirical Study of Creator Practices in Artistic Image Generation`{=html}](https://arxiv.org/abs/2607.10538 "'Navigating the Open-Source Model Ecosystem: An Empirical Study of Creator Practices in Artistic Image Generation', Wei et al 2026"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Tracing the Thought of a Grandmaster-Level Chess-Playing Transformer”, Lin et al 2026`{=html} {#lin-et-al-2026-section} **[`Tracing the Thought of a Grandmaster-level Chess-Playing Transformer`{=html}](https://arxiv.org/abs/2604.10158 "'Tracing the Thought of a Grandmaster-level Chess-Playing Transformer', Lin et al 2026"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Doc-To-LoRA (D2L): Learning to Instantly Internalize Contexts”, Charakorn et al 2026`{=html} {#charakorn-et-al-2026-section} **[`Doc-to-LoRA (D2L): Learning to Instantly Internalize Contexts`{=html}](https://arxiv.org/abs/2602.15902#sakana "'Doc-to-LoRA (D2L): Learning to Instantly Internalize Contexts', Charakorn et al 2026"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Split Personality Training (SPT): Revealing Latent Knowledge Through Alternate Personalities”, Dietz et al 2026`{=html} {#dietz-et-al-2026-section} **[`Split Personality Training (SPT): Revealing Latent Knowledge Through Alternate Personalities`{=html}](https://arxiv.org/abs/2602.05532 "'Split Personality Training (SPT): Revealing Latent Knowledge Through Alternate Personalities', Dietz et al 2026"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Learning to Reason in 13 Parameters”, Morris et al 2026`{=html} {#morris-et-al-2026-section} **[`Learning to Reason in 13 Parameters`{=html}](https://arxiv.org/abs/2602.04118 "'Learning to Reason in 13 Parameters', Morris et al 2026"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models”, Cheng et al 2026`{=html} {#cheng-et-al-2026-section} **[`Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models`{=html}](https://arxiv.org/abs/2601.07372#deepseek "'Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models', Cheng et al 2026"){.link-annotated .id-not .include-annotation link-icon="deepseek" link-icon-type="svg" link-icon-color="#4d6bfe"}** ## `“SonicMoE: Accelerating MoE With IO and Tile-Aware Optimizations”, Guo et al 2025`{=html} {#guo-et-al-2025-section} **[`SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations`{=html}](https://arxiv.org/abs/2512.14080 "'SonicMoE: Accelerating MoE with IO and Tile-aware Optimizations', Guo et al 2025"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Ladder Up, Memory Down: Low-Cost Fine-Tuning With Side Nets (LST/xLadder)”, Zheng et al 2025`{=html} {#zheng-et-al-2025-section} **[`Ladder Up, Memory Down: Low-Cost Fine-Tuning With Side Nets (LST/xLadder)`{=html}](https://arxiv.org/abs/2512.14237 "'Ladder Up, Memory Down: Low-Cost Fine-Tuning With Side Nets (LST/xLadder)', Zheng et al 2025"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“The Universal Weight Subspace Hypothesis”, Kaushik et al 2025`{=html} {#kaushik-et-al-2025-section} **[`The Universal Weight Subspace Hypothesis`{=html}](https://arxiv.org/abs/2512.05117 "'The Universal Weight Subspace Hypothesis', Kaushik et al 2025"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Continual Learning via Sparse Memory Finetuning”, Lin et al 2025`{=html} {#lin-et-al-2025-section} **[`Continual Learning via Sparse Memory Finetuning`{=html}](https://arxiv.org/abs/2510.15103#facebook "'Continual Learning via Sparse Memory Finetuning', Lin et al 2025"){.link-annotated .id-not .include-annotation link-icon="facebook" link-icon-type="svg" link-icon-color="#1877f2"}** ## `“Learning to Interpret Weight Differences in Language Models”, Goel et al 2025`{=html} {#goel-et-al-2025-section} **[`Learning to Interpret Weight Differences in Language Models`{=html}](https://arxiv.org/abs/2510.05092 "'Learning to Interpret Weight Differences in Language Models', Goel et al 2025"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Diff-Interpretation-Tuning: Code for Learning to Interpret Weight Differences in Language Models (Goel Et Al 2025)”, Aviously 2025`{=html} {#aviously-2025-section} **[`diff-interpretation-tuning: Code for Learning to Interpret Weight Differences in Language Models (Goel et al 2025)`{=html}](https://github.com/Aviously/diff-interpretation-tuning "'diff-interpretation-tuning: Code for Learning to Interpret Weight Differences in Language Models (Goel et al 2025)', Aviously 2025"){.link-annotated-partial .id-not .include-annotation link-icon="github" link-icon-type="svg"}** ## `“Learning to Interpret Weight Differences in Language Models [Blog]”, Goel 2025`{=html} {#goel-2025-section} **[`Learning to Interpret Weight Differences in Language Models [blog]`{=html}](https://www.lesswrong.com/posts/EKhTrhrCz2rNg7FmG/learning-to-interpret-weight-differences-in-language-models-1 "'Learning to Interpret Weight Differences in Language Models [blog]', Goel 2025"){.link-annotated-partial .id-not .include-annotation link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"}** ## `“LoRA Without Regret”, Inc 2025`{=html} {#inc-2025-section} **[`LoRA Without Regret`{=html}](https://thinkingmachines.ai/blog/lora/ "'LoRA Without Regret', Inc 2025"){.link-annotated-partial .id-not .include-annotation}** ## `“Learning without Training: The Implicit Dynamics of In-Context Learning”, Dherin et al 2025`{=html} {#dherin-et-al-2025-section} **[`Learning without training: The implicit dynamics of in-context learning`{=html}](https://arxiv.org/abs/2507.16003#google "'Learning without training: The implicit dynamics of in-context learning', Dherin et al 2025"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“Text-To-LoRA (T2L): Instant Transformer Adaption”, Charakorn et al 2025`{=html} {#charakorn-et-al-2025-section} **[`Text-to-LoRA (T2L): Instant Transformer Adaption`{=html}](https://arxiv.org/abs/2506.06105 "'Text-to-LoRA (T2L): Instant Transformer Adaption', Charakorn et al 2025"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Reinforcement Learning Finetunes Small Subnetworks in Large Language Models”, Mukherjee et al 2025`{=html} {#mukherjee-et-al-2025-section} **[`Reinforcement Learning Finetunes Small Subnetworks in Large Language Models`{=html}](https://arxiv.org/abs/2505.11711 "'Reinforcement Learning Finetunes Small Subnetworks in Large Language Models', Mukherjee et al 2025"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Tina: Tiny Reasoning Models via LoRA”, Wang et al 2025`{=html} {#wang-et-al-2025-section} **[`Tina: Tiny Reasoning Models via LoRA`{=html}](https://arxiv.org/abs/2504.15777 "'Tina: Tiny Reasoning Models via LoRA', Wang et al 2025"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“UltraMem: Ultra-Sparse Memory Network”, Huang et al 2024`{=html} {#huang-et-al-2024-section} **[`UltraMem: Ultra-Sparse Memory Network`{=html}](https://arxiv.org/abs/2411.12364#bytedance "'UltraMem: Ultra-Sparse Memory Network', Huang et al 2024"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Convolutional Differentiable Logic Gate Networks”, Petersen et al 2024`{=html} {#petersen-et-al-2024-section} **[`Convolutional Differentiable Logic Gate Networks`{=html}](https://arxiv.org/abs/2411.04732 "'Convolutional Differentiable Logic Gate Networks', Petersen et al 2024"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“LoRA vs Full Fine-Tuning: An Illusion of Equivalence”, Shuttleworth et al 2024`{=html} {#shuttleworth-et-al-2024-section} **[`LoRA vs Full Fine-tuning: An Illusion of Equivalence`{=html}](https://arxiv.org/abs/2410.21228 "'LoRA vs Full Fine-tuning: An Illusion of Equivalence', Shuttleworth et al 2024"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics”, Nikankin et al 2024`{=html} {#nikankin-et-al-2024-section} **[`Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics`{=html}](https://arxiv.org/abs/2410.21272 "'Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics', Nikankin et al 2024"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“On the Complexity of Neural Computation in Superposition”, Adler & Shavit 2024`{=html} {#adler-shavit-2024-section} **[`On the Complexity of Neural Computation in Superposition`{=html}](https://arxiv.org/abs/2409.15318 "'On the Complexity of Neural Computation in Superposition', Adler & Shavit 2024"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“GSoC 2024: Differentiable Logic for Interactive Systems and Generative Music”`{=html} {#_u_ekCs7N-section} **[`GSoC 2024: Differentiable Logic for Interactive Systems and Generative Music`{=html}](https://ijc8.me/2024/08/26/gsoc-difflogic/ "GSoC 2024: Differentiable Logic for Interactive Systems and Generative Music"){.link-annotated-partial .id-not .include-annotation}** ## `“High-Performance Deep Spiking Neural Networks With 0.3 Spikes per Neuron”, Stanojevic et al 2024`{=html} {#stanojevic-et-al-2024-section} **[`High-performance deep spiking neural networks with 0.3 spikes per neuron`{=html}](https://www.nature.com/articles/s41467-024-51110-5 "'High-performance deep spiking neural networks with 0.3 spikes per neuron', Stanojevic et al 2024"){.link-annotated .id-not .include-annotation link-icon="n" link-icon-type="text"}** ## `“LoRA Learns Less and Forgets Less”, Biderman et al 2024`{=html} {#biderman-et-al-2024-section} **[`LoRA Learns Less and Forgets Less`{=html}](https://arxiv.org/abs/2405.09673 "'LoRA Learns Less and Forgets Less', Biderman et al 2024"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“CATS: Contextually-Aware Thresholding for Sparsity in Large Language Models”, Lee et al 2024`{=html} {#lee-et-al-2024-3-section} **[`CATS: Contextually-Aware Thresholding for Sparsity in Large Language Models`{=html}](https://arxiv.org/abs/2404.08763 "'CATS: Contextually-Aware Thresholding for Sparsity in Large Language Models', Lee et al 2024"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Exploring Concept Depth: How Large Language Models Acquire Knowledge at Different Layers?”, Jin et al 2024`{=html} {#jin-et-al-2024-3-section} **[`Exploring Concept Depth: How Large Language Models Acquire Knowledge at Different Layers?`{=html}](https://arxiv.org/abs/2404.07066 "'Exploring Concept Depth: How Large Language Models Acquire Knowledge at Different Layers?', Jin et al 2024"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“ReFT: Representation Finetuning for Language Models”, Wu et al 2024`{=html} {#wu-et-al-2024-3-section} **[`ReFT: Representation Finetuning for Language Models`{=html}](https://arxiv.org/abs/2404.03592 "'ReFT: Representation Finetuning for Language Models', Wu et al 2024"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Mechanistic Design and Scaling of Hybrid Architectures”, Poli et al 2024`{=html} {#poli-et-al-2024-section} **[`Mechanistic Design and Scaling of Hybrid Architectures`{=html}](https://arxiv.org/abs/2403.17844 "'Mechanistic Design and Scaling of Hybrid Architectures', Poli et al 2024"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“LTE: Training Neural Networks from Scratch With Parallel Low-Rank Adapters”, Huh et al 2024`{=html} {#huh-et-al-2024-section} **[`LTE: Training Neural Networks from Scratch with Parallel Low-Rank Adapters`{=html}](https://arxiv.org/abs/2402.16828 "'LTE: Training Neural Networks from Scratch with Parallel Low-Rank Adapters', Huh et al 2024"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Scaling Laws for Fine-Grained Mixture of Experts”, Krajewski et al 2024`{=html} {#krajewski-et-al-2024-section} **[`Scaling Laws for Fine-Grained Mixture of Experts`{=html}](https://arxiv.org/abs/2402.07871 "'Scaling Laws for Fine-Grained Mixture of Experts', Krajewski et al 2024"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet”`{=html} {#_MN0ws55D-section} **[`Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet`{=html}](https://transformer-circuits.pub/2024/scaling-monosemanticity/ "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet"){.link-annotated-partial .id-not .include-annotation link-icon="anthropic" link-icon-type="svg" link-icon-color="#d4a27f"}** ## `“Exponentially Faster Language Modeling”, Belcak & Wattenhofer 2023`{=html} {#belcak-wattenhofer-2023-1-section} **[`Exponentially Faster Language Modeling`{=html}](https://arxiv.org/abs/2311.10770 "'Exponentially Faster Language Modeling', Belcak & Wattenhofer 2023"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“DiLoCo: Distributed Low-Communication Training of Language Models”, Douillard et al 2023`{=html} {#douillard-et-al-2023-section} **[`DiLoCo: Distributed Low-Communication Training of Language Models`{=html}](https://arxiv.org/abs/2311.08105 "'DiLoCo: Distributed Low-Communication Training of Language Models', Douillard et al 2023"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Language Models Are Super Mario (DARE): Absorbing Abilities from Homologous Models As a Free Lunch”, Yu et al 2023`{=html} {#yu-et-al-2023-2-section} **[`Language Models are Super Mario (DARE): Absorbing Abilities from Homologous Models as a Free Lunch`{=html}](https://arxiv.org/abs/2311.03099 "'Language Models are Super Mario (DARE): Absorbing Abilities from Homologous Models as a Free Lunch', Yu et al 2023"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“ProSG: Using Prompt Synthetic Gradients to Alleviate Prompt Forgetting of RNN-Like Language Models”, Luo et al 2023`{=html} {#luo-et-al-2023-2-section} **[`ProSG: Using Prompt Synthetic Gradients to Alleviate Prompt Forgetting of RNN-like Language Models`{=html}](https://arxiv.org/abs/2311.01981 "'ProSG: Using Prompt Synthetic Gradients to Alleviate Prompt Forgetting of RNN-like Language Models', Luo et al 2023"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“An Exact Mapping from ReLU Networks to Spiking Neural Networks”, Stanojevic et al 2023`{=html} {#stanojevic-et-al-2023-section} **[`An exact mapping from ReLU networks to spiking neural networks`{=html}](https://www.sciencedirect.com/science/article/pii/S0893608023005051 "'An exact mapping from ReLU networks to spiking neural networks', Stanojevic et al 2023"){.link-annotated .id-not .include-annotation link-icon="E" link-icon-type="text" link-icon-color="#eb6500"}** ## `“The Impact of Depth and Width on Transformer Language Model Generalization”, Petty et al 2023`{=html} {#petty-et-al-2023-section} **[`The Impact of Depth and Width on Transformer Language Model Generalization`{=html}](https://arxiv.org/abs/2310.19956 "'The Impact of Depth and Width on Transformer Language Model Generalization', Petty et al 2023"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time”, Liu et al 2023`{=html} {#liu-et-al-2023-05-section} **[`Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time`{=html}](https://arxiv.org/abs/2310.17157 "'Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time', Liu et al 2023"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Fast Feedforward Networks”, Belcak & Wattenhofer 2023`{=html} {#belcak-wattenhofer-2023-2-section} **[`Fast Feedforward Networks`{=html}](https://arxiv.org/abs/2308.14711 "'Fast Feedforward Networks', Belcak & Wattenhofer 2023"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“ReLoRA: High-Rank Training Through Low-Rank Updates”, Lialin et al 2023`{=html} {#lialin-et-al-2023-section} **[`ReLoRA: High-Rank Training Through Low-Rank Updates`{=html}](https://arxiv.org/abs/2307.05695 "'ReLoRA: High-Rank Training Through Low-Rank Updates', Lialin et al 2023"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Any Deep ReLU Network Is Shallow”, Villani & Schoots 2023`{=html} {#villani-schoots-2023-section} **[`Any Deep ReLU Network is Shallow`{=html}](https://arxiv.org/abs/2306.11827 "'Any Deep ReLU Network is Shallow', Villani & Schoots 2023"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Cerebras Architecture Deep Dive: First Look Inside the HW/SW Co-Design for Deep Learning [Updated]”, Lie 2023`{=html} {#lie-2023-section} **[`Cerebras Architecture Deep Dive: First Look Inside the HW/SW Co-Design for Deep Learning [Updated]`{=html}](https://cerebras.ai/blog/cerebras-architecture-deep-dive-first-look-inside-the-hw/sw-co-design-for-deep-learning "'Cerebras Architecture Deep Dive: First Look Inside the HW/SW Co-Design for Deep Learning [Updated]', Lie 2023"){.id-not .include-annotation link-icon="C" link-icon-type="text,sans" link-icon-color="#f05a29"}** ## `“JaxPruner: A Concise Library for Sparsity Research”, Lee et al 2023`{=html} {#lee-et-al-2023-5-section} **[`JaxPruner: A concise library for sparsity research`{=html}](https://arxiv.org/abs/2304.14082#google "'JaxPruner: A concise library for sparsity research', Lee et al 2023"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“Reusing Deep Neural Network Models through Model Re-Engineering”, Qi et al 2023`{=html} {#qi-et-al-2023-3-section} **[`Reusing Deep Neural Network Models through Model Re-engineering`{=html}](https://arxiv.org/abs/2304.00245 "'Reusing Deep Neural Network Models through Model Re-engineering', Qi et al 2023"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Accelerating Large GPT Training With Sparse Pre-Training and Dense Fine-Tuning”, Thangarasa 2023`{=html} {#thangarasa-2023-section} **[`Accelerating Large GPT Training with Sparse Pre-Training and Dense Fine-Tuning`{=html}](https://cerebras.ai/machine-learning/accelerating-large-gpt-training-with-sparse-pre-training-and-dense-fine-tuning/ "'Accelerating Large GPT Training with Sparse Pre-Training and Dense Fine-Tuning', Thangarasa 2023"){.id-not .include-annotation link-icon="C" link-icon-type="text,sans" link-icon-color="#f05a29"}** ## `“MUX-PLMs: Pre-Training Language Models With Data Multiplexing”, Murahari et al 2023`{=html} {#murahari-et-al-2023-2-section} **[`MUX-PLMs: Pre-training Language Models with Data Multiplexing`{=html}](https://arxiv.org/abs/2302.12441 "'MUX-PLMs: Pre-training Language Models with Data Multiplexing', Murahari et al 2023"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“DataMUX: Data Multiplexing for Neural Networks”, Murahari et al 2023`{=html} {#murahari-et-al-2023-1-section} **[`DataMUX: Data Multiplexing for Neural Networks`{=html}](https://openreview.net/forum?id=UdgtTVTdswg "'DataMUX: Data Multiplexing for Neural Networks', Murahari et al 2023"){.link-annotated .id-not .include-annotation link-icon="OR" link-icon-type="text,sans" link-icon-color="#8c1b13"}** ## `“Deep Differentiable Logic Gate Networks”, Petersen et al 2022`{=html} {#petersen-et-al-2022-section} **[`Deep Differentiable Logic Gate Networks`{=html}](https://arxiv.org/abs/2210.08277 "'Deep Differentiable Logic Gate Networks', Petersen et al 2022"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers”, Li et al 2022`{=html} {#li-et-al-2022-11-section} **[`The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers`{=html}](https://arxiv.org/abs/2210.06313#google "'The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers', Li et al 2022"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“Neural Net Sparsity”, Gwern 2022`{=html} {#gwern-doc-ai-nn-sparsity-abstract-section} **[`Neural Net Sparsity`{=html}](/doc/ai/nn/sparsity/abstract "'Neural Net Sparsity', Gwern 2022"){.link-annotated-partial .id-not .include-annotation}** ## `“Noise Transforms Feed-Forward Networks into Sparse Coding Networks”, Anonymous 2022`{=html} {#anonymous-2022-4-section} **[`Noise Transforms Feed-Forward Networks into Sparse Coding Networks`{=html}](https://openreview.net/forum?id=P9yXPbfqbvC "'Noise Transforms Feed-Forward Networks into Sparse Coding Networks', Anonymous 2022"){.link-annotated .id-not .include-annotation link-icon="OR" link-icon-type="text,sans" link-icon-color="#8c1b13"}** ## `“Exploring Low Rank Training of Deep Neural Networks”, Kamalakara et al 2022`{=html} {#kamalakara-et-al-2022-section} **[`Exploring Low Rank Training of Deep Neural Networks`{=html}](https://arxiv.org/abs/2209.13569#cohere "'Exploring Low Rank Training of Deep Neural Networks', Kamalakara et al 2022"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Monolith: Real Time Recommendation System With Collisionless Embedding Table”, Liu et al 2022`{=html} {#liu-et-al-2022-12-section} **[`Monolith: Real Time Recommendation System With Collisionless Embedding Table`{=html}](https://arxiv.org/abs/2209.07663#bytedance "'Monolith: Real Time Recommendation System With Collisionless Embedding Table', Liu et al 2022"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“More ConvNets in the 2020s: Scaling up Kernels Beyond 51×51 Using Sparsity (SLaK)”, Liu et al 2022`{=html} {#liu-et-al-2022-15-section} **[`More ConvNets in the 2020s: Scaling up Kernels Beyond 51×51 using Sparsity (SLaK)`{=html}](https://arxiv.org/abs/2207.03620 "'More ConvNets in the 2020s: Scaling up Kernels Beyond 51×51 using Sparsity (SLaK)', Liu et al 2022"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Building Machine Translation Systems for the Next Thousand Languages”, Bapna et al 2022`{=html} {#bapna-et-al-2022-section} **[`Building Machine Translation Systems for the Next Thousand Languages`{=html}](https://arxiv.org/abs/2205.03983#google "'Building Machine Translation Systems for the Next Thousand Languages', Bapna et al 2022"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“Monarch: Expressive Structured Matrices for Efficient and Accurate Training”, Dao et al 2022`{=html} {#dao-et-al-2022-2-section} **[`Monarch: Expressive Structured Matrices for Efficient and Accurate Training`{=html}](https://arxiv.org/abs/2204.00595 "'Monarch: Expressive Structured Matrices for Efficient and Accurate Training', Dao et al 2022"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Efficient Language Modeling With Sparse All-MLP”, Yu et al 2022`{=html} {#yu-et-al-2022-5-section} **[`Efficient Language Modeling with Sparse All-MLP`{=html}](https://arxiv.org/abs/2203.06850 "'Efficient Language Modeling with Sparse All-MLP', Yu et al 2022"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“NeuPL: Neural Population Learning”, Liu et al 2022`{=html} {#liu-et-al-2022-22-section} **[`NeuPL: Neural Population Learning`{=html}](https://arxiv.org/abs/2202.07415#deepmind "'NeuPL: Neural Population Learning', Liu et al 2022"){.link-annotated .id-not .include-annotation link-icon="deepmind" link-icon-type="svg" link-icon-color="#4185f4"}** ## `“Datamodels: Predicting Predictions from Training Data”, Ilyas et al 2022`{=html} {#ilyas-et-al-2022-section} **[`Datamodels: Predicting Predictions from Training Data`{=html}](https://arxiv.org/abs/2202.00622 "'Datamodels: Predicting Predictions from Training Data', Ilyas et al 2022"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Spiking Neural Networks and Their Applications: A Review”, Yamazaki et al 2022`{=html} {#yamazaki-et-al-2022-section} **[`Spiking Neural Networks and Their Applications: A Review`{=html}](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9313413/ "'Spiking Neural Networks and Their Applications: A Review', Yamazaki et al 2022"){.link-annotated .id-not .include-annotation link-icon="nlm-ncbi" link-icon-type="svg" link-icon-color="#20558a"}** ## `“Persia: An Open, Hybrid System Scaling Deep Learning-Based Recommenders up to 100 Trillion Parameters”, Lian et al 2021`{=html} {#lian-et-al-2021-1-section} **[`Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters`{=html}](https://arxiv.org/abs/2111.05897 "'Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters', Lian et al 2021"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“EvilModel: Hiding Malware Inside of Neural Network Models”, Wang et al 2021`{=html} {#wang-et-al-2021-evilmodel-section} **[`EvilModel: Hiding Malware Inside of Neural Network Models`{=html}](https://arxiv.org/abs/2107.08590 "'EvilModel: Hiding Malware Inside of Neural Network Models', Wang et al 2021"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“BitFit: Simple Parameter-Efficient Fine-Tuning for Transformer-Based Masked Language-Models”, Zaken et al 2021`{=html} {#zaken-et-al-2021-section} **[`BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models`{=html}](https://arxiv.org/abs/2106.10199 "'BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models', Zaken et al 2021"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“LoRA: Low-Rank Adaptation of Large Language Models”, Hu et al 2021`{=html} {#hu-et-al-2021-3-section} **[`LoRA: Low-Rank Adaptation of Large Language Models`{=html}](https://arxiv.org/abs/2106.09685#microsoft "'LoRA: Low-Rank Adaptation of Large Language Models', Hu et al 2021"){.link-annotated .id-not .include-annotation link-icon="MS" link-icon-type="text,sans,italic"}** ## `“On the Distribution, Sparsity, and Inference-Time Quantization of Attention Values in Transformers”, Ji et al 2021`{=html} {#ji-et-al-2021-section} **[`On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers`{=html}](https://arxiv.org/abs/2106.01335 "'On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers', Ji et al 2021"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“The Neural Basis of Intelligence in Fine-Grained Cortical Topographies”, Feilong et al 2021`{=html} {#feilong-et-al-2021-section} **[`The neural basis of intelligence in fine-grained cortical topographies`{=html}](https://elifesciences.org/articles/64058 "'The neural basis of intelligence in fine-grained cortical topographies', Feilong et al 2021"){.link-annotated .id-not .include-annotation link-icon="eL" link-icon-type="text,sans"}** ## `“Clusterability in Neural Networks”, Filan et al 2021`{=html} {#filan-et-al-2021-section} **[`Clusterability in Neural Networks`{=html}](https://arxiv.org/abs/2103.03386 "'Clusterability in Neural Networks', Filan et al 2021"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Mind the Gap: Assessing Temporal Generalization in Neural Language Models § Dynamic Evaluation”, Lazaridou et al 2021 (page 7 org deepmind)`{=html} {#lazaridou-et-al-2021-page-7-org-deepmind-section} **[`Mind the Gap: Assessing Temporal Generalization in Neural Language Models § Dynamic Evaluation`{=html}](https://arxiv.org/pdf/2102.01951#page=7&org=deepmind "'Mind the Gap: Assessing Temporal Generalization in Neural Language Models § Dynamic Evaluation', Lazaridou et al 2021 (page 7 org deepmind)"){.link-annotated .id-not .include-annotation link-icon="deepmind" link-icon-type="svg" link-icon-color="#4185f4"}** ## `“Sparsity in Deep Learning: Pruning and Growth for Efficient Inference and Training in Neural Networks”, Hoefler et al 2021`{=html} {#hoefler-et-al-2021-section} **[`Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks`{=html}](https://arxiv.org/abs/2102.00554 "'Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks', Hoefler et al 2021"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning”, Aghajanyan et al 2020`{=html} {#aghajanyan-et-al-2020-section} **[`Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning`{=html}](https://arxiv.org/abs/2012.13255 "'Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning', Aghajanyan et al 2020"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Scaling down Deep Learning”, Greydanus 2020`{=html} {#greydanus-2020-section} **[`Scaling down Deep Learning`{=html}](https://greydanus.github.io/2020/12/01/scaling-down/ "'Scaling down Deep Learning', Greydanus 2020"){.link-annotated .id-not .include-annotation}** ## `“Extreme Model Compression for On-Device Natural Language Understanding”, Sathyendra et al 2020`{=html} {#sathyendra-et-al-2020-section} **[`Extreme Model Compression for On-device Natural Language Understanding`{=html}](https://arxiv.org/abs/2012.00124#amazon "'Extreme Model Compression for On-device Natural Language Understanding', Sathyendra et al 2020"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Training Independent Subnetworks for Robust Prediction”, Havasi et al 2020`{=html} {#havasi-et-al-2020-section} **[`Training independent subnetworks for robust prediction`{=html}](https://arxiv.org/abs/2010.06610 "'Training independent subnetworks for robust prediction', Havasi et al 2020"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“EventProp: Event-Based Backpropagation Can Compute Exact Gradients for Spiking Neural Networks”, Wunderlich & Pehle 2020`{=html} {#wunderlich-pehle-2020-section} **[`EventProp: Event-Based Backpropagation can compute Exact Gradients for Spiking Neural Networks`{=html}](https://arxiv.org/abs/2009.08378 "'EventProp: Event-Based Backpropagation can compute Exact Gradients for Spiking Neural Networks', Wunderlich & Pehle 2020"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“On Linear Identifiability of Learned Representations”, Roeder et al 2020`{=html} {#roeder-et-al-2020-section} **[`On Linear Identifiability of Learned Representations`{=html}](https://arxiv.org/abs/2007.00810#google "'On Linear Identifiability of Learned Representations', Roeder et al 2020"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited”, Maddox et al 2020`{=html} {#maddox-et-al-2020-section} **[`Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited`{=html}](https://arxiv.org/abs/2003.02139 "'Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited', Maddox et al 2020"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Bayesian Deep Learning and a Probabilistic Perspective of Generalization”, Wilson & Izmailov 2020`{=html} {#wilson-izmailov-2020-section} **[`Bayesian Deep Learning and a Probabilistic Perspective of Generalization`{=html}](https://arxiv.org/abs/2002.08791 "'Bayesian Deep Learning and a Probabilistic Perspective of Generalization', Wilson & Izmailov 2020"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Neural Arithmetic Units”, Madsen & Johansen 2020`{=html} {#madsen-johansen-2020-section} **[`Neural Arithmetic Units`{=html}](https://arxiv.org/abs/2001.05016 "'Neural Arithmetic Units', Madsen & Johansen 2020"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Linear Mode Connectivity and the Lottery Ticket Hypothesis”, Frankle et al 2019`{=html} {#frankle-et-al-2019-1-section} **[`Linear Mode Connectivity and the Lottery Ticket Hypothesis`{=html}](https://arxiv.org/abs/1912.05671 "'Linear Mode Connectivity and the Lottery Ticket Hypothesis', Frankle et al 2019"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Learning to Seek: Autonomous Source Seeking With Deep Reinforcement Learning Onboard a Nano Drone Microcontroller”, Duisterhof et al 2019`{=html} {#duisterhof-et-al-2019-section} **[`Learning to Seek: Autonomous Source Seeking with Deep Reinforcement Learning Onboard a Nano Drone Microcontroller`{=html}](https://arxiv.org/abs/1909.11236 "'Learning to Seek: Autonomous Source Seeking with Deep Reinforcement Learning Onboard a Nano Drone Microcontroller', Duisterhof et al 2019"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Does Learning Require Memorization? A Short Tale about a Long Tail”, Feldman 2019`{=html} {#feldman-2019-section} **[`Does Learning Require Memorization? A Short Tale about a Long Tail`{=html}](https://arxiv.org/abs/1906.05271#google "'Does Learning Require Memorization? A Short Tale about a Long Tail', Feldman 2019"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“WANN: Weight Agnostic Neural Networks”, Gaier & Ha 2019`{=html} {#gaier-ha-2019-section} **[`WANN: Weight Agnostic Neural Networks`{=html}](https://arxiv.org/abs/1906.04358#google "'WANN: Weight Agnostic Neural Networks', Gaier & Ha 2019"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“StyleNAS: An Empirical Study of Neural Architecture Search to Uncover Surprisingly Fast End-To-End Universal Style Transfer Networks”, An et al 2019`{=html} {#an-et-al-2019-section} **[`StyleNAS: An Empirical Study of Neural Architecture Search to Uncover Surprisingly Fast End-to-End Universal Style Transfer Networks`{=html}](https://arxiv.org/abs/1906.02470 "'StyleNAS: An Empirical Study of Neural Architecture Search to Uncover Surprisingly Fast End-to-End Universal Style Transfer Networks', An et al 2019"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks”, Tan & Le 2019`{=html} {#tan-le-2019-section} **[`EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks`{=html}](https://arxiv.org/abs/1905.11946#google "'EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks', Tan & Le 2019"){.link-annotated .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** ## `“Superposition of Many Models into One”, Cheung et al 2019`{=html} {#cheung-et-al-2019-section} **[`Superposition of many models into one`{=html}](https://arxiv.org/abs/1902.05522 "'Superposition of many models into one', Cheung et al 2019"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Playing Atari With Six Neurons”, Cuccu et al 2018`{=html} {#cuccu-et-al-2018-section} **[`Playing Atari with Six Neurons`{=html}](https://arxiv.org/abs/1806.01363 "'Playing Atari with Six Neurons', Cuccu et al 2018"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Measuring the Intrinsic Dimension of Objective Landscapes”, Li et al 2018`{=html} {#li-et-al-2018-1-section} **[`Measuring the Intrinsic Dimension of Objective Landscapes`{=html}](https://arxiv.org/abs/1804.08838#uber "'Measuring the Intrinsic Dimension of Objective Landscapes', Li et al 2018"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“SqueezeNext: Hardware-Aware Neural Network Design”, Gholami et al 2018`{=html} {#gholami-et-al-2018-section} **[`SqueezeNext: Hardware-Aware Neural Network Design`{=html}](https://arxiv.org/abs/1803.10615 "'SqueezeNext: Hardware-Aware Neural Network Design', Gholami et al 2018"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Wide Compression: Tensor Ring Nets”, Wang et al 2018`{=html} {#wang-et-al-2018-6-section} **[`Wide Compression: Tensor Ring Nets`{=html}](https://arxiv.org/abs/1802.09052 "'Wide Compression: Tensor Ring Nets', Wang et al 2018"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Intriguing Properties of Randomly Weighted Networks: Generalizing While Learning Next to Nothing”, Rosenfeld & Tsotsos 2018`{=html} {#rosenfeld-tsotsos-2018-section} **[`Intriguing Properties of Randomly Weighted Networks: Generalizing while Learning Next to Nothing`{=html}](https://openreview.net/forum?id=Hy-w-2PSf "'Intriguing Properties of Randomly Weighted Networks: Generalizing while Learning Next to Nothing', Rosenfeld & Tsotsos 2018"){.link-annotated .id-not .include-annotation link-icon="OR" link-icon-type="text,sans" link-icon-color="#8c1b13"}** ## `“Fix Your Classifier: the Marginal Value of Training the Last Weight Layer”, Hoffer et al 2018`{=html} {#hoffer-et-al-2018-section} **[`Fix your classifier: the marginal value of training the last weight layer`{=html}](https://arxiv.org/abs/1801.04540 "'Fix your classifier: the marginal value of training the last weight layer', Hoffer et al 2018"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Learning Compact Recurrent Neural Networks With Block-Term Tensor Decomposition”, Ye et al 2017`{=html} {#ye-et-al-2017-section} **[`Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition`{=html}](https://arxiv.org/abs/1712.05134 "'Learning Compact Recurrent Neural Networks with Block-Term Tensor Decomposition', Ye et al 2017"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“3D Semantic Segmentation With Submanifold Sparse Convolutional Networks”, Graham et al 2017`{=html} {#graham-et-al-2017-1-section} **[`3D Semantic Segmentation with Submanifold Sparse Convolutional Networks`{=html}](https://arxiv.org/abs/1711.10275 "'3D Semantic Segmentation with Submanifold Sparse Convolutional Networks', Graham et al 2017"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“XUnit: Learning a Spatial Activation Function for Efficient Image Restoration”, Kligvasser et al 2017`{=html} {#kligvasser-et-al-2017-section} **[`xUnit: Learning a Spatial Activation Function for Efficient Image Restoration`{=html}](https://arxiv.org/abs/1711.06445 "'xUnit: Learning a Spatial Activation Function for Efficient Image Restoration', Kligvasser et al 2017"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Natural Language Processing With Small Feed-Forward Networks”, Botha et al 2017`{=html} {#botha-et-al-2017-section} **[`Natural Language Processing with Small Feed-Forward Networks`{=html}](https://arxiv.org/abs/1708.00214 "'Natural Language Processing with Small Feed-Forward Networks', Botha et al 2017"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices”, Zhang et al 2017`{=html} {#zhang-et-al-2017-4-section} **[`ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices`{=html}](https://arxiv.org/abs/1707.01083 "'ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices', Zhang et al 2017"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Submanifold Sparse Convolutional Networks”, Graham & Maaten 2017`{=html} {#graham-maaten-2017-section} **[`Submanifold Sparse Convolutional Networks`{=html}](https://arxiv.org/abs/1706.01307 "'Submanifold Sparse Convolutional Networks', Graham & Maaten 2017"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Shake-Shake Regularization of 3-Branch Residual Networks”, Gastaldi 2017`{=html} {#gastaldi-2017-section} **[`Shake-Shake regularization of 3-branch residual networks`{=html}](https://openreview.net/forum?id=HkO-PCmYl "'Shake-Shake regularization of 3-branch residual networks', Gastaldi 2017"){.link-annotated .id-not .include-annotation link-icon="OR" link-icon-type="text,sans" link-icon-color="#8c1b13"}** ## `“Using the Output Embedding to Improve Language Models”, Press & Wolf 2016`{=html} {#press-wolf-2016-section} **[`Using the Output Embedding to Improve Language Models`{=html}](https://arxiv.org/abs/1608.05859 "'Using the Output Embedding to Improve Language Models', Press & Wolf 2016"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Deep Residual Learning for Image Recognition”, He et al 2015`{=html} {#he-et-al-2015-2-section} **[`Deep Residual Learning for Image Recognition`{=html}](https://arxiv.org/abs/1512.03385#microsoft "'Deep Residual Learning for Image Recognition', He et al 2015"){.link-annotated .id-not .include-annotation link-icon="MS" link-icon-type="text,sans,italic"}** ## `“Tensorizing Neural Networks”, Novikov et al 2015`{=html} {#novikov-et-al-2015-section} **[`Tensorizing Neural Networks`{=html}](https://arxiv.org/abs/1509.06569 "'Tensorizing Neural Networks', Novikov et al 2015"){.link-annotated .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Eight Pairs of Descending Visual Neurons in the Dragonfly Give Wing Motor Centers Accurate Population Vector of Prey Direction”, Gonzalez-Bellido et al 2013`{=html} {#gonzalez-bellido-et-al-2013-section} **[`Eight pairs of descending visual neurons in the dragonfly give wing motor centers accurate population vector of prey direction`{=html}](https://www.pnas.org/doi/full/10.1073/pnas.1210489109 "'Eight pairs of descending visual neurons in the dragonfly give wing motor centers accurate population vector of prey direction', Gonzalez-Bellido et al 2013"){.link-annotated .id-not .include-annotation link-icon="PNAS" link-icon-type="text,quad" link-icon-color="#1f75b9"}** ## `“The Cat Is out of the Bag: Cortical Simulations With 109 Neurons, 1013 Synapses”, Ananthanarayanan et al 2009`{=html} {#ananthanarayanan-et-al-2009-section} **[`The cat is out of the bag: cortical simulations with 109 neurons, 1013 synapses`{=html}](/doc/ai/scaling/hardware/2009-ananthanarayanan.pdf "'The cat is out of the bag: cortical simulations with 109 neurons, 1013 synapses', Ananthanarayanan et al 2009"){.link-annotated .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“On the Computational Power of Threshold Circuits With Sparse Activity”, Uchizawa et al 2006`{=html} {#uchizawa-et-al-2006-section} **[`On the Computational Power of Threshold Circuits with Sparse Activity`{=html}](/doc/ai/nn/sparsity/2006-uchizawa.pdf "'On the Computational Power of Threshold Circuits with Sparse Activity', Uchizawa et al 2006"){.link-annotated .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“Networks of Spiking Neurons: The Third Generation of Neural Network Models”, Maass 1997`{=html} {#maass-1997-section} **[`Networks of spiking neurons: The third generation of neural network models`{=html}](/doc/ai/nn/sparsity/1997-maass.pdf "'Networks of spiking neurons: The third generation of neural network models', Maass 1997"){.link-annotated .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“Characteristics of Sparsely Encoded Associative Memory”, Amari 1989`{=html} {#amari-1989-section} **[`Characteristics of sparsely encoded associative memory`{=html}](/doc/ai/nn/sparsity/1989-amari.pdf "'Characteristics of sparsely encoded associative memory', Amari 1989"){.link-annotated .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** ## `“Building a Minimal Transformer for 10-Digit Addition”, Litzenberger 2026`{=html} {#_PYtd7NB4-section} **[`Building a minimal Transformer for 10-digit addition`{=html}](https://alexlitzenberger.com/blog/building_a_minimal_transformer_for_10_digit_addition "'Building a minimal Transformer for 10-digit addition', Litzenberger 2026"){.link-annotated-partial .id-not .include-annotation}** ## `“[2110.08152] Kronecker Decomposition for GPT Compression”`{=html} {#_x_d_lt31-section} **[`[2110.08152] Kronecker Decomposition for GPT Compression`{=html}](https://arxiv.org/abs/2110.08152 "[2110.08152] Kronecker Decomposition for GPT Compression"){.link-annotated-partial .id-not .include-annotation link-icon="𝛘" link-icon-type="text" link-icon-color="#b31b1b"}** ## `“Higher Accuracy on Vision Models With EfficientNet-Lite”`{=html} {#_FBjOci_K-section} **[`Higher accuracy on vision models with EfficientNet-Lite`{=html}](https://blog.tensorflow.org/2020/03/higher-accuracy-on-vision-models-with-efficientnet-lite.html "Higher accuracy on vision models with EfficientNet-Lite"){.link-annotated-partial .id-not .include-annotation link-icon="tensorflow" link-icon-type="svg" link-icon-color="#ff6f00"}** ## `“Something Weird Is Happening With LLMs and Chess”, Dynomight 2026`{=html} {#_wHZYrxKi-section} **[`Something weird is happening with LLMs and chess`{=html}](https://dynomight.net/chess/ "'Something weird is happening with LLMs and chess', Dynomight 2026"){.link-annotated-partial .id-not .include-annotation link-icon="D" link-icon-type="text,mono"}** ## `“Delivering Real-Time AI in the Palm of Your Hand”`{=html} {#_9rcb_S-5-section} **[`Delivering real-time AI in the palm of your hand`{=html}](https://engineering.fb.com/2016/11/08/android/delivering-real-time-ai-in-the-palm-of-your-hand/ "Delivering real-time AI in the palm of your hand"){.link-annotated-partial .id-not .include-annotation link-icon="facebook" link-icon-type="svg" link-icon-color="#1877f2"}** ## `“AdderBoard: Contest for Smallest Transformer That Can Add Two 10-Digit Numbers”`{=html} {#_4olwZWTw-section} **[`AdderBoard: contest for smallest Transformer that can add two 10-digit numbers`{=html}](https://github.com/anadim/AdderBoard "AdderBoard: contest for smallest Transformer that can add two 10-digit numbers"){.link-annotated-partial .id-not .include-annotation link-icon="github" link-icon-type="svg"}** ## `“Sparsity-Aware Deep Learning Inference Runtime for CPUs”`{=html} {#_oEVHvE1p-section} **[`Sparsity-aware deep learning inference runtime for CPUs`{=html}](https://github.com/neuralmagic/deepsparse "Sparsity-aware deep learning inference runtime for CPUs"){.id-not .include-annotation link-icon="github" link-icon-type="svg"}** ## `“Neuralmagic/sparseml: Libraries for Applying Sparsification Recipes to Neural Networks With a Few Lines of Code, Enabling Faster and Smaller Models”`{=html} {#_cHLxPVRV-section} **[`neuralmagic/sparseml: Libraries for applying sparsification recipes to neural networks with a few lines of code, enabling faster and smaller models`{=html}](https://github.com/neuralmagic/sparseml "neuralmagic/sparseml: Libraries for applying sparsification recipes to neural networks with a few lines of code, enabling faster and smaller models"){.id-not .include-annotation link-icon="github" link-icon-type="svg"}** ## `“Digit-Addition-311p: a Grokked 311-Parameter Transformer for 10-Digit Addition”`{=html} {#_yrSJQYVL-section} **[`digit-addition-311p: a grokked 311-parameter Transformer for 10-digit addition`{=html}](https://github.com/rezabyt/digit-addition-311p "digit-addition-311p: a grokked 311-parameter Transformer for 10-digit addition"){.link-annotated-partial .id-not .include-annotation link-icon="github" link-icon-type="svg"}** ## `“Technion-Cs-Nlp/llm-Arithmetic-Heuristics”`{=html} {#_WrubH-Yy-section} **[`technion-cs-nlp/llm-arithmetic-heuristics`{=html}](https://github.com/technion-cs-nlp/llm-arithmetic-heuristics "technion-cs-nlp/llm-arithmetic-heuristics"){.link-annotated-partial .id-not .include-annotation link-icon="github" link-icon-type="svg"}** ## `“An Estimation of the Absolute Number of Axons Indicates That Human Cortical Areas Are Sparsely Connected”`{=html} {#_-3lzk3lX-section} **[`An estimation of the absolute number of axons indicates that human cortical areas are sparsely connected`{=html}](https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3001575 "An estimation of the absolute number of axons indicates that human cortical areas are sparsely connected"){.id-not .include-annotation link-icon="plos" link-icon-type="svg"}** ## `“Creating a 17 KB Style Transfer Model With Layer Pruning and Quantization”, Toole 2026`{=html} {#_XybmjEPN-section} **[`Creating a 17 KB style transfer model with layer pruning and quantization`{=html}](https://medium.com/@jamesonthecrow/creating-a-17kb-style-transfer-model-with-layer-pruning-and-quantization-864d7cc53693 "'Creating a 17 KB style transfer model with layer pruning and quantization', Toole 2026"){.id-not .include-annotation link-icon="𝐌" link-icon-type="text"}** ## `“BERT-Large: Prune Once for DistilBERT Inference Performance”`{=html} {#_G1_Tosgf-section} **[`BERT-Large: Prune Once for DistilBERT Inference Performance`{=html}](https://neuralmagic.com/blog/bert-large-prune-once-for-distilbert-inference-performance/ "BERT-Large: Prune Once for DistilBERT Inference Performance"){.link-annotated-partial .id-not .include-annotation}** ## `“Circuits in Superposition: Compressing Many Small Neural Networks into One”`{=html} {#_zgzcD9sa-section} **[`Circuits in Superposition: Compressing many small neural networks into one`{=html}](https://www.lesswrong.com/posts/roE7SHjFWEoMcGZKd/circuits-in-superposition-compressing-many-small-neural#Read_in_interference "Circuits in Superposition: Compressing many small neural networks into one"){.id-not .include-annotation link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"}** ::: aux-links-transclude-file ::: collapse **View HTML**: [`/doc/www/www.greaterwrong.com/56cb7ccd134aaa922ba1f32126ca7c67fc25fb15.html#Read_in_interference`](/doc/www/www.greaterwrong.com/56cb7ccd134aaa922ba1f32126ca7c67fc25fb15.html#Read_in_interference){.id-not .link-annotated-not .include-content .include-lazy link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"} ::: ::: ## `“Measuring the Intrinsic Dimension of Objective Landscapes [Video]”`{=html} {#_nu2Yjb38-section} **[`Measuring the Intrinsic Dimension of Objective Landscapes [video]`{=html}](https://www.youtube.com/watch?v=uSZWeRADTFI#uber "Measuring the Intrinsic Dimension of Objective Landscapes [video]"){.id-not .include-annotation link-icon="youtube" link-icon-type="svg" link-icon-color="#ff0033"}** ::: aux-links-transclude-file [`https://www.youtube.com/watch?v=uSZWeRADTFI#uber`](https://www.youtube.com/watch?v=uSZWeRADTFI#uber){.link-annotated-not .include-content .width-full} ::: ## Sort By Magic ::: {demo-type="sort-by-magic-preface"} Annotations sorted by machine learning into [inferred \'tags\'](/design#future-tag-features){#gwern-design--future-tag-features .link-annotated-partial}. This provides an alternative way to browse: instead of by *date* order, one can browse in *topic* order. The \'sorted\' list has been automatically clustered into multiple sections & auto-labeled for easier browsing. Beginning with the newest annotation, it uses the embedding of each annotation to attempt to create a list of nearest-neighbor annotations, creating a progression of topics. For more details, see the link. ::: ### `source-seeking` {.collapse title="Machine-generated tag name for the following cluster of links."} [\[see previous entry\]](#duisterhof-et-al-2019-section){.include .include-even-when-collapsed} ### `intelligence-mapping` {.collapse title="Machine-generated tag name for the following cluster of links."} [\[see previous entry\]](#feilong-et-al-2021-section){.include .include-even-when-collapsed} [\[see previous entry\]](#lin-et-al-2026-section){.include .include-even-when-collapsed} ### `neural-coding` {.collapse title="Machine-generated tag name for the following cluster of links."} [\[see previous entry\]](#ananthanarayanan-et-al-2009-section){.include .include-even-when-collapsed} [\[see previous entry\]](#gonzalez-bellido-et-al-2013-section){.include .include-even-when-collapsed} ### `malware-injection` {.collapse title="Machine-generated tag name for the following cluster of links."} [\[see previous entry\]](#dietz-et-al-2026-section){.include .include-even-when-collapsed} [\[see previous entry\]](#wang-et-al-2021-evilmodel-section){.include .include-even-when-collapsed} ### `real-time-recommendation` {.collapse title="Machine-generated tag name for the following cluster of links."} [\[see previous entry\]](#yu-et-al-2023-2-section){.include .include-even-when-collapsed} [\[see previous entry\]](#bapna-et-al-2022-section){.include .include-even-when-collapsed} [\[see previous entry\]](#lian-et-al-2021-1-section){.include .include-even-when-collapsed} [\[see previous entry\]](#liu-et-al-2022-12-section){.include .include-even-when-collapsed} ### `architecture-search` {.collapse title="Machine-generated tag name for the following cluster of links."} [\[see previous entry\]](#hoffer-et-al-2018-section){.include .include-even-when-collapsed} [\[see previous entry\]](#an-et-al-2019-section){.include .include-even-when-collapsed} [\[see previous entry\]](#wei-et-al-2026-section){.include .include-even-when-collapsed} ### `scalability` {.collapse title="Machine-generated tag name for the following cluster of links."} [\[see previous entry\]](#amari-1989-section){.include .include-even-when-collapsed} [\[see previous entry\]](#uchizawa-et-al-2006-section){.include .include-even-when-collapsed} [\[see previous entry\]](#maass-1997-section){.include .include-even-when-collapsed} [\[see previous entry\]](#yamazaki-et-al-2022-section){.include .include-even-when-collapsed} [\[see previous entry\]](#stanojevic-et-al-2024-section){.include .include-even-when-collapsed} [\[see previous entry\]](#stanojevic-et-al-2023-section){.include .include-even-when-collapsed} [\[see previous entry\]](#wunderlich-pehle-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#adler-shavit-2024-section){.include .include-even-when-collapsed} [\[see previous entry\]](#li-et-al-2022-11-section){.include .include-even-when-collapsed} [\[see previous entry\]](#lee-et-al-2024-3-section){.include .include-even-when-collapsed} [\[see previous entry\]](#liu-et-al-2023-05-section){.include .include-even-when-collapsed} [\[see previous entry\]](#huang-et-al-2024-section){.include .include-even-when-collapsed} [\[see previous entry\]](#cheng-et-al-2026-section){.include .include-even-when-collapsed} [\[see previous entry\]](#yu-et-al-2022-5-section){.include .include-even-when-collapsed} [\[see previous entry\]](#belcak-wattenhofer-2023-1-section){.include .include-even-when-collapsed} [\[see previous entry\]](#belcak-wattenhofer-2023-2-section){.include .include-even-when-collapsed} [\[see previous entry\]](#petersen-et-al-2022-section){.include .include-even-when-collapsed} [\[see previous entry\]](#petersen-et-al-2024-section){.include .include-even-when-collapsed} [\[see previous entry\]](#madsen-johansen-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#nikankin-et-al-2024-section){.include .include-even-when-collapsed} [\[see previous entry\]](#morris-et-al-2026-section){.include .include-even-when-collapsed} [\[see previous entry\]](#wang-et-al-2025-section){.include .include-even-when-collapsed} [\[see previous entry\]](#lialin-et-al-2023-section){.include .include-even-when-collapsed} [\[see previous entry\]](#hu-et-al-2021-3-section){.include .include-even-when-collapsed} [\[see previous entry\]](#biderman-et-al-2024-section){.include .include-even-when-collapsed} [\[see previous entry\]](#shuttleworth-et-al-2024-section){.include .include-even-when-collapsed} [\[see previous entry\]](#kamalakara-et-al-2022-section){.include .include-even-when-collapsed} [\[see previous entry\]](#huh-et-al-2024-section){.include .include-even-when-collapsed} [\[see previous entry\]](#liu-et-al-2022-15-section){.include .include-even-when-collapsed} [\[see previous entry\]](#tan-le-2019-section){.include .include-even-when-collapsed} [\[see previous entry\]](#gholami-et-al-2018-section){.include .include-even-when-collapsed} [\[see previous entry\]](#hoefler-et-al-2021-section){.include .include-even-when-collapsed} [\[see previous entry\]](#lee-et-al-2023-5-section){.include .include-even-when-collapsed} [\[see previous entry\]](#dao-et-al-2022-2-section){.include .include-even-when-collapsed} [\[see previous entry\]](#poli-et-al-2024-section){.include .include-even-when-collapsed} [\[see previous entry\]](#krajewski-et-al-2024-section){.include .include-even-when-collapsed} [\[see previous entry\]](#guo-et-al-2025-section){.include .include-even-when-collapsed} [\[see previous entry\]](#zheng-et-al-2025-section){.include .include-even-when-collapsed} [\[see previous entry\]](#wu-et-al-2024-3-section){.include .include-even-when-collapsed} [\[see previous entry\]](#zaken-et-al-2021-section){.include .include-even-when-collapsed} [\[see previous entry\]](#aghajanyan-et-al-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#li-et-al-2018-1-section){.include .include-even-when-collapsed} [\[see previous entry\]](#maddox-et-al-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#wilson-izmailov-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#havasi-et-al-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#rosenfeld-tsotsos-2018-section){.include .include-even-when-collapsed} [\[see previous entry\]](#gaier-ha-2019-section){.include .include-even-when-collapsed} [\[see previous entry\]](#kaushik-et-al-2025-section){.include .include-even-when-collapsed} [\[see previous entry\]](#frankle-et-al-2019-1-section){.include .include-even-when-collapsed} [\[see previous entry\]](#greydanus-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#cuccu-et-al-2018-section){.include .include-even-when-collapsed} [\[see previous entry\]](#liu-et-al-2022-22-section){.include .include-even-when-collapsed} [\[see previous entry\]](#mukherjee-et-al-2025-section){.include .include-even-when-collapsed} [\[see previous entry\]](#dherin-et-al-2025-section){.include .include-even-when-collapsed} [\[see previous entry\]](#charakorn-et-al-2026-section){.include .include-even-when-collapsed} [\[see previous entry\]](#charakorn-et-al-2025-section){.include .include-even-when-collapsed} [\[see previous entry\]](#douillard-et-al-2023-section){.include .include-even-when-collapsed} [\[see previous entry\]](#qi-et-al-2023-3-section){.include .include-even-when-collapsed} [\[see previous entry\]](#ji-et-al-2021-section){.include .include-even-when-collapsed} [\[see previous entry\]](#sathyendra-et-al-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#botha-et-al-2017-section){.include .include-even-when-collapsed} [\[see previous entry\]](#press-wolf-2016-section){.include .include-even-when-collapsed} [\[see previous entry\]](#lazaridou-et-al-2021-page-7-org-deepmind-section){.include .include-even-when-collapsed} [\[see previous entry\]](#goel-et-al-2025-section){.include .include-even-when-collapsed} [\[see previous entry\]](#ilyas-et-al-2022-section){.include .include-even-when-collapsed} [\[see previous entry\]](#filan-et-al-2021-section){.include .include-even-when-collapsed} [\[see previous entry\]](#anonymous-2022-4-section){.include .include-even-when-collapsed} [\[see previous entry\]](#kligvasser-et-al-2017-section){.include .include-even-when-collapsed} [\[see previous entry\]](#he-et-al-2015-2-section){.include .include-even-when-collapsed} [\[see previous entry\]](#gastaldi-2017-section){.include .include-even-when-collapsed} [\[see previous entry\]](#villani-schoots-2023-section){.include .include-even-when-collapsed} [\[see previous entry\]](#roeder-et-al-2020-section){.include .include-even-when-collapsed} [\[see previous entry\]](#feldman-2019-section){.include .include-even-when-collapsed} [\[see previous entry\]](#lin-et-al-2025-section){.include .include-even-when-collapsed} [\[see previous entry\]](#luo-et-al-2023-2-section){.include .include-even-when-collapsed} [\[see previous entry\]](#jin-et-al-2024-3-section){.include .include-even-when-collapsed} [\[see previous entry\]](#petty-et-al-2023-section){.include .include-even-when-collapsed} [\[see previous entry\]](#murahari-et-al-2023-2-section){.include .include-even-when-collapsed} [\[see previous entry\]](#murahari-et-al-2023-1-section){.include .include-even-when-collapsed} [\[see previous entry\]](#cheung-et-al-2019-section){.include .include-even-when-collapsed} [\[see previous entry\]](#novikov-et-al-2015-section){.include .include-even-when-collapsed} [\[see previous entry\]](#wang-et-al-2018-6-section){.include .include-even-when-collapsed} [\[see previous entry\]](#ye-et-al-2017-section){.include .include-even-when-collapsed} [\[see previous entry\]](#graham-maaten-2017-section){.include .include-even-when-collapsed} [\[see previous entry\]](#graham-et-al-2017-1-section){.include .include-even-when-collapsed} [\[see previous entry\]](#zhang-et-al-2017-4-section){.include .include-even-when-collapsed} ## Wikipedia (2) {#titled-links-wikipedia .collapse} 1. **[`Autoencoder § Sparse autoencoder (SAE)`{=html}](https://en.wikipedia.org/wiki/Autoencoder#Sparse_autoencoder_(SAE) "Autoencoder § Sparse autoencoder (SAE)"){.id-not .include-annotation link-icon="wikipedia" link-icon-type="svg"}** ::: aux-links-transclude-file [`https://en.wikipedia.org/wiki/Autoencoder#Sparse_autoencoder_(SAE)`](https://en.wikipedia.org/wiki/Autoencoder#Sparse_autoencoder_(SAE) "Autoencoder § Sparse autoencoder (SAE)"){.id-not .include-content include-template="$annotationFileIncludeTemplate"} ::: 2. **[`Sparse distributed memory`{=html}](https://en.wikipedia.org/wiki/Sparse_distributed_memory "Sparse distributed memory"){.id-not .include-annotation link-icon="wikipedia" link-icon-type="svg"}** ::: aux-links-transclude-file [`https://en.wikipedia.org/wiki/Sparse_distributed_memory`](https://en.wikipedia.org/wiki/Sparse_distributed_memory "Sparse distributed memory"){.id-not .include-content include-template="$annotationFileIncludeTemplate"} ::: # Miscellaneous - **[`/doc/ai/nn/sparsity/2022-bapna-figure2-googletranslateneuralmachinetranslationscalingbylanguagecorpussize.jpg`{=html}](/doc/ai/nn/sparsity/2022-bapna-figure2-googletranslateneuralmachinetranslationscalingbylanguagecorpussize.jpg){.link-annotated-partial .id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** - **[`/doc/ai/nn/sparsity/2018-cheng.pdf`{=html}](/doc/ai/nn/sparsity/2018-cheng.pdf){.link-annotated-partial .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** - **[`/doc/ai/nn/cnn/2017-rawat.pdf`{=html}](/doc/ai/nn/cnn/2017-rawat.pdf){.link-annotated-partial .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** - **[`http://manikvarma.org/pubs/kusupati18.pdf`{=html}](http://manikvarma.org/pubs/kusupati18.pdf){.link-annotated-partial .id-not .include-annotation link-icon="pdf" link-icon-type="svg" link-icon-color="#f40f02"}** - **[`https://ai.meta.com/blog/a-highly-efficient-real-time-text-to-speech-system-deployed-on-cpus/`{=html}](https://ai.meta.com/blog/a-highly-efficient-real-time-text-to-speech-system-deployed-on-cpus/){.link-annotated-partial .id-not .include-annotation link-icon="facebook" link-icon-type="svg" link-icon-color="#1877f2"}** - **[`https://blog.floydhub.com/knowledge-distillation/`{=html}](https://blog.floydhub.com/knowledge-distillation/){.link-annotated-partial .id-not .include-annotation}** - **[`https://blog.roblox.com/2020/05/scaled-bert-serve-1-billion-daily-requests-cpus/`{=html}](https://blog.roblox.com/2020/05/scaled-bert-serve-1-billion-daily-requests-cpus/){.link-annotated-partial .id-not .include-annotation}** - **[`https://cprimozic.net/blog/growing-sparse-computational-graphs-with-rnns/`{=html}](https://cprimozic.net/blog/growing-sparse-computational-graphs-with-rnns/){.link-annotated-partial .id-not .include-annotation}** - **[`https://old.reddit.com/r/slatestarcodex/comments/1201v68/10word_quote_a_short_and_simple_failure_mode_of/jdjsx43/`{=html}](https://old.reddit.com/r/slatestarcodex/comments/1201v68/10word_quote_a_short_and_simple_failure_mode_of/jdjsx43/){.id-not .include-annotation link-icon="reddit" link-icon-type="svg" link-icon-color="#ff4500"}** - **[`https://openai.com/pricing#fine-tuning-models`{=html}](https://openai.com/pricing#fine-tuning-models){.link-annotated-partial .id-not .include-annotation link-icon="openai" link-icon-type="svg"}** - **[`https://research.google/blog/an-all-neural-on-device-speech-recognizer/`{=html}](https://research.google/blog/an-all-neural-on-device-speech-recognizer/){.id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** - **[`https://research.google/blog/auto-generated-summaries-in-google-docs/`{=html}](https://research.google/blog/auto-generated-summaries-in-google-docs/){.id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** - **[`https://research.google/blog/custom-on-device-ml-models-with-learn2compress/`{=html}](https://research.google/blog/custom-on-device-ml-models-with-learn2compress/){.id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** - **[`https://research.google/blog/efficient-sequence-modeling-for-on-device-ml/`{=html}](https://research.google/blog/efficient-sequence-modeling-for-on-device-ml/){.id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** - **[`https://research.google/blog/grammar-correction-as-you-type-on-pixel-6/`{=html}](https://research.google/blog/grammar-correction-as-you-type-on-pixel-6/){.id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** - **[`https://research.google/blog/training-machine-learning-models-more-efficiently-with-dataset-distillation/`{=html}](https://research.google/blog/training-machine-learning-models-more-efficiently-with-dataset-distillation/){.id-not .include-annotation link-icon="alphabet" link-icon-type="svg" link-icon-color="#4285f4"}** - **[`https://tech.pic-collage.com/distillation-of-clip-model-and-other-experiments-f8394b7321ce`{=html}](https://tech.pic-collage.com/distillation-of-clip-model-and-other-experiments-f8394b7321ce){.link-annotated-partial .id-not .include-annotation}** - **[`https://www.lesswrong.com/posts/7fxusXdkMNmAhkAfc/finding-sparse-linear-connections-between-features-in-llms`{=html}](https://www.lesswrong.com/posts/7fxusXdkMNmAhkAfc/finding-sparse-linear-connections-between-features-in-llms){.link-annotated-partial .id-not .include-annotation link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"}** - **[`https://www.lesswrong.com/posts/qmQFHCgCyEEjuy5a7/lora-fine-tuning-efficiently-undoes-safety-training-from`{=html}](https://www.lesswrong.com/posts/qmQFHCgCyEEjuy5a7/lora-fine-tuning-efficiently-undoes-safety-training-from){.link-annotated-partial .id-not .include-annotation link-icon="LW" link-icon-type="text" link-icon-color="#7faf83"}** - **[`https://www.quantamagazine.org/sparse-neural-networks-point-physicists-to-useful-data-20230608/`{=html}](https://www.quantamagazine.org/sparse-neural-networks-point-physicists-to-useful-data-20230608/){.link-annotated-partial .id-not .include-annotation link-icon="quanta" link-icon-type="svg" link-icon-color="#ff8600"}** - **[`https://www.reddit.com/r/LocalLLaMA/comments/18luk10/wait_llama_and_falcon_are_also_moe/`{=html}](https://www.reddit.com/r/LocalLLaMA/comments/18luk10/wait_llama_and_falcon_are_also_moe/){.id-not .include-annotation link-icon="reddit" link-icon-type="svg" link-icon-color="#ff4500"}** - **[`https://x.com/CFGeek/status/1826749739502895618`{=html}](https://x.com/CFGeek/status/1826749739502895618 "Foster 2026"){.id-not .include-annotation link-icon="twitter" link-icon-type="svg" link-icon-color="#1da1f2"}** ::: aux-links-transclude-file [`https://x.com/CFGeek/status/1826749739502895618`](https://x.com/CFGeek/status/1826749739502895618 "Foster 2026"){.id-not .include-content} ::: # Bibliography {#link-bibliography-section} 1. `https://arxiv.org/abs/2510.15103#facebook`: [`“Continual Learning via Sparse Memory Finetuning”`{=html}](https://arxiv.org/abs/2510.15103#facebook "'Continual Learning via Sparse Memory Finetuning', Lin et al 2025"){#lin-et-al-2025 .link-annotated}, [Jessy Lin, [Luke Zettlemoyer](https://scholar.google.com/citations?user=UjpbO6IAAAAJ "Luke Zettlemoyer"){.id-not}, Gargi Ghosh[, Wen-Tau Yih, Aram Markosyan, Vincent-Pierre Berges, Barlas Oğuz]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2510.15103%2523facebook.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2510.15103#facebook"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 2. `https://arxiv.org/abs/2506.06105`: [`“Text-To-LoRA (T2L): Instant Transformer Adaption”`{=html}](https://arxiv.org/abs/2506.06105 "'Text-to-LoRA (T2L): Instant Transformer Adaption', Charakorn et al 2025"){#charakorn-et-al-2025 .link-annotated}, [Rujikorn Charakorn, Edoardo Cetin, Yujin Tang, Robert Tjarko Lange]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2506.06105.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2506.06105"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 3. `https://arxiv.org/abs/2505.11711`: [`“Reinforcement Learning Finetunes Small Subnetworks in Large Language Models”`{=html}](https://arxiv.org/abs/2505.11711 "'Reinforcement Learning Finetunes Small Subnetworks in Large Language Models', Mukherjee et al 2025"){#mukherjee-et-al-2025 .link-annotated}, [Sagnik Mukherjee, [Lifan Yuan](https://en.wikipedia.org/wiki/Lifan_Yuan "Lifan Yuan"){.id-not}, Dilek Hakkani-Tur, Hao Peng]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2505.11711.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2505.11711"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 4. `https://arxiv.org/abs/2403.17844`: [`“Mechanistic Design and Scaling of Hybrid Architectures”`{=html}](https://arxiv.org/abs/2403.17844 "'Mechanistic Design and Scaling of Hybrid Architectures', Poli et al 2024"){#poli-et-al-2024 .link-annotated}, [Michael Poli, Armin W. Thomas, Eric Nguyen[, Pragaash Ponnusamy, Björn Deiseroth, [Kristian Kersting](https://en.wikipedia.org/wiki/Kristian_Kersting "Kristian Kersting"){.id-not}, Taiji Suzuki, Brian Hie, [Stefano Ermon](https://scholar.google.com/citations?user=ogXTOZ4AAAAJ "Stefano Ermon"){.id-not}, [Christopher Ré](https://en.wikipedia.org/wiki/Christopher_R%C3%A9 "Christopher Ré"){.id-not}, Ce Zhang, Stefano Massaroli]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2403.17844.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2403.17844"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 5. `https://arxiv.org/abs/2311.10770`: [`“Exponentially Faster Language Modeling”`{=html}](https://arxiv.org/abs/2311.10770 "'Exponentially Faster Language Modeling', Belcak & Wattenhofer 2023"){#belcak-wattenhofer-2023-1 .link-annotated}, [Peter Belcak, [Roger Wattenhofer](https://en.wikipedia.org/wiki/Roger_Wattenhofer "Roger Wattenhofer"){.id-not}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2311.10770.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2311.10770"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 6. `https://www.sciencedirect.com/science/article/pii/S0893608023005051`: [`“An Exact Mapping from ReLU Networks to Spiking Neural Networks”`{=html}](https://www.sciencedirect.com/science/article/pii/S0893608023005051 "'An exact mapping from ReLU networks to spiking neural networks', Stanojevic et al 2023"){#stanojevic-et-al-2023 .link-annotated}, [Ana Stanojevic, Stanisław Woźniak, Guillaume Bellec[, Giovanni Cherubini, Angeliki Pantazi, [Wulfram Gerstner](https://en.wikipedia.org/wiki/Wulfram_Gerstner "Wulfram Gerstner"){.id-not}]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fwww.sciencedirect.com%252Fscience%252Farticle%252Fpii%252FS0893608023005051.html "Directory-tag link-bibliography for link https://www.sciencedirect.com/science/article/pii/S0893608023005051"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 7. `https://arxiv.org/abs/2310.17157`: [`“Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time”`{=html}](https://arxiv.org/abs/2310.17157 "'Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time', Liu et al 2023"){#liu-et-al-2023-05 .link-annotated}, [Zichang Liu, [Jue Wang](https://en.wikipedia.org/wiki/Jue_Wang "Jue Wang"){.id-not}, [Tri Dao](https://tridao.me/ "Tri Dao"){.link-annotated-partial .id-not}[, Tianyi Zhou, Binhang Yuan, [Zhao Song](https://en.wikipedia.org/wiki/Zhao_Song "Zhao Song"){.id-not}, Anshumali Shrivastava, Ce Zhang, [Yuandong Tian](https://yuandong-tian.com/ "Yuandong Tian’s webpage"){.link-annotated-partial .id-not}, [Christopher Re](https://en.wikipedia.org/wiki/Christopher_Re "Christopher Re"){.id-not}, Beidi Chen]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2310.17157.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2310.17157"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 8. `https://arxiv.org/abs/2308.14711`: [`“Fast Feedforward Networks”`{=html}](https://arxiv.org/abs/2308.14711 "'Fast Feedforward Networks', Belcak & Wattenhofer 2023"){#belcak-wattenhofer-2023-2 .link-annotated}, [Peter Belcak, [Roger Wattenhofer](https://en.wikipedia.org/wiki/Roger_Wattenhofer "Roger Wattenhofer"){.id-not}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2308.14711.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2308.14711"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 9. `https://arxiv.org/abs/2302.12441`: [`“MUX-PLMs: Pre-Training Language Models With Data Multiplexing”`{=html}](https://arxiv.org/abs/2302.12441 "'MUX-PLMs: Pre-training Language Models with Data Multiplexing', Murahari et al 2023"){#murahari-et-al-2023-2 .link-annotated}, [Vishvak Murahari, Ameet Deshpande, Carlos E. Jimenez[, Izhak Shafran, Mingqiu Wang, [Yuan Cao](https://en.wikipedia.org/wiki/Yuan_Cao "Yuan Cao"){.id-not}, [Karthik Rajagopal Narasimhan](https://karthikncode.github.io/ "'Karthik R. Narasimhan homepage', Narasimhan 2026"){.id-not}]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2302.12441.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2302.12441"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 10. `https://arxiv.org/abs/2210.06313#google`: [`“The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers”`{=html}](https://arxiv.org/abs/2210.06313#google "'The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers', Li et al 2022"){#li-et-al-2022-11 .link-annotated}, [Zonglin Li, Chong You, Srinadh Bhojanapalli[, Daliang Li, Ankit Singh Rawat, Sashank J. Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, [Sanjiv Kumar](https://scholar.google.com/citations?user=08CNqrYAAAAJ&view_op=list_works&sortby=pubdate "Sanjiv Kumar"){.id-not}]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2210.06313%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2210.06313#google"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 11. `https://arxiv.org/abs/2207.03620`: [`“More ConvNets in the 2020s: Scaling up Kernels Beyond 51×51 Using Sparsity (SLaK)”`{=html}](https://arxiv.org/abs/2207.03620 "'More ConvNets in the 2020s: Scaling up Kernels Beyond 51×51 using Sparsity (SLaK)', Liu et al 2022"){#liu-et-al-2022-15 .link-annotated}, [Shiwei Liu, Tianlong Chen, Xiaohan Chen[, Xuxi Chen, Qiao Xiao, Boqian Wu, Mykola Pechenizkiy, Decebal Mocanu, Zhangyang Wang]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2207.03620.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2207.03620"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 12. `https://arxiv.org/abs/2205.03983#google`: [`“Building Machine Translation Systems for the Next Thousand Languages”`{=html}](https://arxiv.org/abs/2205.03983#google "'Building Machine Translation Systems for the Next Thousand Languages', Bapna et al 2022"){#bapna-et-al-2022 .link-annotated}, [Ankur Bapna, Isaac Caswell, Julia Kreutzer[, [Orhan Firat](https://orhanfirat.com/ "Orhan Fırat"){.link-annotated-partial .id-not}, Daan van Esch, Aditya Siddhant, Mengmeng Niu, Pallavi Baljekar, Xavier Garcia, Wolfgang Macherey, Theresa Breiner, Vera Axelrod, Jason Riesa, [Yuan Cao](https://en.wikipedia.org/wiki/Yuan_Cao "Yuan Cao"){.id-not}, Mia Xu Chen, Klaus Macherey, Maxim Krikun, Pidong Wang, Alexander Gutkin, Apurva Shah, [Yanping Huang](https://scholar.google.com/citations?user=uEtBQScAAAAJ "Yanping Huang"){.id-not}, [Zhifeng Chen](https://scholar.google.com/citations?user=EemUE4gAAAAJ "Zhifeng Chen"){.id-not}, [Yonghui Wu](https://scholar.google.com/citations?user=bMWMlk4AAAAJ "Yonghui Wu"){.id-not}, Macduff Hughes]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2205.03983%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2205.03983#google"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 13. `https://arxiv.org/abs/2204.00595`: [`“Monarch: Expressive Structured Matrices for Efficient and Accurate Training”`{=html}](https://arxiv.org/abs/2204.00595 "'Monarch: Expressive Structured Matrices for Efficient and Accurate Training', Dao et al 2022"){#dao-et-al-2022-2 .link-annotated}, [[Tri Dao](https://tridao.me/ "Tri Dao"){.link-annotated-partial .id-not}, Beidi Chen, Nimit Sohoni[, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, [Atri Rudra](https://cse.buffalo.edu/faculty/atri/ "Atri Rudra"){.id-not}, [Christopher Ré](https://en.wikipedia.org/wiki/Christopher_R%C3%A9 "Christopher Ré"){.id-not}]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2204.00595.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2204.00595"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 14. `https://arxiv.org/abs/2203.06850`: [`“Efficient Language Modeling With Sparse All-MLP”`{=html}](https://arxiv.org/abs/2203.06850 "'Efficient Language Modeling with Sparse All-MLP', Yu et al 2022"){#yu-et-al-2022-5 .link-annotated}, [Ping Yu, [Mikel Artetxe](https://en.wikipedia.org/wiki/Mikel_Artetxe "Mikel Artetxe"){.id-not}, Myle Ott[, Sam Shleifer, Hongyu Gong, Ves Stoyanov, Xian Li]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2203.06850.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2203.06850"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 15. `https://arxiv.org/abs/2202.07415#deepmind`: [`“NeuPL: Neural Population Learning”`{=html}](https://arxiv.org/abs/2202.07415#deepmind "'NeuPL: Neural Population Learning', Liu et al 2022"){#liu-et-al-2022-22 .link-annotated}, [Siqi Liu, Luke Marris, Daniel Hennes[, Josh Merel, [Nicolas Heess](https://scholar.google.com/citations?user=79k7bGEAAAAJ "Nicolas Heess"){.id-not}, [Thore Graepel](https://thoregraepel.github.io/ "'Thore Graepel', Graepel 2026"){.link-annotated-partial .id-not}]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2202.07415%2523deepmind.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2202.07415#deepmind"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 16. `https://arxiv.org/abs/2106.10199`: [`“BitFit: Simple Parameter-Efficient Fine-Tuning for Transformer-Based Masked Language-Models”`{=html}](https://arxiv.org/abs/2106.10199 "'BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models', Zaken et al 2021"){#zaken-et-al-2021 .link-annotated}, [Elad Ben Zaken, Shauli Ravfogel, [Yoav Goldberg](https://scholar.google.com/citations?user=0rskDKgAAAAJ "Yoav Goldberg"){.id-not}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2106.10199.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2106.10199"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 17. `https://arxiv.org/abs/2106.09685#microsoft`: [`“LoRA: Low-Rank Adaptation of Large Language Models”`{=html}](https://arxiv.org/abs/2106.09685#microsoft "'LoRA: Low-Rank Adaptation of Large Language Models', Hu et al 2021"){#hu-et-al-2021-3 .link-annotated}, [Edward J. Hu, Yelong Shen, Phillip Wallis[, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, [Weizhu Chen](https://scholar.google.com/citations?user=LG_E-4EAAAAJ "Weizhu Chen"){.id-not}]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2106.09685%2523microsoft.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2106.09685#microsoft"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 18. `https://arxiv.org/pdf/2102.01951#page=7&org=deepmind`: [`“Mind the Gap: Assessing Temporal Generalization in Neural Language Models § Dynamic Evaluation”`{=html}](https://arxiv.org/pdf/2102.01951#page=7&org=deepmind "'Mind the Gap: Assessing Temporal Generalization in Neural Language Models § Dynamic Evaluation', Lazaridou et al 2021 (page 7 org deepmind)"){#lazaridou-et-al-2021-page-7-org-deepmind .link-annotated}, [Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya[, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d'Autume, Tomas Kocisky, Sebastian Ruder, [Dani Yogatama](https://dyogatama.github.io/ "Dani Yogatama"){.link-annotated-partial .id-not}, Kris Cao, Susannah Young, Phil Blunsom]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fpdf%252F2102.01951%2523page%253D7%2526org%253Ddeepmind.html "Directory-tag link-bibliography for link https://arxiv.org/pdf/2102.01951#page=7&org=deepmind"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 19. `https://arxiv.org/abs/1912.05671`: [`“Linear Mode Connectivity and the Lottery Ticket Hypothesis”`{=html}](https://arxiv.org/abs/1912.05671 "'Linear Mode Connectivity and the Lottery Ticket Hypothesis', Frankle et al 2019"){#frankle-et-al-2019-1 .link-annotated}, [[Jonathan Frankle](http://www.jfrankle.com/ "Jonathan Frankle—Chief Neural network Scientist at Databricks"){.link-annotated-partial .id-not}, Gintare Karolina Dziugaite, Daniel M. Roy, [Michael Carbin](https://people.csail.mit.edu/mcarbin/ "Michael Carbin"){.link-annotated-partial .id-not}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F1912.05671.html "Directory-tag link-bibliography for link https://arxiv.org/abs/1912.05671"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 20. `https://arxiv.org/abs/1905.11946#google`: [`“EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks”`{=html}](https://arxiv.org/abs/1905.11946#google "'EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks', Tan & Le 2019"){#tan-le-2019 .link-annotated}, [Mingxing Tan, [Quoc V. Le](https://en.wikipedia.org/wiki/Quoc_V._Le "Quoc V. Le"){.id-not}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F1905.11946%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/1905.11946#google"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 21. `https://arxiv.org/abs/1803.10615`: [`“SqueezeNext: Hardware-Aware Neural Network Design”`{=html}](https://arxiv.org/abs/1803.10615 "'SqueezeNext: Hardware-Aware Neural Network Design', Gholami et al 2018"){#gholami-et-al-2018 .link-annotated}, [Amir Gholami, Kiseok Kwon, Bichen Wu[, Zizheng Tai, Xiangyu Yue, Peter Jin, Sicheng Zhao, [Kurt Keutzer](https://en.wikipedia.org/wiki/Kurt_Keutzer "Kurt Keutzer"){.id-not}]{.collapse}]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F1803.10615.html "Directory-tag link-bibliography for link https://arxiv.org/abs/1803.10615"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block} 22. `https://arxiv.org/abs/1512.03385#microsoft`: [`“Deep Residual Learning for Image Recognition”`{=html}](https://arxiv.org/abs/1512.03385#microsoft "'Deep Residual Learning for Image Recognition', He et al 2015"){#he-et-al-2015-2 .link-annotated}, [[Kaiming He](https://en.wikipedia.org/wiki/Kaiming_He "Kaiming He"){.id-not}, Xiangyu Zhang, Shaoqing Ren, Jian Sun]{.author} [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F1512.03385%2523microsoft.html "Directory-tag link-bibliography for link https://arxiv.org/abs/1512.03385#microsoft"){.id-not .include-even-when-collapsed}]{.collapse .tag-index-link-bibliography-block}