---
title: '‘MLP NN’ directory'
description: "Bibliography for directory <code>ai/nn/fully-connected</code>, most recent first: 2 <a class='icon-not' href='/doc/ai/nn/fully-connected/index#see-alsos'>related tags</a>, 248 <a class='icon-not' href='/doc/ai/nn/fully-connected/index#links'>annotations</a>, & 49 <a class='icon-not' href='/doc/ai/nn/fully-connected/index#miscellaneous'>links</a> (<a href='/doc/ai/nn/index' class='link-page link-tag directory-indexes-upwards link-annotated' data-link-icon='arrow-up-left' data-link-icon-type='svg' rel='tag' title='Link to parent directory'>parent</a>)."
thumbnail: /doc/ai/nn/fully-connected/2025-polaczek-figure3-neuralsvgarchitectureofmlppredictingoutputspershape.jpg
thumbnail-text: ''
thumbnail-css: "outline"
created: 2019-08-31
modified: 2026-07-22
status: in progress
previous: /doc/ai/nn/dynamic-evaluation/index
next: /doc/ai/nn/gan/index
confidence: log
importance: 0
css-extension: dropcaps-not
index: True
backlink: False
...

::: {#manual-annotation .abstract-small .abstract-tag-directory}
[\[page summary\]](/doc/ai/nn/fully-connected/abstract "Transclude link for doc/ai/nn/fully-connected/ tag-abstract page."){#gwern-doc-ai-nn-fully-connected-abstract
.link-annotated-partial .include-content-core .include-strict
.link-page}
:::

# See Also

::: {#see-alsos .directory-indexes .columns}
-   [Parent ('*`NN`{=html}*' tag)](/doc/ai/nn/index "Link to parent directory 'doc/ai/nn/' (ascending)"){#_ujmXzDuq
    .link-annotated .link-tag .directory-indexes-upwards rel="tag"}

-   [*`RNN`{=html}*](/doc/ai/nn/rnn/index "‘RNN’ directory"){#_BHyr1fFu
    .link-annotated .link-tag .directory-indexes-sideways rel="tag"}

-   [*`Transformer`{=html}*](/doc/ai/nn/transformer/index "‘Transformer’ directory"){#_ojWSXRFs
    .link-annotated .link-tag .directory-indexes-sideways rel="tag"}
:::

# Gwern {.display-pop-not}

## `“Dat Tail, Dat Flank—Never Forget”, Gwern 2025`{=html} {#gwern-2025-section}

**[`Dat Tail, Dat Flank—Never Forget`{=html}](/doc/fiction/science-fiction/2025-05-01-gwern-gpto3-irememberlemuria-arltailandflank.png "'Dat Tail, Dat Flank—Never Forget', Gwern 2025"){.link-annotated-partial
.id-not .include-annotation link-icon="image" link-icon-type="svg"}**

## `“Absolute Unit NNs: Regression-Based MLPs for Everything”, Gwern 2023`{=html} {#gwern-aunn-section}

**[`Absolute Unit NNs: Regression-Based MLPs for Everything`{=html}](/aunn "'Absolute Unit NNs: Regression-Based MLPs for Everything', Gwern 2023"){.link-annotated
.id-not .include-annotation}**

## `“Research Ideas”, Gwern 2017`{=html} {#gwern-idea-section}

**[`Research Ideas`{=html}](/idea "'Research Ideas', Gwern 2017"){.link-annotated
.id-not .include-annotation}**

## `“Modular Brain AUNNs for Uploads”, Gwern 2023`{=html} {#gwern-aunn-brain-section}

**[`Modular Brain AUNNs for Uploads`{=html}](/aunn-brain "'Modular Brain AUNNs for Uploads', Gwern 2023"){.link-annotated
.id-not .include-annotation}**

## `“Language-Conditioned Absolute Unit NNs”, Gwern 2022`{=html} {#gwern-aunn-papyrus-section}

**[`Language-Conditioned Absolute Unit NNs`{=html}](/aunn-papyrus "'Language-Conditioned Absolute Unit NNs', Gwern 2022"){.link-annotated
.id-not .include-annotation}**

# Links {.display-pop-not}

## `“How Do Neural Networks Regress Cubic Polynomials? Apparently, They Use a Trick Invented in Milan 500 Years Ago”, Bottazzi 2026`{=html} {#bottazzi-2026-section}

**[`How do neural networks regress cubic polynomials? Apparently, they use a trick invented in Milan 500 years ago`{=html}](https://www.lesswrong.com/posts/ysztF7doGvEbvTMqN/how-do-neural-networks-regress-cubic-polynomials-apparently "'How do neural networks regress cubic polynomials? Apparently, they use a trick invented in Milan 500 years ago', Bottazzi 2026"){.link-annotated-partial
.id-not .include-annotation link-icon="LW" link-icon-type="text"
link-icon-color="#7faf83"}**

## `“MLPs Are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers”, Garcia et al 2026`{=html} {#garcia-et-al-2026-section}

**[`MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers`{=html}](https://arxiv.org/abs/2607.10034 "'MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers', Garcia et al 2026"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Neural Networks Learn Bloom Filters”`{=html} {#_fgzA2_VD-section}

**[`Neural Networks learn Bloom Filters`{=html}](https://www.lesswrong.com/posts/buxBdp8NtHGgBwabv/neural-networks-learn-bloom-filters "Neural Networks learn Bloom Filters"){.link-annotated-partial
.id-not .include-annotation link-icon="LW" link-icon-type="text"
link-icon-color="#7faf83"}**

## `“Every Feedforward Neural Network Definable in an O-Minimal Structure Has Finite Sample Complexity”, Kratsios et al 2026`{=html} {#kratsios-et-al-2026-section}

**[`Every Feedforward Neural Network Definable in an o-Minimal Structure Has Finite Sample Complexity`{=html}](https://arxiv.org/abs/2605.07097 "'Every Feedforward Neural Network Definable in an o-Minimal Structure Has Finite Sample Complexity', Kratsios et al 2026"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Friendship Is All You Need: Subliminal Pony Propagation in Large Language Models, Or, How I Learned to Stop Worrying and Love the Sparkle”, Sparkle et al 2026`{=html} {#sparkle-et-al-2026-section}

**[`Friendship Is All You Need: Subliminal Pony Propagation in Large Language Models, or, How I Learned to Stop Worrying and Love the Sparkle`{=html}](https://weavers.neocities.org/mlp-subliminal/paper "'Friendship Is All You Need: Subliminal Pony Propagation in Large Language Models, or, How I Learned to Stop Worrying and Love the Sparkle', Sparkle et al 2026"){.link-annotated-partial
.id-not .include-annotation}**

## `“Shared Sensitivity to Data Distribution during Learning in Humans and Transformer Networks”, Lerousseau & Summerfield 2025`{=html} {#lerousseau-summerfield-2025-section}

**[`Shared sensitivity to data distribution during learning in humans and Transformer networks`{=html}](https://www.nature.com/articles/s41562-025-02359-3 "'Shared sensitivity to data distribution during learning in humans and Transformer networks', Lerousseau & Summerfield 2025"){.link-annotated
.id-not .include-annotation link-icon="n" link-icon-type="text"}**

## `“Constructing Efficient Fact-Storing MLPs for Transformers”, Dugan et al 2025`{=html} {#dugan-et-al-2025-section}

**[`Constructing Efficient Fact-Storing MLPs for Transformers`{=html}](https://arxiv.org/abs/2512.00207 "'Constructing Efficient Fact-Storing MLPs for Transformers', Dugan et al 2025"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Learning without Training: The Implicit Dynamics of In-Context Learning”, Dherin et al 2025`{=html} {#dherin-et-al-2025-section}

**[`Learning without training: The implicit dynamics of in-context learning`{=html}](https://arxiv.org/abs/2507.16003#google "'Learning without training: The implicit dynamics of in-context learning', Dherin et al 2025"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning”, Shinnick et al 2025`{=html} {#shinnick-et-al-2025-section}

**[`Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning`{=html}](https://arxiv.org/abs/2505.22308 "'Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning', Shinnick et al 2025"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Compiling a Differentiable Logic Gate Neural Net to C for a 1,744× Speedup”, Clayton 2025`{=html} {#clayton-2025-section}

**[`Compiling a Differentiable Logic Gate Neural Net to C for a 1,744× speedup`{=html}](https://slightknack.dev/blog/difflogic/ "'Compiling a Differentiable Logic Gate Neural Net to C for a 1,744× speedup', Clayton 2025"){.link-annotated-partial
.id-not .include-annotation}**

## `“<em>μ</em>PC: Scaling Predictive Coding to 100+ Layer Networks”, Innocenti et al 2025`{=html} {#innocenti-et-al-2025-section}

**[`<em>μ</em>PC: Scaling Predictive Coding to 100+ Layer Networks`{=html}](https://arxiv.org/abs/2505.13124 "'<em>μ</em>PC: Scaling Predictive Coding to 100+ Layer Networks', Innocenti et al 2025"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models”, Chang & Bergen 2025`{=html} {#chang-bergen-2025-section}

**[`Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models`{=html}](https://arxiv.org/abs/2504.15471 "'Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models', Chang & Bergen 2025"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Progress Report on a Toy Model Of Memorization”, Brave 2025`{=html} {#brave-2025-section}

**[`Progress report on a Toy Model Of Memorization`{=html}](https://www.boristhebrave.com/2025/04/11/progress-report-on-a-toy-model-of-memorization/ "'Progress report on a Toy Model Of Memorization', Brave 2025"){.link-annotated-partial
.id-not .include-annotation}**

## `“NeuRa<span class="logotype-latex">L<span class="logotype-latex-a">a</span>T<span class="logotype-latex-e">e</span>X</span>: A Machine Learning Library Written in Pure <span class="logotype-latex">L<span class="logotype-latex-a">a</span>T<span class="logotype-latex-e">e</span>X</span>”, Gardner et al 2025`{=html} {#gardner-et-al-2025-section}

**[`NeuRa<span class="logotype-latex">L<span class="logotype-latex-a">a</span>T<span class="logotype-latex-e">e</span>X</span>: A machine learning library written in pure <span class="logotype-latex">L<span class="logotype-latex-a">a</span>T<span class="logotype-latex-e">e</span>X</span>`{=html}](https://arxiv.org/abs/2503.24187 "'NeuRa<span class="logotype-latex">L<span class="logotype-latex-a">a</span>T<span class="logotype-latex-e">e</span>X</span>: A machine learning library written in pure <span class="logotype-latex">L<span class="logotype-latex-a">a</span>T<span class="logotype-latex-e">e</span>X</span>', Gardner et al 2025"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“NeuralSVG: An Implicit Representation for Text-To-Vector Generation”, Polaczek et al 2025`{=html} {#polaczek-et-al-2025-section}

**[`NeuralSVG: An Implicit Representation for Text-to-Vector Generation`{=html}](https://arxiv.org/abs/2501.03992 "'NeuralSVG: An Implicit Representation for Text-to-Vector Generation', Polaczek et al 2025"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Titans: Learning to Memorize at Test Time”, Behrouz et al 2024`{=html} {#behrouz-et-al-2024-section}

**[`Titans: Learning to Memorize at Test Time`{=html}](https://arxiv.org/abs/2501.00663#google "'Titans: Learning to Memorize at Test Time', Behrouz et al 2024"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“AUNN: Simple Implementation of Gwern’s AUNN Proposal”, Roland 2024`{=html} {#roland-2024-section}

**[`AUNN: Simple implementation of Gwern’s AUNN proposal`{=html}](https://github.com/ethan-w-roland/AUNN "'AUNN: Simple implementation of Gwern’s AUNN proposal', Roland 2024"){.id-not
.include-annotation link-icon="github" link-icon-type="svg"}**

## `“Flexible Task Abstractions Emerge in Linear Networks With Fast and Bounded Units”, Sandbrink et al 2024`{=html} {#sandbrink-et-al-2024-section}

**[`Flexible task abstractions emerge in linear networks with fast and bounded units`{=html}](https://arxiv.org/abs/2411.03840 "'Flexible task abstractions emerge in linear networks with fast and bounded units', Sandbrink et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“The Slingshot Helps With Learning”, Wu 2024`{=html} {#wu-2024-section}

**[`The slingshot helps with learning`{=html}](https://www.lesswrong.com/posts/LncYobrn3vRr7qkZW/the-slingshot-helps-with-learning "'The slingshot helps with learning', Wu 2024"){.link-annotated
.id-not .include-annotation link-icon="LW" link-icon-type="text"
link-icon-color="#7faf83"}**

## `“Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics”, Nikankin et al 2024`{=html} {#nikankin-et-al-2024-section}

**[`Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics`{=html}](https://arxiv.org/abs/2410.21272 "'Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics', Nikankin et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning”, Lee et al 2024`{=html} {#lee-et-al-2024-5-section}

**[`SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning`{=html}](https://arxiv.org/abs/2410.09754 "'SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning', Lee et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Bilinear MLPs Enable Weight-Based Mechanistic Interpretability”, Pearce et al 2024`{=html} {#pearce-et-al-2024-section}

**[`Bilinear MLPs enable weight-based mechanistic interpretability`{=html}](https://arxiv.org/abs/2410.08417 "'Bilinear MLPs enable weight-based mechanistic interpretability', Pearce et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“NGPT: Normalized Transformer With Representation Learning on the Hypersphere”, Loshchilov et al 2024`{=html} {#loshchilov-et-al-2024-section}

**[`nGPT: Normalized Transformer with Representation Learning on the Hypersphere`{=html}](https://arxiv.org/abs/2410.01131 "'nGPT: Normalized Transformer with Representation Learning on the Hypersphere', Loshchilov et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“How Feature Learning Can Improve Neural Scaling Laws”, Bordelon et al 2024`{=html} {#bordelon-et-al-2024-section}

**[`How Feature Learning Can Improve Neural Scaling Laws`{=html}](https://arxiv.org/abs/2409.17858 "'How Feature Learning Can Improve Neural Scaling Laws', Bordelon et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Magika: AI-Powered Content-Type Detection”, Fratantonio et al 2024`{=html} {#fratantonio-et-al-2024-section}

**[`Magika: AI-Powered Content-Type Detection`{=html}](https://arxiv.org/abs/2409.13768#google "'Magika: AI-Powered Content-Type Detection', Fratantonio et al 2024"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“On the Complexity of Neural Computation in Superposition”, Adler & Shavit 2024`{=html} {#adler-shavit-2024-section}

**[`On the Complexity of Neural Computation in Superposition`{=html}](https://arxiv.org/abs/2409.15318 "'On the Complexity of Neural Computation in Superposition', Adler & Shavit 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Masked Mixers for Language Generation and Retrieval”, Badger 2024`{=html} {#badger-2024-section}

**[`Masked Mixers for Language Generation and Retrieval`{=html}](https://arxiv.org/abs/2409.01482 "'Masked Mixers for Language Generation and Retrieval', Badger 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“GSoC 2024: Differentiable Logic for Interactive Systems and Generative Music”`{=html} {#_u_ekCs7N-section}

**[`GSoC 2024: Differentiable Logic for Interactive Systems and Generative Music`{=html}](https://ijc8.me/2024/08/26/gsoc-difflogic/ "GSoC 2024: Differentiable Logic for Interactive Systems and Generative Music"){.link-annotated-partial
.id-not .include-annotation}**

## `“Mathematical Models of Computation in Superposition”, Hänni et al 2024`{=html} {#hänni-et-al-2024-section}

**[`Mathematical Models of Computation in Superposition`{=html}](https://arxiv.org/abs/2408.05451 "'Mathematical Models of Computation in Superposition', Hänni et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“PEER: Mixture of A Million Experts”, He 2024`{=html} {#he-2024-section}

**[`PEER: Mixture of A Million Experts`{=html}](https://arxiv.org/abs/2407.04153#deepmind "'PEER: Mixture of A Million Experts', He 2024"){.link-annotated
.id-not .include-annotation link-icon="deepmind" link-icon-type="svg"
link-icon-color="#4185f4"}**

## `“What Matters in Transformers? Not All Attention Is Needed”, He et al 2024`{=html} {#he-et-al-2024-section}

**[`What Matters in Transformers? Not All Attention is Needed`{=html}](https://arxiv.org/abs/2406.15786 "'What Matters in Transformers? Not All Attention is Needed', He et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“When Parts Are Greater Than Sums: Individual LLM Components Can Outperform Full Models”, Chang et al 2024`{=html} {#chang-et-al-2024-1-section}

**[`When Parts are Greater Than Sums: Individual LLM Components Can Outperform Full Models`{=html}](https://arxiv.org/abs/2406.13131 "'When Parts are Greater Than Sums: Individual LLM Components Can Outperform Full Models', Chang et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Probing the Decision Boundaries of In-Context Learning in Large Language Models”, Zhao et al 2024`{=html} {#zhao-et-al-2024-2-section}

**[`Probing the Decision Boundaries of In-context Learning in Large Language Models`{=html}](https://arxiv.org/abs/2406.11233 "'Probing the Decision Boundaries of In-context Learning in Large Language Models', Zhao et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“MAR: Autoregressive Image Generation without Vector Quantization”, Li et al 2024`{=html} {#li-et-al-2024-02-section}

**[`MAR: Autoregressive Image Generation without Vector Quantization`{=html}](https://arxiv.org/abs/2406.11838 "'MAR: Autoregressive Image Generation without Vector Quantization', Li et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Grokking Modular Polynomials”, Doshi et al 2024`{=html} {#doshi-et-al-2024-1-section}

**[`Grokking Modular Polynomials`{=html}](https://arxiv.org/abs/2406.03495 "'Grokking Modular Polynomials', Doshi et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Grokfast: Accelerated Grokking by Amplifying Slow Gradients”, Lee et al 2024`{=html} {#lee-et-al-2024-2-section}

**[`Grokfast: Accelerated Grokking by Amplifying Slow Gradients`{=html}](https://arxiv.org/abs/2405.20233 "'Grokfast: Accelerated Grokking by Amplifying Slow Gradients', Lee et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Lateralization MLP: A Simple Brain-Inspired Architecture for Diffusion”, Hu & Rostami 2024`{=html} {#hu-rostami-2024-section}

**[`Lateralization MLP: A Simple Brain-inspired Architecture for Diffusion`{=html}](https://arxiv.org/abs/2405.16098 "'Lateralization MLP: A Simple Brain-inspired Architecture for Diffusion', Hu & Rostami 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Bigger, Regularized, Optimistic: Scaling for Compute and Sample-Efficient Continuous Control”, Nauman et al 2024`{=html} {#nauman-et-al-2024-section}

**[`Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control`{=html}](https://arxiv.org/abs/2405.16158 "'Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control', Nauman et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“MLPs Learn In-Context”, Tong & Pehlevan 2024`{=html} {#tong-pehlevan-2024-section}

**[`MLPs Learn In-Context`{=html}](https://arxiv.org/abs/2405.15618 "'MLPs Learn In-Context', Tong & Pehlevan 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Verified Neural Compressed Sensing”, Bunel et al 2024`{=html} {#bunel-et-al-2024-section}

**[`Verified Neural Compressed Sensing`{=html}](https://arxiv.org/abs/2405.04260#deepmind "'Verified Neural Compressed Sensing', Bunel et al 2024"){.link-annotated
.id-not .include-annotation link-icon="deepmind" link-icon-type="svg"
link-icon-color="#4185f4"}**

## `“An Exactly Solvable Model for Emergence and Scaling Laws in the Multitask Sparse Parity Problem”, Nam et al 2024`{=html} {#nam-et-al-2024-section}

**[`An exactly solvable model for emergence and scaling laws in the multitask sparse parity problem`{=html}](https://arxiv.org/abs/2404.17563 "'An exactly solvable model for emergence and scaling laws in the multitask sparse parity problem', Nam et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Neural Redshift: Random Networks Are Not Random Functions”, Teney et al 2024`{=html} {#teney-et-al-2024-section}

**[`Neural Redshift: Random Networks are not Random Functions`{=html}](https://arxiv.org/abs/2403.02241 "'Neural Redshift: Random Networks are not Random Functions', Teney et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Verifiable Evaluations of Machine Learning Models Using ZkSNARKs”, South et al 2024`{=html} {#south-et-al-2024-section}

**[`Verifiable evaluations of machine learning models using zkSNARKs`{=html}](https://arxiv.org/abs/2402.02675 "'Verifiable evaluations of machine learning models using zkSNARKs', South et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Surfing the OCEAN: The Machine Learning Psycholexical Approach 2.0 to Detect Personality Traits in Texts”, Giannini et al 2024`{=html} {#giannini-et-al-2024-section}

**[`Surfing the OCEAN: The machine learning psycholexical approach 2.0 to detect personality traits in texts`{=html}](/doc/psychology/personality/2024-giannini.pdf "'Surfing the OCEAN: The machine learning psycholexical approach 2.0 to detect personality traits in texts', Giannini et al 2024"){.link-annotated
.id-not .include-annotation link-icon="pdf" link-icon-type="svg"
link-icon-color="#f40f02"}**

## `“Neural Spline Fields for Burst Image Fusion and Layer Separation”, Chugunov et al 2023`{=html} {#chugunov-et-al-2023-section}

**[`Neural Spline Fields for Burst Image Fusion and Layer Separation`{=html}](https://arxiv.org/abs/2312.14235 "'Neural Spline Fields for Burst Image Fusion and Layer Separation', Chugunov et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“SwitchHead: Accelerating Transformers With Mixture-Of-Experts Attention”, Csordás et al 2023`{=html} {#csordás-et-al-2023-section}

**[`SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention`{=html}](https://arxiv.org/abs/2312.07987#schmidhuber "'SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention', Csordás et al 2023"){.link-annotated
.id-not .include-annotation link-icon="SMDH"
link-icon-type="text,quad,sans"}**

## `“SMERF: Streamable Memory Efficient Radiance Fields for Real-Time Large-Scene Exploration”, Duckworth et al 2023`{=html} {#duckworth-et-al-2023-section}

**[`SMERF: Streamable Memory Efficient Radiance Fields for Real-Time Large-Scene Exploration`{=html}](https://arxiv.org/abs/2312.07541#google "'SMERF: Streamable Memory Efficient Radiance Fields for Real-Time Large-Scene Exploration', Duckworth et al 2023"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Grokking Group Multiplication With Cosets”, Stander et al 2023`{=html} {#stander-et-al-2023-section}

**[`Grokking Group Multiplication with Cosets`{=html}](https://arxiv.org/abs/2312.06581#eleutherai "'Grokking Group Multiplication with Cosets', Stander et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks As an Alternative to Attention Layers in Transformers”, Bozic et al 2023`{=html} {#bozic-et-al-2023-section}

**[`Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers`{=html}](https://arxiv.org/abs/2311.10642 "'Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers', Bozic et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“HyperFields: Towards Zero-Shot Generation of NeRFs from Text”, Babu et al 2023`{=html} {#babu-et-al-2023-section}

**[`HyperFields: Towards Zero-Shot Generation of NeRFs from Text`{=html}](https://arxiv.org/abs/2310.17075 "'HyperFields: Towards Zero-Shot Generation of NeRFs from Text', Babu et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Grokking Beyond Neural Networks: An Empirical Exploration With Model Complexity”, Miller et al 2023`{=html} {#miller-et-al-2023-3-section}

**[`Grokking Beyond Neural Networks: An Empirical Exploration with Model Complexity`{=html}](https://arxiv.org/abs/2310.17247 "'Grokking Beyond Neural Networks: An Empirical Exploration with Model Complexity', Miller et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“To Grok or Not to Grok: Disentangling Generalization and Memorization on Corrupted Algorithmic Datasets”, Doshi et al 2023`{=html} {#doshi-et-al-2023-section}

**[`To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasets`{=html}](https://arxiv.org/abs/2310.13061 "'To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasets', Doshi et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Polynomial Time Cryptanalytic Extraction of Neural Network Models”, Shamir et al 2023`{=html} {#shamir-et-al-2023-section}

**[`Polynomial Time Cryptanalytic Extraction of Neural Network Models`{=html}](https://arxiv.org/abs/2310.08708 "'Polynomial Time Cryptanalytic Extraction of Neural Network Models', Shamir et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“One Wide Feedforward Is All You Need”, Pires et al 2023`{=html} {#pires-et-al-2023-section}

**[`One Wide Feedforward is All You Need`{=html}](https://arxiv.org/abs/2309.01826#apple "'One Wide Feedforward is All You Need', Pires et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla”, Lieberum et al 2023`{=html} {#lieberum-et-al-2023-section}

**[`Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla`{=html}](https://arxiv.org/abs/2307.09458#deepmind "'Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla', Lieberum et al 2023"){.link-annotated
.id-not .include-annotation link-icon="deepmind" link-icon-type="svg"
link-icon-color="#4185f4"}**

## `“Self Expanding Neural Networks”, Mitchell et al 2023`{=html} {#mitchell-et-al-2023-2-section}

**[`Self Expanding Neural Networks`{=html}](https://arxiv.org/abs/2307.04526 "'Self Expanding Neural Networks', Mitchell et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks”, Zhong et al 2023`{=html} {#zhong-et-al-2023-section}

**[`The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks`{=html}](https://arxiv.org/abs/2306.17844 "'The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks', Zhong et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Scaling MLPs: A Tale of Inductive Bias”, Bachmann et al 2023`{=html} {#bachmann-et-al-2023-section}

**[`Scaling MLPs: A Tale of Inductive Bias`{=html}](https://arxiv.org/abs/2306.13575 "'Scaling MLPs: A Tale of Inductive Bias', Bachmann et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Any Deep ReLU Network Is Shallow”, Villani & Schoots 2023`{=html} {#villani-schoots-2023-section}

**[`Any Deep ReLU Network is Shallow`{=html}](https://arxiv.org/abs/2306.11827 "'Any Deep ReLU Network is Shallow', Villani & Schoots 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Does the First Letter of One’s Name Affect Life Decisions? A Natural Language Processing Examination of Nominative Determinism”, Chatterjee et al 2023`{=html} {#chatterjee-et-al-2023-section}

**[`Does the First Letter of One’s Name Affect Life Decisions? A Natural Language Processing Examination of Nominative Determinism`{=html}](/doc/psychology/2023-chatterjee.pdf "'Does the First Letter of One’s Name Affect Life Decisions? A Natural Language Processing Examination of Nominative Determinism', Chatterjee et al 2023"){.link-annotated
.id-not .include-annotation link-icon="pdf" link-icon-type="svg"
link-icon-color="#f40f02"}**

## `“Simplicial Hopfield Networks”, Burns & Fukai 2023`{=html} {#burns-fukai-2023-section}

**[`Simplicial Hopfield networks`{=html}](https://arxiv.org/abs/2305.05179 "'Simplicial Hopfield networks', Burns & Fukai 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“How Does GPT-2 Compute Greater-Than?: Interpreting Mathematical Abilities in a Pre-Trained Language Model”, Hanna et al 2023`{=html} {#hanna-et-al-2023-section}

**[`How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model`{=html}](https://arxiv.org/abs/2305.00586 "'How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model', Hanna et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Two-Step Training: Adjustable Sketch Colorization via Reference Image and Text Tag”, Yan et al 2023`{=html} {#yan-et-al-2023-1-section}

**[`Two-Step Training: Adjustable Sketch Colorization via Reference Image and Text Tag`{=html}](https://onlinelibrary.wiley.com/doi/full/10.1111/cgf.14791 "'Two-Step Training: Adjustable Sketch Colorization via Reference Image and Text Tag', Yan et al 2023"){.link-annotated
.id-not .include-annotation link-icon="W" link-icon-type="text,sans"}**

## `“HyperDiffusion: Generating Implicit Neural Fields With Weight-Space Diffusion”, Erkoç et al 2023`{=html} {#erkoç-et-al-2023-section}

**[`HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion`{=html}](https://arxiv.org/abs/2303.17015 "'HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion', Erkoç et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“The Quantization Model of Neural Scaling”, Michaud et al 2023`{=html} {#michaud-et-al-2023-section}

**[`The Quantization Model of Neural Scaling`{=html}](https://arxiv.org/abs/2303.13506 "'The Quantization Model of Neural Scaling', Michaud et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“TSMixer: An All-MLP Architecture for Time Series Forecasting”, Chen et al 2023`{=html} {#chen-et-al-2023-14-section}

**[`TSMixer: An All-MLP Architecture for Time Series Forecasting`{=html}](https://arxiv.org/abs/2303.06053#google "'TSMixer: An All-MLP Architecture for Time Series Forecasting', Chen et al 2023"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Loss Landscapes Are All You Need: Neural Network Generalization Can Be Explained Without the Implicit Bias of Gradient Descent”, Chiang et al 2023`{=html} {#chiang-et-al-2023-1-section}

**[`Loss Landscapes are All You Need: Neural Network Generalization Can Be Explained Without the Implicit Bias of Gradient Descent`{=html}](https://openreview.net/forum?id=QC10RmRbZy9 "'Loss Landscapes are All You Need: Neural Network Generalization Can Be Explained Without the Implicit Bias of Gradient Descent', Chiang et al 2023"){.link-annotated
.id-not .include-annotation link-icon="OR" link-icon-type="text,sans"
link-icon-color="#8c1b13"}**

## `“A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations”, Chughtai et al 2023`{=html} {#chughtai-et-al-2023-section}

**[`A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations`{=html}](https://arxiv.org/abs/2302.03025 "'A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations', Chughtai et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Looped Transformers As Programmable Computers”, Giannou et al 2023`{=html} {#giannou-et-al-2023-section}

**[`Looped Transformers as Programmable Computers`{=html}](https://arxiv.org/abs/2301.13196 "'Looped Transformers as Programmable Computers', Giannou et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Organic Reaction Mechanism Classification Using Machine Learning”, Burés & Larrosa 2023`{=html} {#burés-larrosa-2023-section}

**[`Organic reaction mechanism classification using machine learning`{=html}](/doc/science/chemistry/2023-bures.pdf "'Organic reaction mechanism classification using machine learning', Burés & Larrosa 2023"){.link-annotated
.id-not .include-annotation link-icon="pdf" link-icon-type="svg"
link-icon-color="#f40f02"}**

## `“DataMUX: Data Multiplexing for Neural Networks”, Murahari et al 2023`{=html} {#murahari-et-al-2023-1-section}

**[`DataMUX: Data Multiplexing for Neural Networks`{=html}](https://openreview.net/forum?id=UdgtTVTdswg "'DataMUX: Data Multiplexing for Neural Networks', Murahari et al 2023"){.link-annotated
.id-not .include-annotation link-icon="OR" link-icon-type="text,sans"
link-icon-color="#8c1b13"}**

## `“Merging Enzymatic and Synthetic Chemistry With Computational Synthesis Planning”, Levin et al 2022`{=html} {#levin-et-al-2022-1-section}

**[`Merging enzymatic and synthetic chemistry with computational synthesis planning`{=html}](https://www.nature.com/articles/s41467-022-35422-y "'Merging enzymatic and synthetic chemistry with computational synthesis planning', Levin et al 2022"){.link-annotated
.id-not .include-annotation link-icon="n" link-icon-type="text"}**

## `“Magic3D: High-Resolution Text-To-3D Content Creation”, Lin et al 2022`{=html} {#lin-et-al-2022-04-section}

**[`Magic3D: High-Resolution Text-to-3D Content Creation`{=html}](https://arxiv.org/abs/2211.10440#nvidia "'Magic3D: High-Resolution Text-to-3D Content Creation', Lin et al 2022"){.link-annotated
.id-not .include-annotation link-icon="n"
link-icon-type="text,sans,italic" link-icon-color="#77ba00"}**

## `“DINER: Disorder-Invariant Implicit Neural Representation”, Xie et al 2022`{=html} {#xie-et-al-2022-section}

**[`DINER: Disorder-Invariant Implicit Neural Representation`{=html}](https://arxiv.org/abs/2211.07871 "'DINER: Disorder-Invariant Implicit Neural Representation', Xie et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers”, Hassid et al 2022`{=html} {#hassid-et-al-2022-section}

**[`How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers`{=html}](https://arxiv.org/abs/2211.03495 "'How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers', Hassid et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Compressing Multidimensional Weather and Climate Data into Neural Networks”, Huang & Hoefler 2022`{=html} {#huang-hoefler-2022-section}

**[`Compressing multidimensional weather and climate data into neural networks`{=html}](https://arxiv.org/abs/2210.12538 "'Compressing multidimensional weather and climate data into neural networks', Huang & Hoefler 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Deep Differentiable Logic Gate Networks”, Petersen et al 2022`{=html} {#petersen-et-al-2022-section}

**[`Deep Differentiable Logic Gate Networks`{=html}](https://arxiv.org/abs/2210.08277 "'Deep Differentiable Logic Gate Networks', Petersen et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers”, Li et al 2022`{=html} {#li-et-al-2022-11-section}

**[`The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers`{=html}](https://arxiv.org/abs/2210.06313#google "'The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers', Li et al 2022"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Neural Networks Are Decision Trees”, Aytekin 2022`{=html} {#aytekin-2022-section}

**[`Neural Networks are Decision Trees`{=html}](https://arxiv.org/abs/2210.05189 "'Neural Networks are Decision Trees', Aytekin 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“The Unreasonable Effectiveness of Fully-Connected Layers for Low-Data Regimes”, Kocsis et al 2022`{=html} {#kocsis-et-al-2022-section}

**[`The Unreasonable Effectiveness of Fully-Connected Layers for Low-Data Regimes`{=html}](https://arxiv.org/abs/2210.05657 "'The Unreasonable Effectiveness of Fully-Connected Layers for Low-Data Regimes', Kocsis et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Scaling Forward Gradient With Local Losses”, Ren et al 2022`{=html} {#ren-et-al-2022-1-section}

**[`Scaling Forward Gradient With Local Losses`{=html}](https://arxiv.org/abs/2210.03310#google "'Scaling Forward Gradient With Local Losses', Ren et al 2022"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“The Lie Derivative for Measuring Learned Equivariance”, Gruver et al 2022`{=html} {#gruver-et-al-2022-section}

**[`The Lie Derivative for Measuring Learned Equivariance`{=html}](https://arxiv.org/abs/2210.02984 "'The Lie Derivative for Measuring Learned Equivariance', Gruver et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Omnigrok: Grokking Beyond Algorithmic Data”, Liu et al 2022`{=html} {#liu-et-al-2022-10-section}

**[`Omnigrok: Grokking Beyond Algorithmic Data`{=html}](https://arxiv.org/abs/2210.01117 "'Omnigrok: Grokking Beyond Algorithmic Data', Liu et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“DreamFusion: Text-To-3D Using 2D Diffusion”, Poole et al 2022`{=html} {#poole-et-al-2022-section}

**[`DreamFusion: Text-to-3D using 2D Diffusion`{=html}](https://arxiv.org/abs/2209.14988#google "'DreamFusion: Text-to-3D using 2D Diffusion', Poole et al 2022"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“<code>g.pt</code>: Learning to Learn With Generative Models of Neural Network Checkpoints”, Peebles et al 2022`{=html} {#peebles-et-al-2022-section}

**[`<code>g.pt</code>: Learning to Learn with Generative Models of Neural Network Checkpoints`{=html}](https://arxiv.org/abs/2209.12892 "'<code>g.pt</code>: Learning to Learn with Generative Models of Neural Network Checkpoints', Peebles et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Random Initializations Performing above Chance and How to Find Them”, Benzing et al 2022`{=html} {#benzing-et-al-2022-section}

**[`Random initializations performing above chance and how to find them`{=html}](https://arxiv.org/abs/2209.07509 "'Random initializations performing above chance and how to find them', Benzing et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Scaling Laws vs Model Architectures: How Does Inductive Bias Influence Scaling?”, Tay et al 2022`{=html} {#tay-et-al-2022-1-section}

**[`Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?`{=html}](https://arxiv.org/abs/2207.10551#google "'Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?', Tay et al 2022"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Why Do Tree-Based Models Still Outperform Deep Learning on Tabular Data?”, Grinsztajn et al 2022`{=html} {#grinsztajn-et-al-2022-section}

**[`Why do tree-based models still outperform deep learning on tabular data?`{=html}](https://arxiv.org/abs/2207.08815 "'Why do tree-based models still outperform deep learning on tabular data?', Grinsztajn et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Revisiting Pretraining Objectives for Tabular Deep Learning”, Rubachev et al 2022`{=html} {#rubachev-et-al-2022-section}

**[`Revisiting Pretraining Objectives for Tabular Deep Learning`{=html}](https://arxiv.org/abs/2207.03208 "'Revisiting Pretraining Objectives for Tabular Deep Learning', Rubachev et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“RHO-LOSS: Prioritized Training on Points That Are Learnable, Worth Learning, and Not Yet Learnt”, Mindermann et al 2022`{=html} {#mindermann-et-al-2022-section}

**[`RHO-LOSS: Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt`{=html}](https://arxiv.org/abs/2206.07137 "'RHO-LOSS: Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt', Mindermann et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“MLP-3D: A MLP-Like 3D Architecture With Grouped Time Mixing”, Qiu et al 2022`{=html} {#qiu-et-al-2022-section}

**[`MLP-3D: A MLP-like 3D Architecture with Grouped Time Mixing`{=html}](https://arxiv.org/abs/2206.06292#jd "'MLP-3D: A MLP-like 3D Architecture with Grouped Time Mixing', Qiu et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“ChordMixer: A Scalable Neural Attention Model for Sequences With Different Lengths”, Khalitov et al 2022`{=html} {#khalitov-et-al-2022-section}

**[`ChordMixer: A Scalable Neural Attention Model for Sequences with Different Lengths`{=html}](https://arxiv.org/abs/2206.05852 "'ChordMixer: A Scalable Neural Attention Model for Sequences with Different Lengths', Khalitov et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Sparse Mixers: Combining MoE and Mixing to Build a More Efficient BERT”, Lee-Thorp & Ainslie 2022`{=html} {#lee-thorp-ainslie-2022-section}

**[`Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT`{=html}](https://arxiv.org/abs/2205.12399#google "'Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT', Lee-Thorp & Ainslie 2022"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Towards Understanding Grokking: An Effective Theory of Representation Learning”, Liu et al 2022`{=html} {#liu-et-al-2022-18-section}

**[`Towards Understanding Grokking: An Effective Theory of Representation Learning`{=html}](https://arxiv.org/abs/2205.10343 "'Towards Understanding Grokking: An Effective Theory of Representation Learning', Liu et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Paramixer: Parameterizing Mixing Links in Sparse Factors Works Better Than Dot-Product Self-Attention”, Yu et al 2022`{=html} {#yu-et-al-2022-4-section}

**[`Paramixer: Parameterizing Mixing Links in Sparse Factors Works Better than Dot-Product Self-Attention`{=html}](https://arxiv.org/abs/2204.10670 "'Paramixer: Parameterizing Mixing Links in Sparse Factors Works Better than Dot-Product Self-Attention', Yu et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Deep Learning Meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?”, Zhang & Wang 2022`{=html} {#zhang-wang-2022-section}

**[`Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?`{=html}](https://arxiv.org/abs/2204.09664 "'Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?', Zhang & Wang 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Efficient Language Modeling With Sparse All-MLP”, Yu et al 2022`{=html} {#yu-et-al-2022-5-section}

**[`Efficient Language Modeling with Sparse All-MLP`{=html}](https://arxiv.org/abs/2203.06850 "'Efficient Language Modeling with Sparse All-MLP', Yu et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“HyperMixer: An MLP-Based Low Cost Alternative to Transformers”, Mai et al 2022`{=html} {#mai-et-al-2022-section}

**[`HyperMixer: An MLP-based Low Cost Alternative to Transformers`{=html}](https://arxiv.org/abs/2203.03691 "'HyperMixer: An MLP-based Low Cost Alternative to Transformers', Mai et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“MLP-ASR: Sequence-Length Agnostic All-MLP Architectures for Speech Recognition”, Sakuma et al 2022`{=html} {#sakuma-et-al-2022-section}

**[`MLP-ASR: Sequence-length agnostic all-MLP architectures for speech recognition`{=html}](https://arxiv.org/abs/2202.08456 "'MLP-ASR: Sequence-length agnostic all-MLP architectures for speech recognition', Sakuma et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Mixing and Shifting: Exploiting Global and Local Dependencies in Vision MLPs”, Zheng et al 2022`{=html} {#zheng-et-al-2022-2-section}

**[`Mixing and Shifting: Exploiting Global and Local Dependencies in Vision MLPs`{=html}](https://arxiv.org/abs/2202.06510#microsoft "'Mixing and Shifting: Exploiting Global and Local Dependencies in Vision MLPs', Zheng et al 2022"){.link-annotated
.id-not .include-annotation link-icon="MS"
link-icon-type="text,sans,italic"}**

## `“PNLP-Mixer: an Efficient All-MLP Architecture for Language”, Fusco et al 2022`{=html} {#fusco-et-al-2022-section}

**[`pNLP-Mixer: an Efficient all-MLP Architecture for Language`{=html}](https://arxiv.org/abs/2202.04350 "'pNLP-Mixer: an Efficient all-MLP Architecture for Language', Fusco et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Data-Driven Emergence of Convolutional Structure in Neural Networks”, Ingrosso & Goldt 2022`{=html} {#ingrosso-goldt-2022-section}

**[`Data-driven emergence of convolutional structure in neural networks`{=html}](https://arxiv.org/abs/2202.00565 "'Data-driven emergence of convolutional structure in neural networks', Ingrosso & Goldt 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanism (ShiftViT)”, Wang et al 2022`{=html} {#wang-et-al-2022-22-section}

**[`When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanism (ShiftViT)`{=html}](https://arxiv.org/abs/2201.10801 "'When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanism (ShiftViT)', Wang et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“ConvMixer: Patches Are All You Need?”, Trockman & Kolter 2022`{=html} {#trockman-kolter-2022-section}

**[`ConvMixer: Patches Are All You Need?`{=html}](https://arxiv.org/abs/2201.09792 "'ConvMixer: Patches Are All You Need?', Trockman & Kolter 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“MAXIM: Multi-Axis MLP for Image Processing”, Tu et al 2022`{=html} {#tu-et-al-2022-3-section}

**[`MAXIM: Multi-Axis MLP for Image Processing`{=html}](https://arxiv.org/abs/2201.02973#google "'MAXIM: Multi-Axis MLP for Image Processing', Tu et al 2022"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets [Paper]”, Power et al 2022`{=html} {#power-et-al-2022-section}

**[`Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets [paper]`{=html}](https://arxiv.org/abs/2201.02177#openai "'Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets [paper]', Power et al 2022"){.link-annotated
.id-not .include-annotation link-icon="openai" link-icon-type="svg"}**

## `“The GatedTabTransformer: An Enhanced Deep Learning Architecture for Tabular Modeling”, Cholakov & Kolev 2022`{=html} {#cholakov-kolev-2022-section}

**[`The GatedTabTransformer: An enhanced deep learning architecture for tabular modeling`{=html}](https://arxiv.org/abs/2201.00199#amazon "'The GatedTabTransformer: An enhanced deep learning architecture for tabular modeling', Cholakov & Kolev 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“MLP Architectures for Vision-And-Language Modeling: An Empirical Study”, Nie et al 2021`{=html} {#nie-et-al-2021-section}

**[`MLP Architectures for Vision-and-Language Modeling: An Empirical Study`{=html}](https://arxiv.org/abs/2112.04453 "'MLP Architectures for Vision-and-Language Modeling: An Empirical Study', Nie et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Noether Networks: Meta-Learning Useful Conserved Quantities”, Alet et al 2021`{=html} {#alet-et-al-2021-section}

**[`Noether Networks: Meta-Learning Useful Conserved Quantities`{=html}](https://arxiv.org/abs/2112.03321 "'Noether Networks: Meta-Learning Useful Conserved Quantities', Alet et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Residual Pathway Priors for Soft Equivariance Constraints”, Finzi et al 2021`{=html} {#finzi-et-al-2021-section}

**[`Residual Pathway Priors for Soft Equivariance Constraints`{=html}](https://arxiv.org/abs/2112.01388 "'Residual Pathway Priors for Soft Equivariance Constraints', Finzi et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Zero-Shot Text-Guided Object Generation With Dream Fields”, Jain et al 2021`{=html} {#jain-et-al-2021-2-section}

**[`Zero-Shot Text-Guided Object Generation with Dream Fields`{=html}](https://arxiv.org/abs/2112.01455#google "'Zero-Shot Text-Guided Object Generation with Dream Fields', Jain et al 2021"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“MorphMLP: A Self-Attention Free, MLP-Like Backbone for Image and Video”, Zhang et al 2021`{=html} {#zhang-et-al-2021-morphmlp-section}

**[`MorphMLP: A Self-Attention Free, MLP-Like Backbone for Image and Video`{=html}](https://arxiv.org/abs/2111.12527 "'MorphMLP: A Self-Attention Free, MLP-Like Backbone for Image and Video', Zhang et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“PointMixer: MLP-Mixer for Point Cloud Understanding”, Choe et al 2021`{=html} {#choe-et-al-2021-section}

**[`PointMixer: MLP-Mixer for Point Cloud Understanding`{=html}](https://arxiv.org/abs/2111.11187 "'PointMixer: MLP-Mixer for Point Cloud Understanding', Choe et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“MetaFormer Is Actually What You Need for Vision”, Yu et al 2021`{=html} {#yu-et-al-2021-1-section}

**[`MetaFormer is Actually What You Need for Vision`{=html}](https://arxiv.org/abs/2111.11418 "'MetaFormer is Actually What You Need for Vision', Yu et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Deep Learning without Shortcuts: Shaping the Kernel With Tailored Rectifiers”, Zhang et al 2021`{=html} {#zhang-et-al-2021-02-section}

**[`Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers`{=html}](https://openreview.net/forum?id=U0k7XNTiFEq "'Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers', Zhang et al 2021"){.link-annotated
.id-not .include-annotation link-icon="OR" link-icon-type="text,sans"
link-icon-color="#8c1b13"}**

## `“ZerO Initialization: Initializing Residual Networks With Only Zeros and Ones”, Zhao et al 2021`{=html} {#zhao-et-al-2021-3-section}

**[`ZerO Initialization: Initializing Residual Networks with only Zeros and Ones`{=html}](https://arxiv.org/abs/2110.12661 "'ZerO Initialization: Initializing Residual Networks with only Zeros and Ones', Zhao et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Wide Neural Networks Forget Less Catastrophically”, Mirzadeh et al 2021`{=html} {#mirzadeh-et-al-2021-section}

**[`Wide Neural Networks Forget Less Catastrophically`{=html}](https://arxiv.org/abs/2110.11526#deepmind "'Wide Neural Networks Forget Less Catastrophically', Mirzadeh et al 2021"){.link-annotated
.id-not .include-annotation link-icon="deepmind" link-icon-type="svg"
link-icon-color="#4185f4"}**

## `“ADOP: Approximate Differentiable One-Pixel Point Rendering”, Rückert et al 2021`{=html} {#rückert-et-al-2021-section}

**[`ADOP: Approximate Differentiable One-Pixel Point Rendering`{=html}](https://arxiv.org/abs/2110.06635 "'ADOP: Approximate Differentiable One-Pixel Point Rendering', Rückert et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Rapid Training of Deep Neural Networks without Skip Connections or Normalization Layers Using Deep Kernel Shaping”, Martens et al 2021`{=html} {#martens-et-al-2021-section}

**[`Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping`{=html}](https://arxiv.org/abs/2110.01765#deepmind "'Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping', Martens et al 2021"){.link-annotated
.id-not .include-annotation link-icon="deepmind" link-icon-type="svg"
link-icon-color="#4185f4"}**

## `“Exploring the Limits of Large Scale Pre-Training”, Abnar et al 2021`{=html} {#abnar-et-al-2021-section}

**[`Exploring the Limits of Large Scale Pre-training`{=html}](https://arxiv.org/abs/2110.02095#google "'Exploring the Limits of Large Scale Pre-training', Abnar et al 2021"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Sparse MLP for Image Recognition: Is Self-Attention Really Necessary?”, Tang et al 2021`{=html} {#tang-et-al-2021-2-section}

**[`Sparse MLP for Image Recognition: Is Self-Attention Really Necessary?`{=html}](https://arxiv.org/abs/2109.05422 "'Sparse MLP for Image Recognition: Is Self-Attention Really Necessary?', Tang et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“AFT: An Attention Free Transformer”, Zhai et al 2021`{=html} {#zhai-et-al-2021-section}

**[`AFT: An Attention Free Transformer`{=html}](https://openreview.net/forum?id=pW--cu2FCHY#apple "'AFT: An Attention Free Transformer', Zhai et al 2021"){.link-annotated
.id-not .include-annotation link-icon="OR" link-icon-type="text,sans"
link-icon-color="#8c1b13"}**

## `“ConvMLP: Hierarchical Convolutional MLPs for Vision”, Li et al 2021`{=html} {#li-et-al-2021-5-section}

**[`ConvMLP: Hierarchical Convolutional MLPs for Vision`{=html}](https://arxiv.org/abs/2109.04454 "'ConvMLP: Hierarchical Convolutional MLPs for Vision', Li et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Sparse-MLP: A Fully-MLP Architecture With Conditional Computation”, Lou et al 2021`{=html} {#lou-et-al-2021-section}

**[`Sparse-MLP: A Fully-MLP Architecture with Conditional Computation`{=html}](https://arxiv.org/abs/2109.02008 "'Sparse-MLP: A Fully-MLP Architecture with Conditional Computation', Lou et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Implicit Behavioral Cloning”, Florence et al 2021`{=html} {#florence-et-al-2021-section}

**[`Implicit Behavioral Cloning`{=html}](https://arxiv.org/abs/2109.00137#google "'Implicit Behavioral Cloning', Florence et al 2021"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP”, Zhao et al 2021`{=html} {#zhao-et-al-2021-4-section}

**[`A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP`{=html}](https://arxiv.org/abs/2108.13002#microsoft "'A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP', Zhao et al 2021"){.link-annotated
.id-not .include-annotation link-icon="MS"
link-icon-type="text,sans,italic"}**

## `“Hire-MLP: Vision MLP via Hierarchical Rearrangement”, Guo et al 2021`{=html} {#guo-et-al-2021-hiremlp-section}

**[`Hire-MLP: Vision MLP via Hierarchical Rearrangement`{=html}](https://arxiv.org/abs/2108.13341#huawei "'Hire-MLP: Vision MLP via Hierarchical Rearrangement', Guo et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“RaftMLP: How Much Can Be Done Without Attention and With Less Spatial Locality?”, Tatsunami & Taki 2021`{=html} {#tatsunami-taki-2021-section}

**[`RaftMLP: How Much Can Be Done Without Attention and with Less Spatial Locality?`{=html}](https://arxiv.org/abs/2108.04384 "'RaftMLP: How Much Can Be Done Without Attention and with Less Spatial Locality?', Tatsunami & Taki 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“S<sup>2</sup>-MLPv2: Improved Spatial-Shift MLP Architecture for Vision”, Yu et al 2021`{=html} {#yu-et-al-2021-s2mlpv2-section}

**[`S<sup>2</sup>-MLPv2: Improved Spatial-Shift MLP Architecture for Vision`{=html}](https://arxiv.org/abs/2108.01072#baidu "'S<sup>2</sup>-MLPv2: Improved Spatial-Shift MLP Architecture for Vision', Yu et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“CycleMLP: A MLP-Like Architecture for Dense Prediction”, Chen et al 2021`{=html} {#chen-et-al-2021-cyclemlp-section}

**[`CycleMLP: A MLP-like Architecture for Dense Prediction`{=html}](https://arxiv.org/abs/2107.10224 "'CycleMLP: A MLP-like Architecture for Dense Prediction', Chen et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“AS-MLP: An Axial Shifted MLP Architecture for Vision”, Lian et al 2021`{=html} {#lian-et-al-2021-2-section}

**[`AS-MLP: An Axial Shifted MLP Architecture for Vision`{=html}](https://arxiv.org/abs/2107.08391 "'AS-MLP: An Axial Shifted MLP Architecture for Vision', Lian et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition”, Hou et al 2021`{=html} {#hou-et-al-2021-section}

**[`Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition`{=html}](https://arxiv.org/abs/2106.12368 "'Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition', Hou et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Real-Time Neural Radiance Caching for Path Tracing”, Müller et al 2021`{=html} {#müller-et-al-2021-4-section}

**[`Real-time Neural Radiance Caching for Path Tracing`{=html}](https://arxiv.org/abs/2106.12372#nvidia "'Real-time Neural Radiance Caching for Path Tracing', Müller et al 2021"){.link-annotated
.id-not .include-annotation link-icon="n"
link-icon-type="text,sans,italic" link-icon-color="#77ba00"}**

## `“Towards Biologically Plausible Convolutional Networks”, Pogodin et al 2021`{=html} {#pogodin-et-al-2021-section}

**[`Towards Biologically Plausible Convolutional Networks`{=html}](https://arxiv.org/abs/2106.13031 "'Towards Biologically Plausible Convolutional Networks', Pogodin et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Well-Tuned Simple Nets Excel on Tabular Datasets”, Kadra et al 2021`{=html} {#kadra-et-al-2021-section}

**[`Well-tuned Simple Nets Excel on Tabular Datasets`{=html}](https://arxiv.org/abs/2106.11189 "'Well-tuned Simple Nets Excel on Tabular Datasets', Kadra et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“MLP Singer: Towards Rapid Parallel Korean Singing Voice Synthesis”, Tae et al 2021`{=html} {#tae-et-al-2021-section}

**[`MLP Singer: Towards Rapid Parallel Korean Singing Voice Synthesis`{=html}](https://arxiv.org/abs/2106.07886 "'MLP Singer: Towards Rapid Parallel Korean Singing Voice Synthesis', Tae et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“PairConnect: A Compute-Efficient MLP Alternative to Attention”, Xu et al 2021`{=html} {#xu-et-al-2021-6-section}

**[`PairConnect: A Compute-Efficient MLP Alternative to Attention`{=html}](https://arxiv.org/abs/2106.08235 "'PairConnect: A Compute-Efficient MLP Alternative to Attention', Xu et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“S<sup>2</sup>-MLP: Spatial-Shift MLP Architecture for Vision”, Yu et al 2021`{=html} {#yu-et-al-2021-s2mlp-section}

**[`S<sup>2</sup>-MLP: Spatial-Shift MLP Architecture for Vision`{=html}](https://arxiv.org/abs/2106.07477#baidu "'S<sup>2</sup>-MLP: Spatial-Shift MLP Architecture for Vision', Yu et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“When Vision Transformers Outperform ResNets without Pre-Training or Strong Data Augmentations”, Chen et al 2021`{=html} {#chen-et-al-2021-10-section}

**[`When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations`{=html}](https://arxiv.org/abs/2106.01548 "'When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations', Chen et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Container: Context Aggregation Network”, Gao et al 2021`{=html} {#gao-et-al-2021-3-section}

**[`Container: Context Aggregation Network`{=html}](https://arxiv.org/abs/2106.01401 "'Container: Context Aggregation Network', Gao et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“MixerGAN: An MLP-Based Architecture for Unpaired Image-To-Image Translation”, Cazenavette & Guevara 2021`{=html} {#cazenavette-guevara-2021-section}

**[`MixerGAN: An MLP-Based Architecture for Unpaired Image-to-Image Translation`{=html}](https://arxiv.org/abs/2105.14110 "'MixerGAN: An MLP-Based Architecture for Unpaired Image-to-Image Translation', Cazenavette & Guevara 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“One4all User Representation for Recommender Systems in E-Commerce”, Shin et al 2021`{=html} {#shin-et-al-2021-2-section}

**[`One4all User Representation for Recommender Systems in E-commerce`{=html}](https://arxiv.org/abs/2106.00573#naver "'One4all User Representation for Recommender Systems in E-commerce', Shin et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Pay Attention to MLPs”, Liu et al 2021`{=html} {#liu-et-al-2021-gmlp-section}

**[`Pay Attention to MLPs`{=html}](https://arxiv.org/abs/2105.08050#google "'Pay Attention to MLPs', Liu et al 2021"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“FNet: Mixing Tokens With Fourier Transforms”, Lee-Thorp et al 2021`{=html} {#lee-thorp-et-al-2021-section}

**[`FNet: Mixing Tokens with Fourier Transforms`{=html}](https://arxiv.org/abs/2105.03824#google "'FNet: Mixing Tokens with Fourier Transforms', Lee-Thorp et al 2021"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“ResMLP: Feedforward Networks for Image Classification With Data-Efficient Training”, Touvron et al 2021`{=html} {#touvron-et-al-2021-section}

**[`ResMLP: Feedforward networks for image classification with data-efficient training`{=html}](https://arxiv.org/abs/2105.03404#facebook "'ResMLP: Feedforward networks for image classification with data-efficient training', Touvron et al 2021"){.link-annotated
.id-not .include-annotation link-icon="facebook" link-icon-type="svg"
link-icon-color="#1877f2"}**

## `“Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet”, Melas-Kyriazi 2021`{=html} {#melas-kyriazi-2021-section}

**[`Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet`{=html}](https://arxiv.org/abs/2105.02723 "'Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet', Melas-Kyriazi 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Multi-Scale Inference of Genetic Trait Architecture Using Biologically Annotated Neural Networks”, Demetci et al 2021`{=html} {#demetci-et-al-2021-section}

**[`Multi-scale Inference of Genetic Trait Architecture using Biologically Annotated Neural Networks`{=html}](https://www.biorxiv.org/content/10.1101/2020.07.02.184465.full "'Multi-scale Inference of Genetic Trait Architecture using Biologically Annotated Neural Networks', Demetci et al 2021"){.link-annotated
.id-not .include-annotation link-icon="chi-dna" link-icon-type="svg"
link-icon-color="#bd2736"}**

## `“RepMLP: Re-Parameterizing Convolutions into Fully-Connected Layers for Image Recognition”, Ding et al 2021`{=html} {#ding-et-al-2021-3-section}

**[`RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognition`{=html}](https://arxiv.org/abs/2105.01883 "'RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognition', Ding et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“MLP-Mixer: An All-MLP Architecture for Vision”, Tolstikhin et al 2021`{=html} {#tolstikhin-et-al-2021-section}

**[`MLP-Mixer: An all-MLP Architecture for Vision`{=html}](https://arxiv.org/abs/2105.01601#google "'MLP-Mixer: An all-MLP Architecture for Vision', Tolstikhin et al 2021"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Grokking: Generalization Beyond Overfitting On Small Algorithmic Datasets”, Power et al 2021`{=html} {#power-et-al-2021-section}

**[`Grokking: Generalization Beyond Overfitting On Small Algorithmic Datasets`{=html}](/doc/ai/nn/fully-connected/2021-power.pdf#openai "'Grokking: Generalization Beyond Overfitting On Small Algorithmic Datasets', Power et al 2021"){.link-annotated
.id-not .include-annotation link-icon="openai" link-icon-type="svg"}**

## `“Sifting out the Features by Pruning: Are Convolutional Networks the Winning Lottery Ticket of Fully Connected Ones?”, Pellegrini & Biroli 2021`{=html} {#pellegrini-biroli-2021-section}

**[`Sifting out the features by pruning: Are convolutional networks the winning lottery ticket of fully connected ones?`{=html}](https://arxiv.org/abs/2104.13343 "'Sifting out the features by pruning: Are convolutional networks the winning lottery ticket of fully connected ones?', Pellegrini & Biroli 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Fully-Connected Neural Nets”, Gwern 2021`{=html} {#gwern-doc-ai-nn-fully-connected-abstract-section}

**[`Fully-Connected Neural Nets`{=html}](/doc/ai/nn/fully-connected/abstract "'Fully-Connected Neural Nets', Gwern 2021"){.link-annotated-partial
.id-not .include-annotation}**

## `“Revisiting Simple Neural Probabilistic Language Models”, Sun & Iyyer 2021`{=html} {#sun-iyyer-2021-section}

**[`Revisiting Simple Neural Probabilistic Language Models`{=html}](https://arxiv.org/abs/2104.03474 "'Revisiting Simple Neural Probabilistic Language Models', Sun & Iyyer 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“KiloNeRF: Speeding up Neural Radiance Fields With Thousands of Tiny MLPs”, Reiser et al 2021`{=html} {#reiser-et-al-2021-section}

**[`KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs`{=html}](https://arxiv.org/abs/2103.13744 "'KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs', Reiser et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows”, Liu et al 2021`{=html} {#liu-et-al-2021-swintranformer-section}

**[`Swin Transformer: Hierarchical Vision Transformer using Shifted Windows`{=html}](https://arxiv.org/abs/2103.14030 "'Swin Transformer: Hierarchical Vision Transformer using Shifted Windows', Liu et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Attention Is Not All You Need: Pure Attention Loses Rank Doubly Exponentially With Depth”, Dong et al 2021`{=html} {#dong-et-al-2021-1-section}

**[`Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth`{=html}](https://arxiv.org/abs/2103.03404 "'Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth', Dong et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Clusterability in Neural Networks”, Filan et al 2021`{=html} {#filan-et-al-2021-section}

**[`Clusterability in Neural Networks`{=html}](https://arxiv.org/abs/2103.03386 "'Clusterability in Neural Networks', Filan et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Training Larger Networks for Deep Reinforcement Learning”, Ota et al 2021`{=html} {#ota-et-al-2021-section}

**[`Training Larger Networks for Deep Reinforcement Learning`{=html}](https://arxiv.org/abs/2102.07920 "'Training Larger Networks for Deep Reinforcement Learning', Ota et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Explaining Neural Scaling Laws”, Bahri et al 2021`{=html} {#bahri-et-al-2021-2-section}

**[`Explaining Neural Scaling Laws`{=html}](https://arxiv.org/abs/2102.06701#deepmind "'Explaining Neural Scaling Laws', Bahri et al 2021"){.link-annotated
.id-not .include-annotation link-icon="deepmind" link-icon-type="svg"
link-icon-color="#4185f4"}**

## `“Neural Geometric Level of Detail: Real-Time Rendering With Implicit 3D Shapes”, Takikawa et al 2021`{=html} {#takikawa-et-al-2021-section}

**[`Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes`{=html}](https://arxiv.org/abs/2101.10994 "'Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes', Takikawa et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Is MLP-Mixer a CNN in Disguise? As Part of This Blog Post, We Look at the MLP Mixer Architecture in Detail and Also Understand Why It Is Not Considered Convolution Free.”`{=html} {#_I43yAqTI-section}

**[`Is MLP-Mixer a CNN in disguise? As part of this blog post, we look at the MLP Mixer architecture in detail and also understand why it is not considered convolution free.`{=html}](/doc/ai/nn/fully-connected/2021-arora.html "Is MLP-Mixer a CNN in disguise? As part of this blog post, we look at the MLP Mixer architecture in detail and also understand why it is not considered convolution free."){.link-annotated-partial
.id-not .include-annotation link-icon="internet-archive"
link-icon-type="svg"}**

## `“Transformer Feed-Forward Layers Are Key-Value Memories”, Pipek et al 2020`{=html} {#pipek-et-al-2020-section}

**[`Transformer Feed-Forward Layers Are Key-Value Memories`{=html}](https://arxiv.org/abs/2012.14913 "'Transformer Feed-Forward Layers Are Key-Value Memories', Pipek et al 2020"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“AdnFM: An Attentive DenseNet Based Factorization Machine for CTR Prediction”, Wang et al 2020`{=html} {#wang-et-al-2020-05-section}

**[`AdnFM: An Attentive DenseNet based Factorization Machine for CTR Prediction`{=html}](https://arxiv.org/abs/2012.10820#tencent "'AdnFM: An Attentive DenseNet based Factorization Machine for CTR Prediction', Wang et al 2020"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“TabTransformer: Tabular Data Modeling Using Contextual Embeddings”, Huang et al 2020`{=html} {#huang-et-al-2020-1-section}

**[`TabTransformer: Tabular Data Modeling Using Contextual Embeddings`{=html}](https://arxiv.org/abs/2012.06678#amazon "'TabTransformer: Tabular Data Modeling Using Contextual Embeddings', Huang et al 2020"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Scaling down Deep Learning”, Greydanus 2020`{=html} {#greydanus-2020-section}

**[`Scaling down Deep Learning`{=html}](https://greydanus.github.io/2020/12/01/scaling-down/ "'Scaling down Deep Learning', Greydanus 2020"){.link-annotated
.id-not .include-annotation}**

## `“Image Generators With Conditionally-Independent Pixel Synthesis”, Anokhin et al 2020`{=html} {#anokhin-et-al-2020-section}

**[`Image Generators with Conditionally-Independent Pixel Synthesis`{=html}](https://arxiv.org/abs/2011.13775 "'Image Generators with Conditionally-Independent Pixel Synthesis', Anokhin et al 2020"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“D2RL: Deep Dense Architectures in Reinforcement Learning”, Sinha et al 2020`{=html} {#sinha-et-al-2020-1-section}

**[`D2RL: Deep Dense Architectures in Reinforcement Learning`{=html}](https://arxiv.org/abs/2010.09163 "'D2RL: Deep Dense Architectures in Reinforcement Learning', Sinha et al 2020"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Fourier Neural Operator for Parametric Partial Differential Equations”, Li et al 2020`{=html} {#li-et-al-2020-2-section}

**[`Fourier Neural Operator for Parametric Partial Differential Equations`{=html}](https://arxiv.org/abs/2010.08895 "'Fourier Neural Operator for Parametric Partial Differential Equations', Li et al 2020"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Intelligent Matrix Exponentiation”, Fischbacher et al 2020`{=html} {#fischbacher-et-al-2020-section}

**[`Intelligent Matrix Exponentiation`{=html}](https://arxiv.org/abs/2008.03936 "'Intelligent Matrix Exponentiation', Fischbacher et al 2020"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Towards Learning Convolutions from Scratch”, Neyshabur 2020`{=html} {#neyshabur-2020-section}

**[`Towards Learning Convolutions from Scratch`{=html}](https://arxiv.org/abs/2007.13657 "'Towards Learning Convolutions from Scratch', Neyshabur 2020"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Hyperparameter Ensembles for Robustness and Uncertainty Quantification”, Wenzel et al 2020`{=html} {#wenzel-et-al-2020-section}

**[`Hyperparameter Ensembles for Robustness and Uncertainty Quantification`{=html}](https://arxiv.org/abs/2006.13570#google "'Hyperparameter Ensembles for Robustness and Uncertainty Quantification', Wenzel et al 2020"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains”, Tancik et al 2020`{=html} {#tancik-et-al-2020-section}

**[`Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains`{=html}](https://arxiv.org/abs/2006.10739 "'Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains', Tancik et al 2020"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“SIREN: Implicit Neural Representations With Periodic Activation Functions”, Sitzmann et al 2020`{=html} {#sitzmann-et-al-2020-section}

**[`SIREN: Implicit Neural Representations with Periodic Activation Functions`{=html}](https://arxiv.org/abs/2006.09661 "'SIREN: Implicit Neural Representations with Periodic Activation Functions', Sitzmann et al 2020"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Linformer: Self-Attention With Linear Complexity”, Wang et al 2020`{=html} {#wang-et-al-2020-09-section}

**[`Linformer: Self-Attention with Linear Complexity`{=html}](https://arxiv.org/abs/2006.04768#facebook "'Linformer: Self-Attention with Linear Complexity', Wang et al 2020"){.link-annotated
.id-not .include-annotation link-icon="facebook" link-icon-type="svg"
link-icon-color="#1877f2"}**

## `“A Map of Object Space in Primate Inferotemporal Cortex”, Bao et al 2020`{=html} {#bao-et-al-2020-section}

**[`A map of object space in primate inferotemporal cortex`{=html}](/doc/ai/nn/fully-connected/2020-bao.pdf "'A map of object space in primate inferotemporal cortex', Bao et al 2020"){.link-annotated
.id-not .include-annotation link-icon="pdf" link-icon-type="svg"
link-icon-color="#f40f02"}**

## `“Synthesizer: Rethinking Self-Attention in Transformer Models”, Tay et al 2020`{=html} {#tay-et-al-2020-2-section}

**[`Synthesizer: Rethinking Self-Attention in Transformer Models`{=html}](https://arxiv.org/abs/2005.00743#google "'Synthesizer: Rethinking Self-Attention in Transformer Models', Tay et al 2020"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Deep Learning Training in Facebook Data Centers: Design of Scale-Up and Scale-Out Systems”, Naumov et al 2020`{=html} {#naumov-et-al-2020-section}

**[`Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems`{=html}](https://arxiv.org/abs/2003.09518#facebook "'Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out Systems', Naumov et al 2020"){.link-annotated
.id-not .include-annotation link-icon="facebook" link-icon-type="svg"
link-icon-color="#1877f2"}**

## `“NeRF: Representing Scenes As Neural Radiance Fields for View Synthesis”, Mildenhall et al 2020`{=html} {#mildenhall-et-al-2020-section}

**[`NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis`{=html}](https://arxiv.org/abs/2003.08934 "'NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis', Mildenhall et al 2020"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Cryptanalytic Extraction of Neural Network Models”, Carlini et al 2020`{=html} {#carlini-et-al-2020-2-section}

**[`Cryptanalytic Extraction of Neural Network Models`{=html}](https://arxiv.org/abs/2003.04884 "'Cryptanalytic Extraction of Neural Network Models', Carlini et al 2020"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“ReZero Is All You Need: Fast Convergence at Large Depth”, Bachlechner et al 2020`{=html} {#bachlechner-et-al-2020-section}

**[`ReZero is All You Need: Fast Convergence at Large Depth`{=html}](https://arxiv.org/abs/2003.04887 "'ReZero is All You Need: Fast Convergence at Large Depth', Bachlechner et al 2020"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Train-By-Reconnect: Decoupling Locations of Weights from Their Values (LaPerm)”, Qiu & Suda 2020`{=html} {#qiu-suda-2020-section}

**[`Train-by-Reconnect: Decoupling Locations of Weights from their Values (LaPerm)`{=html}](https://arxiv.org/abs/2003.02570 "'Train-by-Reconnect: Decoupling Locations of Weights from their Values (LaPerm)', Qiu & Suda 2020"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Can Increasing Input Dimensionality Improve Deep Reinforcement Learning?”, Ota et al 2020`{=html} {#ota-et-al-2020-section}

**[`Can Increasing Input Dimensionality Improve Deep Reinforcement Learning?`{=html}](https://arxiv.org/abs/2003.01629 "'Can Increasing Input Dimensionality Improve Deep Reinforcement Learning?', Ota et al 2020"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Quasi-Equivalence of Width and Depth of Neural Networks”, Fan et al 2020`{=html} {#fan-et-al-2020-4-section}

**[`Quasi-Equivalence of Width and Depth of Neural Networks`{=html}](https://arxiv.org/abs/2002.02515 "'Quasi-Equivalence of Width and Depth of Neural Networks', Fan et al 2020"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Gesticulator: A Framework for Semantically-Aware Speech-Driven Gesture Generation”, Kucherenko et al 2020`{=html} {#kucherenko-et-al-2020-section}

**[`Gesticulator: A framework for semantically-aware speech-driven gesture generation`{=html}](https://arxiv.org/abs/2001.09326 "'Gesticulator: A framework for semantically-aware speech-driven gesture generation', Kucherenko et al 2020"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“What’s Hidden in a Randomly Weighted Neural Network?”, Ramanujan et al 2019`{=html} {#ramanujan-et-al-2019-section}

**[`What’s Hidden in a Randomly Weighted Neural Network?`{=html}](https://arxiv.org/abs/1911.13299 "'What’s Hidden in a Randomly Weighted Neural Network?', Ramanujan et al 2019"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Understanding the Generalization of ‘Lottery Tickets’ in Neural Networks”, Morcos & Tian 2019`{=html} {#morcos-tian-2019-section}

**[`Understanding the generalization of ‘lottery tickets’ in neural networks`{=html}](https://ai.meta.com/blog/understanding-the-generalization-of-lottery-tickets-in-neural-networks/ "'Understanding the generalization of ‘lottery tickets’ in neural networks', Morcos & Tian 2019"){.link-annotated
.id-not .include-annotation link-icon="facebook" link-icon-type="svg"
link-icon-color="#1877f2"}**

## `“The Bouncer Problem: Challenges to Remote Explainability”, Merrer & Tredan 2019`{=html} {#merrer-tredan-2019-section}

**[`The Bouncer Problem: Challenges to Remote Explainability`{=html}](https://arxiv.org/abs/1910.01432 "'The Bouncer Problem: Challenges to Remote Explainability', Merrer & Tredan 2019"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“3D Human Pose Estimation via Human Structure-Aware Fully Connected Network”, Zhang et al 2019d`{=html} {#zhang-et-al-2019d-section}

**[`3D human pose estimation via human structure-aware fully connected network`{=html}](/doc/ai/nn/fully-connected/2019-zhang-4.pdf "'3D human pose estimation via human structure-aware fully connected network', Zhang et al 2019d"){.link-annotated
.id-not .include-annotation link-icon="pdf" link-icon-type="svg"
link-icon-color="#f40f02"}**

## `“Finding the Needle in the Haystack With Convolutions: on the Benefits of Architectural Bias”, d’Ascoli et al 2019`{=html} {#dascoli-et-al-2019-section}

**[`Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias`{=html}](https://arxiv.org/abs/1906.06766 "'Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias', d’Ascoli et al 2019"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Generalization Guarantees for Neural Networks via Harnessing the Low-Rank Structure of the Jacobian”, Oymak et al 2019`{=html} {#oymak-et-al-2019-section}

**[`Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian`{=html}](https://arxiv.org/abs/1906.05392 "'Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian', Oymak et al 2019"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Implicit Regularization in Deep Matrix Factorization”, Arora et al 2019`{=html} {#arora-et-al-2019-section}

**[`Implicit Regularization in Deep Matrix Factorization`{=html}](https://arxiv.org/abs/1905.13655 "'Implicit Regularization in Deep Matrix Factorization', Arora et al 2019"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“MoGlow: Probabilistic and Controllable Motion Synthesis Using Normalizing Flows”, Henter et al 2019`{=html} {#henter-et-al-2019-section}

**[`MoGlow: Probabilistic and controllable motion synthesis using normalizing flows`{=html}](https://arxiv.org/abs/1905.06598 "'MoGlow: Probabilistic and controllable motion synthesis using normalizing flows', Henter et al 2019"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Fixup Initialization: Residual Learning Without Normalization”, Zhang et al 2019`{=html} {#zhang-et-al-2019-10-section}

**[`Fixup Initialization: Residual Learning Without Normalization`{=html}](https://arxiv.org/abs/1901.09321 "'Fixup Initialization: Residual Learning Without Normalization', Zhang et al 2019"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“SwitchNet: a Neural Network Model for Forward and Inverse Scattering Problems”, Khoo & Ying 2018`{=html} {#khoo-ying-2018-section}

**[`SwitchNet: a neural network model for forward and inverse scattering problems`{=html}](https://arxiv.org/abs/1810.09675 "'SwitchNet: a neural network model for forward and inverse scattering problems', Khoo & Ying 2018"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“A Jamming Transition from Under-Parameterization to Over-Parameterization Affects Loss Landscape and Generalization”, Spigler et al 2018`{=html} {#spigler-et-al-2018-section}

**[`A jamming transition from under-parameterization to over-parameterization affects loss landscape and generalization`{=html}](https://arxiv.org/abs/1810.09665 "'A jamming transition from under-parameterization to over-parameterization affects loss landscape and generalization', Spigler et al 2018"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Neural Arithmetic Logic Units”, Trask et al 2018`{=html} {#trask-et-al-2018-section}

**[`Neural Arithmetic Logic Units`{=html}](https://arxiv.org/abs/1808.00508#deepmind "'Neural Arithmetic Logic Units', Trask et al 2018"){.link-annotated
.id-not .include-annotation link-icon="deepmind" link-icon-type="svg"
link-icon-color="#4185f4"}**

## `“The Goldilocks Zone: Towards Better Understanding of Neural Network Loss Landscapes”, Fort & Scherlis 2018`{=html} {#fort-scherlis-2018-section}

**[`The Goldilocks zone: Towards better understanding of neural network loss landscapes`{=html}](https://arxiv.org/abs/1807.02581 "'The Goldilocks zone: Towards better understanding of neural network loss landscapes', Fort & Scherlis 2018"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Scalable Training of Artificial Neural Networks With Adaptive Sparse Connectivity Inspired by Network Science”, Mocanu et al 2018`{=html} {#mocanu-et-al-2018-section}

**[`Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science`{=html}](https://www.nature.com/articles/s41467-018-04316-3 "'Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science', Mocanu et al 2018"){.link-annotated
.id-not .include-annotation link-icon="n" link-icon-type="text"}**

## `“Deep Learning Generalizes Because the Parameter-Function Map Is Biased towards Simple Functions”, Valle-Pérez et al 2018`{=html} {#valle-pérez-et-al-2018-section}

**[`Deep learning generalizes because the parameter-function map is biased towards simple functions`{=html}](https://arxiv.org/abs/1805.08522 "'Deep learning generalizes because the parameter-function map is biased towards simple functions', Valle-Pérez et al 2018"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Bidirectional Learning for Robust Neural Networks”, Pontes-Filho & Liwicki 2018`{=html} {#pontes-filho-liwicki-2018-section}

**[`Bidirectional Learning for Robust Neural Networks`{=html}](https://arxiv.org/abs/1805.08006 "'Bidirectional Learning for Robust Neural Networks', Pontes-Filho & Liwicki 2018"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations”, Ciccone et al 2018`{=html} {#ciccone-et-al-2018-section}

**[`NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations`{=html}](https://arxiv.org/abs/1804.07209 "'NAIS-Net: Stable Deep Networks from Non-Autonomous Differential Equations', Ciccone et al 2018"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Large Scale Distributed Neural Network Training through Online Distillation”, Anil et al 2018`{=html} {#anil-et-al-2018-section}

**[`Large scale distributed neural network training through online distillation`{=html}](https://arxiv.org/abs/1804.03235#google "'Large scale distributed neural network training through online distillation', Anil et al 2018"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Meta-Learning Update Rules for Unsupervised Representation Learning”, Metz et al 2018`{=html} {#metz-et-al-2018-2-section}

**[`Meta-Learning Update Rules for Unsupervised Representation Learning`{=html}](https://arxiv.org/abs/1804.00222#google "'Meta-Learning Update Rules for Unsupervised Representation Learning', Metz et al 2018"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Learning and Memorization”, Chatterjee 2018`{=html} {#chatterjee-2018-section}

**[`Learning and Memorization`{=html}](https://openreview.net/forum?id=S1XFzYyPMw "'Learning and Memorization', Chatterjee 2018"){.link-annotated
.id-not .include-annotation link-icon="OR" link-icon-type="text,sans"
link-icon-color="#8c1b13"}**

## `“Repurposing High-Throughput Image Assays Enables Biological Activity Prediction for Drug Discovery”, Simm et al 2018`{=html} {#simm-et-al-2018-section}

**[`Repurposing High-Throughput Image Assays Enables Biological Activity Prediction for Drug Discovery`{=html}](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6031326/ "'Repurposing High-Throughput Image Assays Enables Biological Activity Prediction for Drug Discovery', Simm et al 2018"){.link-annotated
.id-not .include-annotation link-icon="nlm-ncbi" link-icon-type="svg"
link-icon-color="#20558a"}**

## `“Improving Palliative Care With Deep Learning”, An et al 2018`{=html} {#an-et-al-2018-section}

**[`Improving palliative care with deep learning`{=html}](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6290509/ "'Improving palliative care with deep learning', An et al 2018"){.link-annotated
.id-not .include-annotation link-icon="nlm-ncbi" link-icon-type="svg"
link-icon-color="#20558a"}**

## `“Learning to Play Chess With Minimal Lookahead and Deep Value Neural Networks”, Sabatelli 2017 (page 3)`{=html} {#sabatelli-2017-page-3-section}

**[`Learning to Play Chess with Minimal Lookahead and Deep Value Neural Networks`{=html}](/doc/reinforcement-learning/chess/2017-sabatelli.pdf#page=3 "'Learning to Play Chess with Minimal Lookahead and Deep Value Neural Networks', Sabatelli 2017 (page 3)"){.link-annotated
.id-not .include-annotation link-icon="pdf" link-icon-type="svg"
link-icon-color="#f40f02"}**

## `“Neural Collaborative Filtering”, He et al 2017`{=html} {#he-et-al-2017-1-section}

**[`Neural Collaborative Filtering`{=html}](https://arxiv.org/abs/1708.05031 "'Neural Collaborative Filtering', He et al 2017"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Sharp Models on Dull Hardware: Fast and Accurate Neural Machine Translation Decoding on the CPU”, Devlin 2017`{=html} {#devlin-2017-section}

**[`Sharp Models on Dull Hardware: Fast and Accurate Neural Machine Translation Decoding on the CPU`{=html}](https://arxiv.org/abs/1705.01991 "'Sharp Models on Dull Hardware: Fast and Accurate Neural Machine Translation Decoding on the CPU', Devlin 2017"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“The Shattered Gradients Problem: If Resnets Are the Answer, Then What Is the Question?”, Balduzzi et al 2017`{=html} {#balduzzi-et-al-2017-section}

**[`The Shattered Gradients Problem: If resnets are the answer, then what is the question?`{=html}](https://arxiv.org/abs/1702.08591 "'The Shattered Gradients Problem: If resnets are the answer, then what is the question?', Balduzzi et al 2017"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Gender-From-Iris or Gender-From-Mascara?”, Kuehlkamp et al 2017`{=html} {#kuehlkamp-et-al-2017-section}

**[`Gender-From-Iris or Gender-From-Mascara?`{=html}](https://arxiv.org/abs/1702.01304 "'Gender-From-Iris or Gender-From-Mascara?', Kuehlkamp et al 2017"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Skip Connections Eliminate Singularities”, Orhan & Pitkow 2017`{=html} {#orhan-pitkow-2017-section}

**[`Skip Connections Eliminate Singularities`{=html}](https://arxiv.org/abs/1701.09175 "'Skip Connections Eliminate Singularities', Orhan & Pitkow 2017"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Deep Information Propagation”, Schoenholz et al 2016`{=html} {#schoenholz-et-al-2016-section}

**[`Deep Information Propagation`{=html}](https://arxiv.org/abs/1611.01232 "'Deep Information Propagation', Schoenholz et al 2016"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Topology and Geometry of Half-Rectified Network Optimization”, Freeman & Bruna 2016`{=html} {#freeman-bruna-2016-section}

**[`Topology and Geometry of Half-Rectified Network Optimization`{=html}](https://arxiv.org/abs/1611.01540 "'Topology and Geometry of Half-Rectified Network Optimization', Freeman & Bruna 2016"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima”, Keskar et al 2016`{=html} {#keskar-et-al-2016-section}

**[`On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima`{=html}](https://arxiv.org/abs/1609.04836#intel "'On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima', Keskar et al 2016"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Decoupled Neural Interfaces Using Synthetic Gradients”, Jaderberg et al 2016`{=html} {#jaderberg-et-al-2016-section}

**[`Decoupled Neural Interfaces using Synthetic Gradients`{=html}](https://arxiv.org/abs/1608.05343#deepmind "'Decoupled Neural Interfaces using Synthetic Gradients', Jaderberg et al 2016"){.link-annotated
.id-not .include-annotation link-icon="deepmind" link-icon-type="svg"
link-icon-color="#4185f4"}**

## `“Learning to Optimize”, Li & Malik 2016`{=html} {#li-malik-2016-section}

**[`Learning to Optimize`{=html}](https://arxiv.org/abs/1606.01885 "'Learning to Optimize', Li & Malik 2016"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Do Deep Convolutional Nets Really Need to Be Deep and Convolutional?”, Urban et al 2016`{=html} {#urban-et-al-2016-section}

**[`Do Deep Convolutional Nets Really Need to be Deep and Convolutional?`{=html}](https://arxiv.org/abs/1603.05691 "'Do Deep Convolutional Nets Really Need to be Deep and Convolutional?', Urban et al 2016"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Network Morphism”, Wei et al 2016`{=html} {#wei-et-al-2016-section}

**[`Network Morphism`{=html}](https://arxiv.org/abs/1603.01670 "'Network Morphism', Wei et al 2016"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Adding Gradient Noise Improves Learning for Very Deep Networks”, Neelakantan et al 2015`{=html} {#neelakantan-et-al-2015-section}

**[`Adding Gradient Noise Improves Learning for Very Deep Networks`{=html}](https://arxiv.org/abs/1511.06807#google "'Adding Gradient Noise Improves Learning for Very Deep Networks', Neelakantan et al 2015"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“How Far Can We Go without Convolution: Improving Fully-Connected Networks”, Lin et al 2015`{=html} {#lin-et-al-2015-section}

**[`How far can we go without convolution: Improving fully-connected networks`{=html}](https://arxiv.org/abs/1511.02580 "'How far can we go without convolution: Improving fully-connected networks', Lin et al 2015"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“BinaryConnect: Training Deep Neural Networks With Binary Weights during Propagations”, Courbariaux et al 2015`{=html} {#courbariaux-et-al-2015-section}

**[`BinaryConnect: Training Deep Neural Networks with binary weights during propagations`{=html}](https://arxiv.org/abs/1511.00363 "'BinaryConnect: Training Deep Neural Networks with binary weights during propagations', Courbariaux et al 2015"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Tensorizing Neural Networks”, Novikov et al 2015`{=html} {#novikov-et-al-2015-section}

**[`Tensorizing Neural Networks`{=html}](https://arxiv.org/abs/1509.06569 "'Tensorizing Neural Networks', Novikov et al 2015"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“A Neural Attention Model for Abstractive Sentence Summarization”, Rush et al 2015`{=html} {#rush-et-al-2015-section}

**[`A Neural Attention Model for Abstractive Sentence Summarization`{=html}](https://arxiv.org/abs/1509.00685#facebook "'A Neural Attention Model for Abstractive Sentence Summarization', Rush et al 2015"){.link-annotated
.id-not .include-annotation link-icon="facebook" link-icon-type="svg"
link-icon-color="#1877f2"}**

## `“Deep Neural Networks for Large Vocabulary Handwritten Text Recognition”, Bluche 2015`{=html} {#bluche-2015-section}

**[`Deep Neural Networks for Large Vocabulary Handwritten Text Recognition`{=html}](/doc/ai/nn/fully-connected/2015-bluche.pdf "'Deep Neural Networks for Large Vocabulary Handwritten Text Recognition', Bluche 2015"){.link-annotated
.id-not .include-annotation link-icon="pdf" link-icon-type="svg"
link-icon-color="#f40f02"}**

## `“In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning”, Neyshabur et al 2014`{=html} {#neyshabur-et-al-2014-section}

**[`In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning`{=html}](https://arxiv.org/abs/1412.6614 "'In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning', Neyshabur et al 2014"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“The Loss Surfaces of Multilayer Networks”, Choromanska et al 2014`{=html} {#choromanska-et-al-2014-section}

**[`The Loss Surfaces of Multilayer Networks`{=html}](https://arxiv.org/abs/1412.0233 "'The Loss Surfaces of Multilayer Networks', Choromanska et al 2014"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“On the Number of Linear Regions of Deep Neural Networks”, Montúfar et al 2014`{=html} {#montúfar-et-al-2014-section}

**[`On the Number of Linear Regions of Deep Neural Networks`{=html}](https://arxiv.org/abs/1402.1869 "'On the Number of Linear Regions of Deep Neural Networks', Montúfar et al 2014"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Do Deep Nets Really Need to Be Deep?”, Ba & Caruana 2013`{=html} {#ba-caruana-2013-section}

**[`Do Deep Nets Really Need to be Deep?`{=html}](https://arxiv.org/abs/1312.6184 "'Do Deep Nets Really Need to be Deep?', Ba & Caruana 2013"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“On the Number of Response Regions of Deep Feed Forward Networks With Piece-Wise Linear Activations”, Pascanu et al 2013`{=html} {#pascanu-et-al-2013-section}

**[`On the number of response regions of deep feed forward networks with piece-wise linear activations`{=html}](https://arxiv.org/abs/1312.6098 "'On the number of response regions of deep feed forward networks with piece-wise linear activations', Pascanu et al 2013"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Network In Network”, Lin et al 2013`{=html} {#lin-et-al-2013-section}

**[`Network In Network`{=html}](https://arxiv.org/abs/1312.4400 "'Network In Network', Lin et al 2013"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Deep Big Multilayer Perceptrons for Digit Recognition”, Cireşan et al 2012`{=html} {#cireşan-et-al-2012-2-section}

**[`Deep Big Multilayer Perceptrons for Digit Recognition`{=html}](/doc/ai/nn/fully-connected/2012-ciresan.pdf#schmidhuber "'Deep Big Multilayer Perceptrons for Digit Recognition', Cireşan et al 2012"){.link-annotated
.id-not .include-annotation link-icon="SMDH"
link-icon-type="text,quad,sans"}**

## `“Natural Language Processing (Almost) from Scratch”, Collobert et al 2011`{=html} {#collobert-et-al-2011-section}

**[`Natural Language Processing (Almost) from Scratch`{=html}](/doc/psychology/linguistics/2011-collobert.pdf "'Natural Language Processing (Almost) from Scratch', Collobert et al 2011"){.link-annotated
.id-not .include-annotation link-icon="pdf" link-icon-type="svg"
link-icon-color="#f40f02"}**

## `“Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition”, Ciresan et al 2010`{=html} {#ciresan-et-al-2010-section}

**[`Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition`{=html}](https://arxiv.org/abs/1003.0358#schmidhuber "'Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition', Ciresan et al 2010"){.link-annotated
.id-not .include-annotation link-icon="SMDH"
link-icon-type="text,quad,sans"}**

## `“Compositional Pattern Producing Networks: A Novel Abstraction of Development”, Stanley 2007`{=html} {#stanley-2007-section}

**[`Compositional pattern producing networks: A novel abstraction of development`{=html}](/doc/ai/nn/fully-connected/2007-stanley.pdf "'Compositional pattern producing networks: A novel abstraction of development', Stanley 2007"){.link-annotated
.id-not .include-annotation link-icon="pdf" link-icon-type="svg"
link-icon-color="#f40f02"}**

## `“Extraction De Séquences Numériques Dans Des Documents Manuscrits Quelconques”, Chatelain 2006`{=html} {#chatelain-2006-section}

**[`Extraction de séquences numériques dans des documents manuscrits quelconques`{=html}](/doc/ai/nn/fully-connected/2006-chatelain.pdf "'Extraction de séquences numériques dans des documents manuscrits quelconques', Chatelain 2006"){.link-annotated
.id-not .include-annotation link-icon="pdf" link-icon-type="svg"
link-icon-color="#f40f02"}**

## `“Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis”, Simard et al 2003`{=html} {#simard-et-al-2003-section}

**[`Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis`{=html}](/doc/ai/nn/fully-connected/2003-simard.pdf#microsoft "'Best Practices for Convolutional Neural Networks Applied to Visual Document Analysis', Simard et al 2003"){.link-annotated
.id-not .include-annotation link-icon="MS"
link-icon-type="text,sans,italic"}**

## `“NEAT: Evolving Neural Networks through Augmenting Topologies”, Stanley & Miikkulainen 2002`{=html} {#stanley-miikkulainen-2002-section}

**[`NEAT: Evolving Neural Networks through Augmenting Topologies`{=html}](/doc/reinforcement-learning/exploration/2002-stanley.pdf "'NEAT: Evolving Neural Networks through Augmenting Topologies', Stanley & Miikkulainen 2002"){.link-annotated
.id-not .include-annotation link-icon="pdf" link-icon-type="svg"
link-icon-color="#f40f02"}**

## `“DARPA and the Quest for Machine Intelligence, 1983–1993”, Roland & Shiman 2002`{=html} {#roland-shiman-2002-section}

**[`DARPA and the Quest for Machine Intelligence, 1983–1993`{=html}](/doc/cs/hardware/2002-roland-strategiccomputing-darpaandthequestformachineintelligence19831993.pdf "'DARPA and the Quest for Machine Intelligence, 1983–1993', Roland & Shiman 2002"){.link-annotated
.id-not .include-annotation link-icon="pdf" link-icon-type="svg"
link-icon-color="#f40f02"}**

## `“Quantitative Analysis of Multivariate Data Using Artificial Neural Networks: A Tutorial Review and Applications to the Deconvolution of Pyrolysis Mass Spectra”, Goodacre et al 1996`{=html} {#goodacre-et-al-1996-section}

**[`Quantitative Analysis of Multivariate Data Using Artificial Neural Networks: A Tutorial Review and Applications to the Deconvolution of Pyrolysis Mass Spectra`{=html}](/doc/ai/nn/fully-connected/1996-goodacre.pdf "'Quantitative Analysis of Multivariate Data Using Artificial Neural Networks: A Tutorial Review and Applications to the Deconvolution of Pyrolysis Mass Spectra', Goodacre et al 1996"){.link-annotated
.id-not .include-annotation link-icon="pdf" link-icon-type="svg"
link-icon-color="#f40f02"}**

## `“Statistical Mechanics of Generalization”, Opper & Kinzel 1996`{=html} {#opper-kinzel-1996-section}

**[`Statistical Mechanics of Generalization`{=html}](/doc/ai/nn/fully-connected/1996-opper.pdf "'Statistical Mechanics of Generalization', Opper & Kinzel 1996"){.link-annotated
.id-not .include-annotation link-icon="pdf" link-icon-type="svg"
link-icon-color="#f40f02"}**

## `“On the Ability of the Optimal Perceptron to Generalize”, Opper et al 1990`{=html} {#opper-et-al-1990-section}

**[`On the ability of the optimal perceptron to generalize`{=html}](/doc/ai/nn/fully-connected/1990-opper.pdf "'On the ability of the optimal perceptron to generalize', Opper et al 1990"){.link-annotated
.id-not .include-annotation link-icon="pdf" link-icon-type="svg"
link-icon-color="#f40f02"}**

## `“Learning To Tell Two Spirals Apart”, Lang & Witbrock 1988`{=html} {#lang-witbrock-1988-section}

**[`Learning To Tell Two Spirals Apart`{=html}](/doc/ai/nn/fully-connected/1988-lang.pdf "'Learning To Tell Two Spirals Apart', Lang & Witbrock 1988"){.link-annotated
.id-not .include-annotation link-icon="pdf" link-icon-type="svg"
link-icon-color="#f40f02"}**

## `“Learning Internal Representations by Error Propagation”, Rumelhart et al 1986`{=html} {#rumelhart-et-al-1986-backprop-section}

**[`Learning Internal Representations by Error Propagation`{=html}](/doc/ai/nn/fully-connected/1986-rumelhart.pdf "'Learning Internal Representations by Error Propagation', Rumelhart et al 1986"){.link-annotated
.id-not .include-annotation link-icon="pdf" link-icon-type="svg"
link-icon-color="#f40f02"}**

## `“Neural Networks and Physical Systems With Emergent Collective Computational Abilities”, Hopfield 1982`{=html} {#hopfield-1982-section}

**[`Neural networks and physical systems with emergent collective computational abilities`{=html}](https://www.pnas.org/doi/10.1073/pnas.79.8.2554 "'Neural networks and physical systems with emergent collective computational abilities', Hopfield 1982"){.link-annotated-partial
.id-not .include-annotation link-icon="PNAS" link-icon-type="text,quad"
link-icon-color="#1f75b9"}**

## `“Can You Reverse Engineer Our Neural Network? <span class=editorial>[Encoding MD5 into a MLP]</span>”`{=html} {#_5Cd-SM0K-section}

**[`Can you reverse engineer our neural network? <span class=editorial>[encoding MD5 into a MLP]</span>`{=html}](https://blog.janestreet.com/can-you-reverse-engineer-our-neural-network/ "Can you reverse engineer our neural network? <span class=editorial>[encoding MD5 into a MLP]</span>"){.link-annotated-partial
.id-not .include-annotation}**

## `“Technion-Cs-Nlp/llm-Arithmetic-Heuristics”`{=html} {#_WrubH-Yy-section}

**[`technion-cs-nlp/llm-arithmetic-heuristics`{=html}](https://github.com/technion-cs-nlp/llm-arithmetic-heuristics "technion-cs-nlp/llm-arithmetic-heuristics"){.link-annotated-partial
.id-not .include-annotation link-icon="github" link-icon-type="svg"}**

## `“Research Log: Monet/PEER Sparse Experts”`{=html} {#_ANcBzH1V-section}

**[`Research Log: Monet/PEER sparse experts`{=html}](https://www.lesswrong.com/posts/uDrDbeM3CvWYunFkt/research-log-monet-peer-sparse-experts "Research Log: Monet/PEER sparse experts"){.link-annotated-partial
.id-not .include-annotation link-icon="LW" link-icon-type="text"
link-icon-color="#7faf83"}**

## Sort By Magic

::: {demo-type="sort-by-magic-preface"}
Annotations sorted by machine learning into [inferred \'tags\'](/design#future-tag-features){#gwern-design--future-tag-features
.link-annotated-partial}. This provides an alternative way to browse: instead of by *date* order, one can browse in *topic* order. The \'sorted\' list has been automatically clustered into multiple sections & auto-labeled for easier browsing.

Beginning with the newest annotation, it uses the embedding of each annotation to attempt to create a list of nearest-neighbor annotations, creating a progression of topics. For more details, see the link.
:::

### `fully-connected` {.collapse title="Machine-generated tag name for the following cluster of links."}

[\[see previous entry\]](#gwern-doc-ai-nn-fully-connected-abstract-section){.include
.include-even-when-collapsed}

### `machine-intelligence` {.collapse title="Machine-generated tag name for the following cluster of links."}

[\[see previous entry\]](#sabatelli-2017-page-3-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#roland-shiman-2002-section){.include
.include-even-when-collapsed}

### `predictive-coding` {.collapse title="Machine-generated tag name for the following cluster of links."}

[\[see previous entry\]](#innocenti-et-al-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#bao-et-al-2020-section){.include
.include-even-when-collapsed}

### `palliative-ai` {.collapse title="Machine-generated tag name for the following cluster of links."}

[\[see previous entry\]](#fratantonio-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#an-et-al-2018-section){.include
.include-even-when-collapsed}

### `user-representation` {.collapse title="Machine-generated tag name for the following cluster of links."}

[\[see previous entry\]](#shin-et-al-2021-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#giannini-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#chatterjee-et-al-2023-section){.include
.include-even-when-collapsed}

### `robust-training` {.collapse title="Machine-generated tag name for the following cluster of links."}

[\[see previous entry\]](#wenzel-et-al-2020-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#mindermann-et-al-2022-section){.include
.include-even-when-collapsed}

### `mlp-architecture` {.collapse title="Machine-generated tag name for the following cluster of links."}

[\[see previous entry\]](#murahari-et-al-2023-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#chen-et-al-2023-14-section){.include
.include-even-when-collapsed}

### `reaction-classification` {.collapse title="Machine-generated tag name for the following cluster of links."}

[\[see previous entry\]](#kuehlkamp-et-al-2017-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#goodacre-et-al-1996-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#burés-larrosa-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#levin-et-al-2022-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#simm-et-al-2018-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#demetci-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#lieberum-et-al-2023-section){.include
.include-even-when-collapsed}

### `collaborative-filtering` {.collapse title="Machine-generated tag name for the following cluster of links."}

[\[see previous entry\]](#huang-hoefler-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#naumov-et-al-2020-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#wang-et-al-2020-05-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#he-et-al-2017-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#zhang-et-al-2019d-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#henter-et-al-2019-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#kucherenko-et-al-2020-section){.include
.include-even-when-collapsed}

### `image-generation, explainability-challenges, cryptanalytic-extraction, neural-representation, model-verification, image-generation` {.collapse title="Machine-generated tag name for the following cluster of links."}

[\[see previous entry\]](#polaczek-et-al-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#yan-et-al-2023-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#li-et-al-2024-02-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#ciccone-et-al-2018-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#wei-et-al-2016-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#khoo-ying-2018-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#shamir-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#carlini-et-al-2020-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#merrer-tredan-2019-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#south-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#bunel-et-al-2024-section){.include
.include-even-when-collapsed}

### `generalization-bias` {.collapse title="Machine-generated tag name for the following cluster of links."}

[\[see previous entry\]](#rumelhart-et-al-1986-backprop-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#opper-kinzel-1996-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#opper-et-al-1990-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#valle-pérez-et-al-2018-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#teney-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#chiang-et-al-2023-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#oymak-et-al-2019-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#arora-et-al-2019-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#neyshabur-et-al-2014-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#choromanska-et-al-2014-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#fort-scherlis-2018-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#freeman-bruna-2016-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#zhang-et-al-2021-02-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#martens-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#orhan-pitkow-2017-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#balduzzi-et-al-2017-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#schoenholz-et-al-2016-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#fan-et-al-2020-4-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#montúfar-et-al-2014-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#pascanu-et-al-2013-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#villani-schoots-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#hänni-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#adler-shavit-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#kratsios-et-al-2026-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#kocsis-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#dascoli-et-al-2019-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#ingrosso-goldt-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#neyshabur-2020-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#pogodin-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#hu-rostami-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#lian-et-al-2021-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#yu-et-al-2021-s2mlpv2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#yu-et-al-2021-s2mlp-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#tolstikhin-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#li-et-al-2021-5-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#ding-et-al-2021-3-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#hou-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#guo-et-al-2021-hiremlp-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#chen-et-al-2021-cyclemlp-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#zhang-et-al-2021-morphmlp-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#tang-et-al-2021-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#melas-kyriazi-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#bozic-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#zhai-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#tay-et-al-2020-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#lee-thorp-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#tancik-et-al-2020-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#bordelon-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#bahri-et-al-2021-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#michaud-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#nam-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#sandbrink-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#mitchell-et-al-2023-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#mocanu-et-al-2018-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#lou-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#yu-et-al-2022-5-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#fusco-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#mai-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#liu-et-al-2021-gmlp-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#nie-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#chen-et-al-2021-10-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#trockman-kolter-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#yu-et-al-2021-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#zheng-et-al-2022-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#qiu-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#cazenavette-guevara-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#choe-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#tatsunami-taki-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#zhao-et-al-2021-4-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#bachmann-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#greydanus-2020-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#power-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#liu-et-al-2022-10-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#liu-et-al-2022-18-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#miller-et-al-2023-3-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#power-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#doshi-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#doshi-et-al-2024-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#stander-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#chughtai-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#zhong-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#shinnick-et-al-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#garcia-et-al-2026-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#dugan-et-al-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#pipek-et-al-2020-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#pires-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#he-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#hassid-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#dong-et-al-2021-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#dherin-et-al-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#tong-pehlevan-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#chang-et-al-2024-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#zhao-et-al-2024-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#metz-et-al-2018-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#jaderberg-et-al-2016-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#ren-et-al-2022-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#pontes-filho-liwicki-2018-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#neelakantan-et-al-2015-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#keskar-et-al-2016-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#abnar-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#tay-et-al-2022-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#grinsztajn-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#kadra-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#rubachev-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#cholakov-kolev-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#huang-et-al-2020-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#liu-et-al-2021-swintranformer-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#wang-et-al-2022-22-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#csordás-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#lee-thorp-ainslie-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#he-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#anil-et-al-2018-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#urban-et-al-2016-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#ba-caruana-2013-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#morcos-tian-2019-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#pellegrini-biroli-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#ramanujan-et-al-2019-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#qiu-suda-2020-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#benzing-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#zhao-et-al-2021-3-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#zhang-et-al-2019-10-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#bachlechner-et-al-2020-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#lin-et-al-2015-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#touvron-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#tu-et-al-2022-3-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#gao-et-al-2021-3-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#lin-et-al-2013-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#courbariaux-et-al-2015-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#novikov-et-al-2015-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#li-et-al-2022-11-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#burns-fukai-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#erkoç-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#poole-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#lin-et-al-2022-04-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#jain-et-al-2021-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#babu-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#mildenhall-et-al-2020-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#reiser-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#duckworth-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#takikawa-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#müller-et-al-2021-4-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#rückert-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#chugunov-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#li-et-al-2020-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#sitzmann-et-al-2020-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#xie-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#anokhin-et-al-2020-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#stanley-2007-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#stanley-miikkulainen-2002-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#li-malik-2016-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#peebles-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#lerousseau-summerfield-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#loshchilov-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#sun-iyyer-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#xu-et-al-2021-6-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#yu-et-al-2022-4-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#khalitov-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#sakuma-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#tae-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#devlin-2017-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#chang-bergen-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#badger-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#rush-et-al-2015-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#collobert-et-al-2011-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#bluche-2015-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#chatelain-2006-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#cireşan-et-al-2012-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#ciresan-et-al-2010-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#simard-et-al-2003-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#lang-witbrock-1988-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#sinha-et-al-2020-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#ota-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#nauman-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#lee-et-al-2024-5-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#ota-et-al-2020-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#florence-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#finzi-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#alet-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#gruver-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#fischbacher-et-al-2020-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#petersen-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#trask-et-al-2018-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#nikankin-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#hanna-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#pearce-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#aytekin-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#filan-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#zhang-wang-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#lee-et-al-2024-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#wu-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#spigler-et-al-2018-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#mirzadeh-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#chatterjee-2018-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#behrouz-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#wang-et-al-2020-09-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#giannou-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#gardner-et-al-2025-section){.include
.include-even-when-collapsed}

## Wikipedia (2) {#titled-links-wikipedia .collapse}

1.  **[`<em>Bear</em> (novel)`{=html}](https://en.wikipedia.org/wiki/Bear_(novel) "<em>Bear</em> (novel)"){.id-not
    .include-annotation link-icon="wikipedia" link-icon-type="svg"}**

    ::: aux-links-transclude-file
    [`https://en.wikipedia.org/wiki/Bear_(novel)`](https://en.wikipedia.org/wiki/Bear_(novel) "<em>Bear</em> (novel)"){.id-not
    .include-content include-template="$annotationFileIncludeTemplate"}
    :::

2.  **[`Hebbian learning`{=html}](https://en.wikipedia.org/wiki/Hebbian_learning "Hebbian learning"){.id-not
    .include-annotation link-icon="wikipedia" link-icon-type="svg"}**

    ::: aux-links-transclude-file
    [`https://en.wikipedia.org/wiki/Hebbian_learning`](https://en.wikipedia.org/wiki/Hebbian_learning "Hebbian learning"){.id-not
    .include-content include-template="$annotationFileIncludeTemplate"}
    :::

# Miscellaneous

-   **[`<code>/doc/ai/nn/fully-connected/2025-04-07-gwern-ifnn-architecture.svg</code>`{=html}](/doc/ai/nn/fully-connected/2025-04-07-gwern-ifnn-architecture.svg "Gwern 2025"){.link-annotated-partial
    .id-not .include-annotation link-icon="image" link-icon-type="svg"
    link-icon-color="#ffb338"}**

-   **[`<code>/doc/ai/nn/fully-connected/2025-polaczek-figure3-neuralsvgarchitectureofmlppredictingoutputspershape.jpg</code>`{=html}](/doc/ai/nn/fully-connected/2025-polaczek-figure3-neuralsvgarchitectureofmlppredictingoutputspershape.jpg){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2024-chang-figure7-mlpandattentionheadsbypredictioncorrectnessshowsbothcanworkforiclmetalearning.png</code>`{=html}](/doc/ai/nn/fully-connected/2024-chang-figure7-mlpandattentionheadsbypredictioncorrectnessshowsbothcanworkforiclmetalearning.png){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2024-zhao-figure1-llmshavemuchrougherdecisionboundariesthanmlpsorsvmsordecisiontrees.png</code>`{=html}](/doc/ai/nn/fully-connected/2024-zhao-figure1-llmshavemuchrougherdecisionboundariesthanmlpsorsvmsordecisiontrees.png){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2023-08-17-gwern-aunn-architecture.png</code>`{=html}](/doc/ai/nn/fully-connected/2023-08-17-gwern-aunn-architecture.png "Gwern 2023"){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2023-08-17-gwern-aunn-architecture.svg</code>`{=html}](/doc/ai/nn/fully-connected/2023-08-17-gwern-aunn-architecture.svg "Gwern 2023"){.link-annotated-partial
    .id-not .include-annotation link-icon="image" link-icon-type="svg"
    link-icon-color="#ffb338"}**

-   **[`<code>/doc/ai/nn/fully-connected/2023-bachmann-figure1-mlpcomputescalingoncifar100.jpg</code>`{=html}](/doc/ai/nn/fully-connected/2023-bachmann-figure1-mlpcomputescalingoncifar100.jpg){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2023-bachmann-figure10-dataaugmentationinducesmoresparselocalfeaturesinfirstlayermlpweights.png</code>`{=html}](/doc/ai/nn/fully-connected/2023-bachmann-figure10-dataaugmentationinducesmoresparselocalfeaturesinfirstlayermlpweights.png){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2023-bachmann-figure4-mlpsscalewellwithincreasingbatchsize.jpg</code>`{=html}](/doc/ai/nn/fully-connected/2023-bachmann-figure4-mlpsscalewellwithincreasingbatchsize.jpg){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2023-bachmann-figure5-scalingofmlpsoncifar10andimagenet1k.png</code>`{=html}](/doc/ai/nn/fully-connected/2023-bachmann-figure5-scalingofmlpsoncifar10andimagenet1k.png){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2023-bachmann-figure6-powerlawincifar100losswhenconstrainingparametersordatasetsize.jpg</code>`{=html}](/doc/ai/nn/fully-connected/2023-bachmann-figure6-powerlawincifar100losswhenconstrainingparametersordatasetsize.jpg){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2023-bachmann-figure7-suprachinchilladatascalingformlpsoncifar100loss.jpg</code>`{=html}](/doc/ai/nn/fully-connected/2023-bachmann-figure7-suprachinchilladatascalingformlpsoncifar100loss.jpg){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2023-bachmann-figure8-mlparchitectureablations.png</code>`{=html}](/doc/ai/nn/fully-connected/2023-bachmann-figure8-mlparchitectureablations.png){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2023-mitchell-figure2-2dvisualizationofannbeingexpandedbysenntobetterapproximatetheline.png</code>`{=html}](/doc/ai/nn/fully-connected/2023-mitchell-figure2-2dvisualizationofannbeingexpandedbysenntobetterapproximatetheline.png){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2023-mitchell-figure3-visualizationofsennlossoveradditionsforhalfmoonstoydataset.jpg</code>`{=html}](/doc/ai/nn/fully-connected/2023-mitchell-figure3-visualizationofsennlossoveradditionsforhalfmoonstoydataset.jpg){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2022-grinsztajn-figure10-treesvsneuralnetson3regressiontasksusingnumericalfeaturesonmediumvslargedatasets.png</code>`{=html}](/doc/ai/nn/fully-connected/2022-grinsztajn-figure10-treesvsneuralnetson3regressiontasksusingnumericalfeaturesonmediumvslargedatasets.png){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2022-grinsztajn-figure11-treesvsneuralnetson2classificationtasksusingallfeaturesonmediumvslargedatasets.png</code>`{=html}](/doc/ai/nn/fully-connected/2022-grinsztajn-figure11-treesvsneuralnetson2classificationtasksusingallfeaturesonmediumvslargedatasets.png){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2022-grinsztajn-figure12-treesvsneuralnetson5regressiontasksusingallfeaturesonmediumvslargedatasets.png</code>`{=html}](/doc/ai/nn/fully-connected/2022-grinsztajn-figure12-treesvsneuralnetson5regressiontasksusingallfeaturesonmediumvslargedatasets.png){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2022-hassid-figure2-contributionoftransformerattentionwhenablatedtomlbenchmarkperformance.jpg</code>`{=html}](/doc/ai/nn/fully-connected/2022-hassid-figure2-contributionoftransformerattentionwhenablatedtomlbenchmarkperformance.jpg){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2021-muller-figure7-fullyfusedfullyconnectednetworkspeedupongpu.jpg</code>`{=html}](/doc/ai/nn/fully-connected/2021-muller-figure7-fullyfusedfullyconnectednetworkspeedupongpu.jpg){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2021-ni-figure2-vilmlpvstransformerbypretrainingdatafraction.png</code>`{=html}](/doc/ai/nn/fully-connected/2021-ni-figure2-vilmlpvstransformerbypretrainingdatafraction.png){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2021-ni-figure3-scalingofmlpvilvsmlpviltinyattentionvstransformeronvisualquestionansweringaccuracy.png</code>`{=html}](/doc/ai/nn/fully-connected/2021-ni-figure3-scalingofmlpvilvsmlpviltinyattentionvstransformeronvisualquestionansweringaccuracy.png){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2021-power-figure1-grokkinglearningcurves.jpg</code>`{=html}](/doc/ai/nn/fully-connected/2021-power-figure1-grokkinglearningcurves.jpg){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2021-power-poster.png#openai</code>`{=html}](/doc/ai/nn/fully-connected/2021-power-poster.png#openai){.link-annotated-partial
    .id-not .include-annotation link-icon="openai"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2021-zhao-figure4-mlpsoverfitbutcanberegularizedbyweightsharingandmultistagearchitecture.jpg</code>`{=html}](/doc/ai/nn/fully-connected/2021-zhao-figure4-mlpsoverfitbutcanberegularizedbyweightsharingandmultistagearchitecture.jpg){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2021-zhao-multistagespachframeworkforcomparingmodularblocksofmlpsvscnnsvstransformers.png</code>`{=html}](/doc/ai/nn/fully-connected/2021-zhao-multistagespachframeworkforcomparingmodularblocksofmlpsvscnnsvstransformers.png){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2020-ota-figure1-densenetmlpschematicarchitecture.jpg</code>`{=html}](/doc/ai/nn/fully-connected/2020-ota-figure1-densenetmlpschematicarchitecture.jpg){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2020-ota-figure2-overallofenetarchitectureshematic.png</code>`{=html}](/doc/ai/nn/fully-connected/2020-ota-figure2-overallofenetarchitectureshematic.png){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2014-montufar-figure1-binaryclassificationdecisionboundaryofshallowvsdeepneuralnetworkshowingdeeperequalssmoother.png</code>`{=html}](/doc/ai/nn/fully-connected/2014-montufar-figure1-binaryclassificationdecisionboundaryofshallowvsdeepneuralnetworkshowingdeeperequalssmoother.png){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2014-pascanu-figure2-topologyofdeepnetworksinfoldingaroundaxislayerbylayer.png</code>`{=html}](/doc/ai/nn/fully-connected/2014-pascanu-figure2-topologyofdeepnetworksinfoldingaroundaxislayerbylayer.png){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/2014-pascanu-figure3-spacefoldingof2dspaceassheetofpapermodeledbydeepneuralnetworks.png</code>`{=html}](/doc/ai/nn/fully-connected/2014-pascanu-figure3-spacefoldingof2dspaceassheetofpapermodeledbydeepneuralnetworks.png){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/fully-connected/1988-lang-figure3-densenetresidualarchitectureforneuralnetsolvingswissspiralproblem.jpg</code>`{=html}](/doc/ai/nn/fully-connected/1988-lang-figure3-densenetresidualarchitectureforneuralnetsolvingswissspiralproblem.jpg){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>https://colab.research.google.com/github/murphyka/ml_colabs/blob/main/Simple_MLP_Visualization.ipynb</code>`{=html}](https://colab.research.google.com/github/murphyka/ml_colabs/blob/main/Simple_MLP_Visualization.ipynb){.id-not
    .include-annotation link-icon="alphabet" link-icon-type="svg"
    link-icon-color="#4285f4"}**

-   **[`<code>https://cpldcpu.wordpress.com/2024/04/24/implementing-neural-networks-on-the-10-cent-risc-v-mcu-without-multiplier/</code>`{=html}](https://cpldcpu.wordpress.com/2024/04/24/implementing-neural-networks-on-the-10-cent-risc-v-mcu-without-multiplier/){.link-annotated-partial
    .id-not .include-annotation}**

-   **[`<code>https://cprimozic.net/blog/reverse-engineering-a-small-neural-network/</code>`{=html}](https://cprimozic.net/blog/reverse-engineering-a-small-neural-network/){.link-annotated-partial
    .id-not .include-annotation}**

-   **[`<code>https://fourmilab.ch/documents/commodore/BrainSim/</code>`{=html}](https://fourmilab.ch/documents/commodore/BrainSim/){.link-annotated-partial
    .id-not .include-annotation}**

-   **[`<code>https://github.com/thomasahle/fastchess</code>`{=html}](https://github.com/thomasahle/fastchess){.link-annotated-partial
    .id-not .include-annotation link-icon="github"
    link-icon-type="svg"}**

-   **[`<code>https://jmlr.org/papers/volume3/bengio03a/bengio03a.pdf</code>`{=html}](https://jmlr.org/papers/volume3/bengio03a/bengio03a.pdf){.link-annotated-partial
    .id-not .include-annotation link-icon="pdf" link-icon-type="svg"
    link-icon-color="#f40f02"}**

-   **[`<code>https://thenumb.at/Neural-Graphics/</code>`{=html}](https://thenumb.at/Neural-Graphics/){.link-annotated-partial
    .id-not .include-annotation}**

-   **[`<code>https://transformer-circuits.pub/2024/jan-update/index.html#mnist-sparse</code>`{=html}](https://transformer-circuits.pub/2024/jan-update/index.html#mnist-sparse){.id-not
    .include-annotation link-icon="anthropic" link-icon-type="svg"
    link-icon-color="#d4a27f"}**

    ::: aux-links-transclude-file
    ::: collapse
    **View HTML**:

    [`https://transformer-circuits.pub/2024/jan-update/index.html#mnist-sparse`](https://transformer-circuits.pub/2024/jan-update/index.html#mnist-sparse){.id-not
    .link-annotated-not .include-content .include-lazy
    link-icon="anthropic" link-icon-type="svg"
    link-icon-color="#d4a27f"}
    :::
    :::

-   **[`<code>https://www.lesswrong.com/posts/7fxusXdkMNmAhkAfc/finding-sparse-linear-connections-between-features-in-llms</code>`{=html}](https://www.lesswrong.com/posts/7fxusXdkMNmAhkAfc/finding-sparse-linear-connections-between-features-in-llms){.link-annotated-partial
    .id-not .include-annotation link-icon="LW" link-icon-type="text"
    link-icon-color="#7faf83"}**

-   **[`<code>https://www.lesswrong.com/posts/K7AyY8LMrcKhwfbyj/no-really-attention-is-all-you-need-attention-can-do</code>`{=html}](https://www.lesswrong.com/posts/K7AyY8LMrcKhwfbyj/no-really-attention-is-all-you-need-attention-can-do){.link-annotated-partial
    .id-not .include-annotation link-icon="LW" link-icon-type="text"
    link-icon-color="#7faf83"}**

-   **[`<code>https://www.lesswrong.com/posts/YmkjnWtZGLbHRbzrP/transcoders-enable-fine-grained-interpretable-circuit</code>`{=html}](https://www.lesswrong.com/posts/YmkjnWtZGLbHRbzrP/transcoders-enable-fine-grained-interpretable-circuit){.link-annotated-partial
    .id-not .include-annotation link-icon="LW" link-icon-type="text"
    link-icon-color="#7faf83"}**

-   **[`<code>https://www.lesswrong.com/postsiGuwZTHWb6DFY3sKB/fact-finding-attempting-to-reverse-engineer-factual-recall</code>`{=html}](https://www.lesswrong.com/postsiGuwZTHWb6DFY3sKB/fact-finding-attempting-to-reverse-engineer-factual-recall){.link-annotated-partial
    .id-not .include-annotation link-icon="LW" link-icon-type="text"
    link-icon-color="#7faf83"}**

-   **[`<code>https://www.lesswrong.com/s/5omSW4wNKbEvYsyje/p/GpSzShaaf8po4rcmA</code>`{=html}](https://www.lesswrong.com/s/5omSW4wNKbEvYsyje/p/GpSzShaaf8po4rcmA){.id-not
    .include-annotation link-icon="LW" link-icon-type="text"
    link-icon-color="#7faf83"}**

    ::: aux-links-transclude-file
    ::: collapse
    **View External Link**:

    [`https://www.lesswrong.com/s/5omSW4wNKbEvYsyje/p/GpSzShaaf8po4rcmA`](https://www.lesswrong.com/s/5omSW4wNKbEvYsyje/p/GpSzShaaf8po4rcmA){.id-not
    .link-annotated-not .include-content .include-lazy link-icon="LW"
    link-icon-type="text" link-icon-color="#7faf83"}
    :::
    :::

-   **[`<code>https://x.com/Bourdain/status/153513491200872448</code>`{=html}](https://x.com/Bourdain/status/153513491200872448 "Bourdain 2026"){.id-not
    .include-annotation link-icon="twitter" link-icon-type="svg"
    link-icon-color="#1da1f2"}**

    ::: aux-links-transclude-file
    [`https://x.com/Bourdain/status/153513491200872448`](https://x.com/Bourdain/status/153513491200872448 "Bourdain 2026"){.id-not
    .include-content}
    :::

-   **[`<code>https://x.com/Bourdain/status/153516004994072576</code>`{=html}](https://x.com/Bourdain/status/153516004994072576 "Bourdain 2026"){.id-not
    .include-annotation link-icon="twitter" link-icon-type="svg"
    link-icon-color="#1da1f2"}**

    ::: aux-links-transclude-file
    [`https://x.com/Bourdain/status/153516004994072576`](https://x.com/Bourdain/status/153516004994072576 "Bourdain 2026"){.id-not
    .include-content}
    :::

-   **[`<code>https://x.com/francoisfleuret/status/1714531085512544760</code>`{=html}](https://x.com/francoisfleuret/status/1714531085512544760 "Fleuret 2026"){.link-annotated-partial
    .id-not .include-annotation link-icon="twitter" link-icon-type="svg"
    link-icon-color="#1da1f2"}**

-   **[`<code>https://x.com/stephenroller/status/1579993017234382849</code>`{=html}](https://x.com/stephenroller/status/1579993017234382849 "stephenroller 2026"){.link-annotated-partial
    .id-not .include-annotation link-icon="twitter" link-icon-type="svg"
    link-icon-color="#1da1f2"}**

# Bibliography {#link-bibliography-section}

1.  `https://arxiv.org/abs/2503.24187`:
    [`“NeuRa<span class="logotype-latex">L<span class="logotype-latex-a">a</span>T<span class="logotype-latex-e">e</span>X</span>: A Machine Learning Library Written in Pure <span class="logotype-latex">L<span class="logotype-latex-a">a</span>T<span class="logotype-latex-e">e</span>X</span>”`{=html}](https://arxiv.org/abs/2503.24187 "'NeuRa<span class="logotype-latex">L<span class="logotype-latex-a">a</span>T<span class="logotype-latex-e">e</span>X</span>: A machine learning library written in pure <span class="logotype-latex">L<span class="logotype-latex-a">a</span>T<span class="logotype-latex-e">e</span>X</span>', Gardner et al 2025"){#gardner-et-al-2025
    .link-annotated},
    [James A. D. Gardner, Will Rowan, William A. P. Smith]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2503.24187.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2503.24187"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

2.  `https://arxiv.org/abs/2501.03992`:
    [`“NeuralSVG: An Implicit Representation for Text-To-Vector Generation”`{=html}](https://arxiv.org/abs/2501.03992 "'NeuralSVG: An Implicit Representation for Text-to-Vector Generation', Polaczek et al 2025"){#polaczek-et-al-2025
    .link-annotated},
    [Sagi Polaczek, Yuval Alaluf, Elad Richardson[, Yael Vinker, Daniel Cohen-Or]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2501.03992.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2501.03992"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

3.  `https://www.lesswrong.com/posts/LncYobrn3vRr7qkZW/the-slingshot-helps-with-learning`:
    [`“The Slingshot Helps With Learning”`{=html}](https://www.lesswrong.com/posts/LncYobrn3vRr7qkZW/the-slingshot-helps-with-learning "'The slingshot helps with learning', Wu 2024"){#wu-2024
    .link-annotated}, [Wilson Wu]{.author .cite-author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fwww.lesswrong.com%252Fposts%252FLncYobrn3vRr7qkZW%252Fthe-slingshot-helps-with-learning.html "Directory-tag link-bibliography for link https://www.lesswrong.com/posts/LncYobrn3vRr7qkZW/the-slingshot-helps-with-learning"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

4.  `https://arxiv.org/abs/2406.15786`:
    [`“What Matters in Transformers? Not All Attention Is Needed”`{=html}](https://arxiv.org/abs/2406.15786 "'What Matters in Transformers? Not All Attention is Needed', He et al 2024"){#he-et-al-2024
    .link-annotated},
    [Shwai He, Guoheng Sun, Zheyu Shen, Ang Li]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2406.15786.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2406.15786"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

5.  `https://arxiv.org/abs/2406.13131`:
    [`“When Parts Are Greater Than Sums: Individual LLM Components Can Outperform Full Models”`{=html}](https://arxiv.org/abs/2406.13131 "'When Parts are Greater Than Sums: Individual LLM Components Can Outperform Full Models', Chang et al 2024"){#chang-et-al-2024-1
    .link-annotated},
    [Ting-Yun Chang, Jesse Thomason, Robin Jia]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2406.13131.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2406.13131"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

6.  `https://arxiv.org/abs/2406.11233`:
    [`“Probing the Decision Boundaries of In-Context Learning in Large Language Models”`{=html}](https://arxiv.org/abs/2406.11233 "'Probing the Decision Boundaries of In-context Learning in Large Language Models', Zhao et al 2024"){#zhao-et-al-2024-2
    .link-annotated},
    [Siyan Zhao, Tung Nguyen, [Aditya Grover](https://scholar.google.com/citations?user=oOhnPUgAAAAJ "Aditya Grover"){.id-not}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2406.11233.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2406.11233"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

7.  `https://arxiv.org/abs/2405.20233`:
    [`“Grokfast: Accelerated Grokking by Amplifying Slow Gradients”`{=html}](https://arxiv.org/abs/2405.20233 "'Grokfast: Accelerated Grokking by Amplifying Slow Gradients', Lee et al 2024"){#lee-et-al-2024-2
    .link-annotated},
    [Jaerin Lee, Bong Gyun Kang, Kihoon Kim, Kyoung Mu Lee]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2405.20233.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2405.20233"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

8.  `https://arxiv.org/abs/2310.13061`:
    [`“To Grok or Not to Grok: Disentangling Generalization and Memorization on Corrupted Algorithmic Datasets”`{=html}](https://arxiv.org/abs/2310.13061 "'To grok or not to grok: Disentangling generalization and memorization on corrupted algorithmic datasets', Doshi et al 2023"){#doshi-et-al-2023
    .link-annotated},
    [Darshil Doshi, Aritra Das, Tianyu He, Andrey Gromov]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2310.13061.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2310.13061"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

9.  `https://arxiv.org/abs/2310.08708`:
    [`“Polynomial Time Cryptanalytic Extraction of Neural Network Models”`{=html}](https://arxiv.org/abs/2310.08708 "'Polynomial Time Cryptanalytic Extraction of Neural Network Models', Shamir et al 2023"){#shamir-et-al-2023
    .link-annotated},
    [[Adi Shamir](https://en.wikipedia.org/wiki/Adi_Shamir "Adi Shamir"){.link-annotated-partial
    .id-not}, Isaac Canales-Martinez, Anna Hambitzer[, Jorge Chavez-Saab, Francisco Rodrigez-Henriquez, Nitin Satpute]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2310.08708.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2310.08708"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

10. `https://arxiv.org/abs/2306.13575`:
    [`“Scaling MLPs: A Tale of Inductive Bias”`{=html}](https://arxiv.org/abs/2306.13575 "'Scaling MLPs: A Tale of Inductive Bias', Bachmann et al 2023"){#bachmann-et-al-2023
    .link-annotated},
    [Gregor Bachmann, Sotiris Anagnostidis, [Thomas Hofmann](https://en.wikipedia.org/wiki/Thomas_Hofmann "Thomas Hofmann"){.id-not}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2306.13575.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2306.13575"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

11. `https://arxiv.org/abs/2303.13506`:
    [`“The Quantization Model of Neural Scaling”`{=html}](https://arxiv.org/abs/2303.13506 "'The Quantization Model of Neural Scaling', Michaud et al 2023"){#michaud-et-al-2023
    .link-annotated},
    [Eric J. Michaud, Ziming Liu, Uzay Girit, [Max Tegmark](https://en.wikipedia.org/wiki/Max_Tegmark "Max Tegmark"){.link-annotated-partial
    .id-not}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2303.13506.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2303.13506"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

12. `https://arxiv.org/abs/2303.06053#google`:
    [`“TSMixer: An All-MLP Architecture for Time Series Forecasting”`{=html}](https://arxiv.org/abs/2303.06053#google "'TSMixer: An All-MLP Architecture for Time Series Forecasting', Chen et al 2023"){#chen-et-al-2023-14
    .link-annotated},
    [Si-An Chen, Chun-Liang Li, Nate Yoder[, Sercan O. Arik, Tomas Pfister]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2303.06053%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2303.06053#google"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

13. `2023-bures.pdf`:
    [`“Organic Reaction Mechanism Classification Using Machine Learning”`{=html}](/doc/science/chemistry/2023-bures.pdf "'Organic reaction mechanism classification using machine learning', Burés & Larrosa 2023"){#burés-larrosa-2023
    .link-annotated},
    [[Jordi Burés](https://en.wikipedia.org/wiki/Jordi_Bur%C3%A9s "Jordi Burés"){.id-not}, [Igor Larrosa](https://en.wikipedia.org/wiki/Igor_Larrosa "Igor Larrosa"){.id-not}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/%252Fdoc%252Fscience%252Fchemistry%252F2023-bures.pdf.html "Directory-tag link-bibliography for link /doc/science/chemistry/2023-bures.pdf"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

14. `https://www.nature.com/articles/s41467-022-35422-y`:
    [`“Merging Enzymatic and Synthetic Chemistry With Computational Synthesis Planning”`{=html}](https://www.nature.com/articles/s41467-022-35422-y "'Merging enzymatic and synthetic chemistry with computational synthesis planning', Levin et al 2022"){#levin-et-al-2022-1
    .link-annotated},
    [Itai Levin, Mengjie Liu, Christopher A. Voigt, Connor W. Coley]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fwww.nature.com%252Farticles%252Fs41467-022-35422-y.html "Directory-tag link-bibliography for link https://www.nature.com/articles/s41467-022-35422-y"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

15. `https://arxiv.org/abs/2211.03495`:
    [`“How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers”`{=html}](https://arxiv.org/abs/2211.03495 "'How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers', Hassid et al 2022"){#hassid-et-al-2022
    .link-annotated},
    [Michael Hassid, Hao Peng, Daniel Rotem[, [Jungo Kasai](https://jungokasai.github.io/ "About me—Jungo Kasai"){.link-annotated-partial
    .id-not}, Ivan Montero, [Noah Smith](https://en.wikipedia.org/wiki/Noah_Smith_(writer) "Noah Smith (writer)"){.id-not}, Roy Schwartz]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2211.03495.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2211.03495"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

16. `https://arxiv.org/abs/2210.06313#google`:
    [`“The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers”`{=html}](https://arxiv.org/abs/2210.06313#google "'The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers', Li et al 2022"){#li-et-al-2022-11
    .link-annotated},
    [Zonglin Li, Chong You, Srinadh Bhojanapalli[, Daliang Li, Ankit Singh Rawat, Sashank J. Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, [Sanjiv Kumar](https://scholar.google.com/citations?user=08CNqrYAAAAJ&view_op=list_works&sortby=pubdate "Sanjiv Kumar"){.id-not}]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2210.06313%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2210.06313#google"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

17. `https://arxiv.org/abs/2210.03310#google`:
    [`“Scaling Forward Gradient With Local Losses”`{=html}](https://arxiv.org/abs/2210.03310#google "'Scaling Forward Gradient With Local Losses', Ren et al 2022"){#ren-et-al-2022-1
    .link-annotated},
    [Mengye Ren, [Simon Kornblith](https://www.zotero.org/simon/cv "'Simon Kornblith CV', Kornblith 2026"){.id-not}, Renjie Liao, [Geoffrey Hinton](https://en.wikipedia.org/wiki/Geoffrey_Hinton "Geoffrey Hinton"){.link-annotated-partial
    .id-not}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2210.03310%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2210.03310#google"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

18. `https://arxiv.org/abs/2210.01117`:
    [`“Omnigrok: Grokking Beyond Algorithmic Data”`{=html}](https://arxiv.org/abs/2210.01117 "'Omnigrok: Grokking Beyond Algorithmic Data', Liu et al 2022"){#liu-et-al-2022-10
    .link-annotated},
    [Ziming Liu, Eric J. Michaud, [Max Tegmark](https://en.wikipedia.org/wiki/Max_Tegmark "Max Tegmark"){.link-annotated-partial
    .id-not}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2210.01117.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2210.01117"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

19. `https://arxiv.org/abs/2209.12892`:
    [`“<code>g.pt</code>: Learning to Learn With Generative Models of Neural Network Checkpoints”`{=html}](https://arxiv.org/abs/2209.12892 "'<code>g.pt</code>: Learning to Learn with Generative Models of Neural Network Checkpoints', Peebles et al 2022"){#peebles-et-al-2022
    .link-annotated},
    [William Peebles, Ilija Radosavovic, Tim Brooks[, [Alexei A. Efros](https://en.wikipedia.org/wiki/Alexei_A._Efros "Alexei A. Efros"){.id-not}, [Jitendra Malik](https://en.wikipedia.org/wiki/Jitendra_Malik "Jitendra Malik"){.id-not}]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2209.12892.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2209.12892"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

20. `https://arxiv.org/abs/2207.10551#google`:
    [`“Scaling Laws vs Model Architectures: How Does Inductive Bias Influence Scaling?”`{=html}](https://arxiv.org/abs/2207.10551#google "'Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?', Tay et al 2022"){#tay-et-al-2022-1
    .link-annotated},
    [[Yi Tay](https://www.yitay.net/ "Yi Tay"){.link-annotated-partial
    .id-not}, [Mostafa Dehghani](https://scholar.google.com/citations?user=MiHOX3QAAAAJ "Mostafa Dehghani"){.id-not}, Samira Abnar[, Hyung Won Chung, [William Fedus](https://scholar.google.com/citations?user=-ZfwQOkAAAAJ "William Fedus"){.id-not}, Jinfeng Rao, [Sharan Narang](https://scholar.google.com/citations?user=CWOixywAAAAJ "Sharan Narang"){.link-modified-recently
    .id-not}, Vinh Q. Tran, [Dani Yogatama](https://dyogatama.github.io/ "Dani Yogatama"){.link-annotated-partial
    .id-not}, [Donald Metzler](https://www.don-metzler.net/ "About—Donald Metzler"){.link-annotated-partial
    .id-not}]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2207.10551%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2207.10551#google"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

21. `https://arxiv.org/abs/2206.07137`:
    [`“RHO-LOSS: Prioritized Training on Points That Are Learnable, Worth Learning, and Not Yet Learnt”`{=html}](https://arxiv.org/abs/2206.07137 "'RHO-LOSS: Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt', Mindermann et al 2022"){#mindermann-et-al-2022
    .link-annotated},
    [Sören Mindermann, Jan Brauner, Muhammed Razzak[, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, [Aidan N. Gomez](https://en.wikipedia.org/wiki/Aidan_Gomez "Aidan Gomez"){.id-not}, Adrien Morisot, Sebastian Farquhar, [Yarin Gal](https://www.cs.ox.ac.uk/people/yarin.gal/website/ "Yarin Gal—Home Page"){.id-not}]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2206.07137.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2206.07137"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

22. `https://arxiv.org/abs/2206.05852`:
    [`“ChordMixer: A Scalable Neural Attention Model for Sequences With Different Lengths”`{=html}](https://arxiv.org/abs/2206.05852 "'ChordMixer: A Scalable Neural Attention Model for Sequences with Different Lengths', Khalitov et al 2022"){#khalitov-et-al-2022
    .link-annotated},
    [Ruslan Khalitov, Tong Yu, Lei Cheng, Zhirong Yang]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2206.05852.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2206.05852"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

23. `https://arxiv.org/abs/2205.12399#google`:
    [`“Sparse Mixers: Combining MoE and Mixing to Build a More Efficient BERT”`{=html}](https://arxiv.org/abs/2205.12399#google "'Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT', Lee-Thorp & Ainslie 2022"){#lee-thorp-ainslie-2022
    .link-annotated}, [James Lee-Thorp, Joshua Ainslie]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2205.12399%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2205.12399#google"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

24. `https://arxiv.org/abs/2205.10343`:
    [`“Towards Understanding Grokking: An Effective Theory of Representation Learning”`{=html}](https://arxiv.org/abs/2205.10343 "'Towards Understanding Grokking: An Effective Theory of Representation Learning', Liu et al 2022"){#liu-et-al-2022-18
    .link-annotated},
    [Ziming Liu, Ouail Kitouni, Niklas Nolte[, Eric J. Michaud, [Max Tegmark](https://en.wikipedia.org/wiki/Max_Tegmark "Max Tegmark"){.link-annotated-partial
    .id-not}, Mike Williams]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2205.10343.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2205.10343"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

25. `https://arxiv.org/abs/2204.10670`:
    [`“Paramixer: Parameterizing Mixing Links in Sparse Factors Works Better Than Dot-Product Self-Attention”`{=html}](https://arxiv.org/abs/2204.10670 "'Paramixer: Parameterizing Mixing Links in Sparse Factors Works Better than Dot-Product Self-Attention', Yu et al 2022"){#yu-et-al-2022-4
    .link-annotated},
    [Tong Yu, Ruslan Khalitov, Lei Cheng, Zhirong Yang]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2204.10670.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2204.10670"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

26. `https://arxiv.org/abs/2203.06850`:
    [`“Efficient Language Modeling With Sparse All-MLP”`{=html}](https://arxiv.org/abs/2203.06850 "'Efficient Language Modeling with Sparse All-MLP', Yu et al 2022"){#yu-et-al-2022-5
    .link-annotated},
    [Ping Yu, [Mikel Artetxe](https://en.wikipedia.org/wiki/Mikel_Artetxe "Mikel Artetxe"){.id-not}, Myle Ott[, Sam Shleifer, Hongyu Gong, Ves Stoyanov, Xian Li]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2203.06850.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2203.06850"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

27. `https://arxiv.org/abs/2203.03691`:
    [`“HyperMixer: An MLP-Based Low Cost Alternative to Transformers”`{=html}](https://arxiv.org/abs/2203.03691 "'HyperMixer: An MLP-based Low Cost Alternative to Transformers', Mai et al 2022"){#mai-et-al-2022
    .link-annotated},
    [Florian Mai, Arnaud Pannatier, Fabio Fehr[, Haolin Chen, Francois Marelli, Francois Fleuret, James Henderson]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2203.03691.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2203.03691"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

28. `https://arxiv.org/abs/2202.06510#microsoft`:
    [`“Mixing and Shifting: Exploiting Global and Local Dependencies in Vision MLPs”`{=html}](https://arxiv.org/abs/2202.06510#microsoft "'Mixing and Shifting: Exploiting Global and Local Dependencies in Vision MLPs', Zheng et al 2022"){#zheng-et-al-2022-2
    .link-annotated},
    [Huangjie Zheng, Pengcheng He, [Weizhu Chen](https://scholar.google.com/citations?user=LG_E-4EAAAAJ "Weizhu Chen"){.id-not}, Mingyuan Zhou]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2202.06510%2523microsoft.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2202.06510#microsoft"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

29. `https://arxiv.org/abs/2201.10801`:
    [`“When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanism (ShiftViT)”`{=html}](https://arxiv.org/abs/2201.10801 "'When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanism (ShiftViT)', Wang et al 2022"){#wang-et-al-2022-22
    .link-annotated},
    [Guangting Wang, Yucheng Zhao, Chuanxin Tang[, Chong Luo, [Wenjun Zeng](https://en.wikipedia.org/wiki/Wenjun_Zeng "Wenjun Zeng"){.id-not}]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2201.10801.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2201.10801"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

30. `https://arxiv.org/abs/2201.09792`:
    [`“ConvMixer: Patches Are All You Need?”`{=html}](https://arxiv.org/abs/2201.09792 "'ConvMixer: Patches Are All You Need?', Trockman & Kolter 2022"){#trockman-kolter-2022
    .link-annotated},
    [Asher Trockman, [J. Zico Kolter](https://en.wikipedia.org/wiki/Zico_Kolter "Zico Kolter"){.id-not}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2201.09792.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2201.09792"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

31. `https://arxiv.org/abs/2111.11418`:
    [`“MetaFormer Is Actually What You Need for Vision”`{=html}](https://arxiv.org/abs/2111.11418 "'MetaFormer is Actually What You Need for Vision', Yu et al 2021"){#yu-et-al-2021-1
    .link-annotated},
    [Weihao Yu, Mi Luo, Pan Zhou[, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, Shuicheng Yan]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2111.11418.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2111.11418"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

32. `https://arxiv.org/abs/2110.11526#deepmind`:
    [`“Wide Neural Networks Forget Less Catastrophically”`{=html}](https://arxiv.org/abs/2110.11526#deepmind "'Wide Neural Networks Forget Less Catastrophically', Mirzadeh et al 2021"){#mirzadeh-et-al-2021
    .link-annotated},
    [Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin[, Huiyi Hu, [Razvan Pascanu](https://sites.google.com/view/razp/home){.link-annotated-partial
    .id-not}, Dilan Gorur, Mehrdad Farajtabar]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2110.11526%2523deepmind.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2110.11526#deepmind"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

33. `https://arxiv.org/abs/2110.02095#google`:
    [`“Exploring the Limits of Large Scale Pre-Training”`{=html}](https://arxiv.org/abs/2110.02095#google "'Exploring the Limits of Large Scale Pre-training', Abnar et al 2021"){#abnar-et-al-2021
    .link-annotated},
    [Samira Abnar, [Mostafa Dehghani](https://scholar.google.com/citations?user=MiHOX3QAAAAJ "Mostafa Dehghani"){.id-not}, [Behnam Neyshabur](https://www.neyshabur.net/ "Behnam Neyshabur"){.id-not}, Hanie Sedghi]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2110.02095%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2110.02095#google"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

34. `https://arxiv.org/abs/2109.05422`:
    [`“Sparse MLP for Image Recognition: Is Self-Attention Really Necessary?”`{=html}](https://arxiv.org/abs/2109.05422 "'Sparse MLP for Image Recognition: Is Self-Attention Really Necessary?', Tang et al 2021"){#tang-et-al-2021-2
    .link-annotated},
    [Chuanxin Tang, Yucheng Zhao, Guangting Wang[, Chong Luo, Wenxuan Xie, [Wenjun Zeng](https://en.wikipedia.org/wiki/Wenjun_Zeng "Wenjun Zeng"){.id-not}]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2109.05422.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2109.05422"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

35. `https://arxiv.org/abs/2109.04454`:
    [`“ConvMLP: Hierarchical Convolutional MLPs for Vision”`{=html}](https://arxiv.org/abs/2109.04454 "'ConvMLP: Hierarchical Convolutional MLPs for Vision', Li et al 2021"){#li-et-al-2021-5
    .link-annotated},
    [Jiachen Li, [Ali Hassani](https://en.wikipedia.org/wiki/Ali_Hassani "Ali Hassani"){.id-not}, Steven Walton, Humphrey Shi]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2109.04454.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2109.04454"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

36. `https://arxiv.org/abs/2108.13002#microsoft`:
    [`“A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP”`{=html}](https://arxiv.org/abs/2108.13002#microsoft "'A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP', Zhao et al 2021"){#zhao-et-al-2021-4
    .link-annotated},
    [Yucheng Zhao, Guangting Wang, Chuanxin Tang[, Chong Luo, [Wenjun Zeng](https://en.wikipedia.org/wiki/Wenjun_Zeng "Wenjun Zeng"){.id-not}, Zheng-Jun Zha]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2108.13002%2523microsoft.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2108.13002#microsoft"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

37. `https://arxiv.org/abs/2108.13341#huawei`:
    [`“Hire-MLP: Vision MLP via Hierarchical Rearrangement”`{=html}](https://arxiv.org/abs/2108.13341#huawei "'Hire-MLP: Vision MLP via Hierarchical Rearrangement', Guo et al 2021"){#guo-et-al-2021-hiremlp
    .link-annotated},
    [Jianyuan Guo, Yehui Tang, Kai Han[, Xinghao Chen, Han Wu, [Chao Xu](https://en.wikipedia.org/wiki/Chao_Xu "Chao Xu"){.id-not}, [Chang Xu](https://en.wikipedia.org/wiki/Chang_Xu "Chang Xu"){.id-not}, Yunhe Wang]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2108.13341%2523huawei.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2108.13341#huawei"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

38. `https://arxiv.org/abs/2108.04384`:
    [`“RaftMLP: How Much Can Be Done Without Attention and With Less Spatial Locality?”`{=html}](https://arxiv.org/abs/2108.04384 "'RaftMLP: How Much Can Be Done Without Attention and with Less Spatial Locality?', Tatsunami & Taki 2021"){#tatsunami-taki-2021
    .link-annotated}, [Yuki Tatsunami, Masato Taki]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2108.04384.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2108.04384"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

39. `https://arxiv.org/abs/2108.01072#baidu`:
    [`“S<sup>2</sup>-MLPv2: Improved Spatial-Shift MLP Architecture for Vision”`{=html}](https://arxiv.org/abs/2108.01072#baidu "'S<sup>2</sup>-MLPv2: Improved Spatial-Shift MLP Architecture for Vision', Yu et al 2021"){#yu-et-al-2021-s2mlpv2
    .link-annotated},
    [[Tan Yu](https://en.wikipedia.org/wiki/Tan_Yu "Tan Yu"){.id-not}, [Xu Li](https://en.wikipedia.org/wiki/Xu_Li "Xu Li"){.id-not}, Yunfeng Cai[, Mingming Sun, Ping Li]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2108.01072%2523baidu.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2108.01072#baidu"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

40. `https://arxiv.org/abs/2107.10224`:
    [`“CycleMLP: A MLP-Like Architecture for Dense Prediction”`{=html}](https://arxiv.org/abs/2107.10224 "'CycleMLP: A MLP-like Architecture for Dense Prediction', Chen et al 2021"){#chen-et-al-2021-cyclemlp
    .link-annotated},
    [Shoufa Chen, Enze Xie, Chongjian Ge[, Runjian Chen, Ding Liang, Ping Luo]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2107.10224.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2107.10224"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

41. `https://arxiv.org/abs/2107.08391`:
    [`“AS-MLP: An Axial Shifted MLP Architecture for Vision”`{=html}](https://arxiv.org/abs/2107.08391 "'AS-MLP: An Axial Shifted MLP Architecture for Vision', Lian et al 2021"){#lian-et-al-2021-2
    .link-annotated},
    [Dongze Lian, Zehao Yu, [Xing Sun](https://en.wikipedia.org/wiki/Xing_Sun "Xing Sun"){.id-not}, Shenghua Gao]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2107.08391.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2107.08391"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

42. `https://arxiv.org/abs/2106.12368`:
    [`“Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition”`{=html}](https://arxiv.org/abs/2106.12368 "'Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition', Hou et al 2021"){#hou-et-al-2021
    .link-annotated},
    [Qibin Hou, Zihang Jiang, [Li Yuan](https://en.wikipedia.org/wiki/Li_Yuan "Li Yuan"){.id-not}[, Ming-Ming Cheng, Shuicheng Yan, Jiashi Feng]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2106.12368.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2106.12368"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

43. `https://arxiv.org/abs/2106.12372#nvidia`:
    [`“Real-Time Neural Radiance Caching for Path Tracing”`{=html}](https://arxiv.org/abs/2106.12372#nvidia "'Real-time Neural Radiance Caching for Path Tracing', Müller et al 2021"){#müller-et-al-2021-4
    .link-annotated},
    [Thomas Müller, Fabrice Rousselle, Jan Novák, Alexander Keller]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2106.12372%2523nvidia.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2106.12372#nvidia"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

44. `https://arxiv.org/abs/2106.07477#baidu`:
    [`“S<sup>2</sup>-MLP: Spatial-Shift MLP Architecture for Vision”`{=html}](https://arxiv.org/abs/2106.07477#baidu "'S<sup>2</sup>-MLP: Spatial-Shift MLP Architecture for Vision', Yu et al 2021"){#yu-et-al-2021-s2mlp
    .link-annotated},
    [[Tan Yu](https://en.wikipedia.org/wiki/Tan_Yu "Tan Yu"){.id-not}, [Xu Li](https://en.wikipedia.org/wiki/Xu_Li "Xu Li"){.id-not}, Yunfeng Cai[, Mingming Sun, Ping Li]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2106.07477%2523baidu.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2106.07477#baidu"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

45. `https://arxiv.org/abs/2106.01548`:
    [`“When Vision Transformers Outperform ResNets without Pre-Training or Strong Data Augmentations”`{=html}](https://arxiv.org/abs/2106.01548 "'When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations', Chen et al 2021"){#chen-et-al-2021-10
    .link-annotated},
    [Xiangning Chen, Cho-Jui Hsieh, Boqing Gong]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2106.01548.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2106.01548"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

46. `https://arxiv.org/abs/2106.01401`:
    [`“Container: Context Aggregation Network”`{=html}](https://arxiv.org/abs/2106.01401 "'Container: Context Aggregation Network', Gao et al 2021"){#gao-et-al-2021-3
    .link-annotated},
    [Peng Gao, Jiasen Lu, [Hongsheng Li](https://www.ee.cuhk.edu.hk/~hsli/){.id-not}[, Roozbeh Mottaghi, Aniruddha Kembhavi]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2106.01401.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2106.01401"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

47. `https://arxiv.org/abs/2105.08050#google`:
    [`“Pay Attention to MLPs”`{=html}](https://arxiv.org/abs/2105.08050#google "'Pay Attention to MLPs', Liu et al 2021"){#liu-et-al-2021-gmlp
    .link-annotated},
    [Hanxiao Liu, [Zihang Dai](https://scholar.google.com/citations?user=uZqsVXkAAAAJ "Zihang Dai"){.id-not}, David R. So, [Quoc V. Le](https://en.wikipedia.org/wiki/Quoc_V._Le "Quoc V. Le"){.id-not}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2105.08050%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2105.08050#google"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

48. `https://arxiv.org/abs/2105.03824#google`:
    [`“FNet: Mixing Tokens With Fourier Transforms”`{=html}](https://arxiv.org/abs/2105.03824#google "'FNet: Mixing Tokens with Fourier Transforms', Lee-Thorp et al 2021"){#lee-thorp-et-al-2021
    .link-annotated},
    [James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontanon]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2105.03824%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2105.03824#google"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

49. `https://arxiv.org/abs/2105.02723`:
    [`“Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet”`{=html}](https://arxiv.org/abs/2105.02723 "'Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet', Melas-Kyriazi 2021"){#melas-kyriazi-2021
    .link-annotated}, [Luke Melas-Kyriazi]{.author .cite-author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2105.02723.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2105.02723"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

50. `https://arxiv.org/abs/2105.01883`:
    [`“RepMLP: Re-Parameterizing Convolutions into Fully-Connected Layers for Image Recognition”`{=html}](https://arxiv.org/abs/2105.01883 "'RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognition', Ding et al 2021"){#ding-et-al-2021-3
    .link-annotated},
    [Xiaohan Ding, Chunlong Xia, Xiangyu Zhang[, Xiaojie Chu, Jungong Han, Guiguang Ding]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2105.01883.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2105.01883"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

51. `https://arxiv.org/abs/2105.01601#google`:
    [`“MLP-Mixer: An All-MLP Architecture for Vision”`{=html}](https://arxiv.org/abs/2105.01601#google "'MLP-Mixer: An all-MLP Architecture for Vision', Tolstikhin et al 2021"){#tolstikhin-et-al-2021
    .link-annotated},
    [Ilya Tolstikhin, [Neil Houlsby](https://neilhoulsby.github.io/ "Neil Houlsby"){.link-annotated-partial
    .id-not}, Alexander Kolesnikov[, [Lucas Beyer](https://scholar.google.com/citations?user=p2gwhK4AAAAJ "Lucas Beyer"){.id-not}, [Xiaohua Zhai](https://sites.google.com/view/xzhai "Xiaohua Zhai (翟晓华)"){.link-annotated-partial
    .id-not}, Thomas Unterthiner, Jessica Yung, Daniel Keysers, [Jakob Uszkoreit](https://scholar.google.com/citations?user=mOG0bwsAAAAJ "Jakob Uszkoreit"){.id-not}, Mario Lucic, Alexey Dosovitskiy]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2105.01601%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2105.01601#google"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

52. `2021-power.pdf#openai`:
    [`“Grokking: Generalization Beyond Overfitting On Small Algorithmic Datasets”`{=html}](/doc/ai/nn/fully-connected/2021-power.pdf#openai "'Grokking: Generalization Beyond Overfitting On Small Algorithmic Datasets', Power et al 2021"){#power-et-al-2021
    .link-annotated},
    [Alethea Power, Yuri Burda, Harri Edwards[, Igor Babuschkin, [Vedant Misra](http://vedantmisra.com/ "Vedant Misra"){.id-not}]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/%252Fdoc%252Fai%252Fnn%252Ffully-connected%252F2021-power.pdf%2523openai.html "Directory-tag link-bibliography for link /doc/ai/nn/fully-connected/2021-power.pdf#openai"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

53. `abstract`:
    [`“Fully-Connected Neural Nets”`{=html}](/doc/ai/nn/fully-connected/abstract "'Fully-Connected Neural Nets', Gwern 2021"){#gwern-doc-ai-nn-fully-connected-abstract-2
    .link-annotated-partial},
    [[Gwern](/index#abstract "'Gwern Bibliography', Gwern 2010"){.link-annotated-partial
    .id-not}]{.author .cite-author}

    [[link-bibliography](/metadata/annotation/link-bibliography/%252Fdoc%252Fai%252Fnn%252Ffully-connected%252Fabstract.html "Directory-tag link-bibliography for link /doc/ai/nn/fully-connected/abstract"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

54. `https://arxiv.org/abs/2103.14030`:
    [`“Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows”`{=html}](https://arxiv.org/abs/2103.14030 "'Swin Transformer: Hierarchical Vision Transformer using Shifted Windows', Liu et al 2021"){#liu-et-al-2021-swintranformer
    .link-annotated},
    [Ze Liu, Yutong Lin, Yue Cao[, Han Hu, Yixuan Wei, [Zheng Zhang](https://web.ece.ucsb.edu/~zhengzhang/){.id-not}, Stephen Lin, Baining Guo]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2103.14030.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2103.14030"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

55. `https://arxiv.org/abs/2011.13775`:
    [`“Image Generators With Conditionally-Independent Pixel Synthesis”`{=html}](https://arxiv.org/abs/2011.13775 "'Image Generators with Conditionally-Independent Pixel Synthesis', Anokhin et al 2020"){#anokhin-et-al-2020
    .link-annotated},
    [Ivan Anokhin, Kirill Demochkin, Taras Khakhulin[, Gleb Sterkin, Victor Lempitsky, Denis Korzhenkov]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2011.13775.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2011.13775"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

56. `https://arxiv.org/abs/2005.00743#google`:
    [`“Synthesizer: Rethinking Self-Attention in Transformer Models”`{=html}](https://arxiv.org/abs/2005.00743#google "'Synthesizer: Rethinking Self-Attention in Transformer Models', Tay et al 2020"){#tay-et-al-2020-2
    .link-annotated},
    [[Yi Tay](https://www.yitay.net/ "Yi Tay"){.link-annotated-partial
    .id-not}, Dara Bahri, [Donald Metzler](https://www.don-metzler.net/ "About—Donald Metzler"){.link-annotated-partial
    .id-not}[, Da-Cheng Juan, Zhe Zhao, Che Zheng]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2005.00743%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2005.00743#google"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

57. `https://arxiv.org/abs/2003.01629`:
    [`“Can Increasing Input Dimensionality Improve Deep Reinforcement Learning?”`{=html}](https://arxiv.org/abs/2003.01629 "'Can Increasing Input Dimensionality Improve Deep Reinforcement Learning?', Ota et al 2020"){#ota-et-al-2020
    .link-annotated},
    [Kei Ota, Tomoaki Oiki, Devesh K. Jha[, Toshisada Mariyama, Daniel Nikovski]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2003.01629.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2003.01629"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

58. `https://arxiv.org/abs/1911.13299`:
    [`“What’s Hidden in a Randomly Weighted Neural Network?”`{=html}](https://arxiv.org/abs/1911.13299 "'What’s Hidden in a Randomly Weighted Neural Network?', Ramanujan et al 2019"){#ramanujan-et-al-2019
    .link-annotated},
    [Vivek Ramanujan, [Mitchell Wortsman](https://mitchellnw.github.io/ "Mitchell Wortsman"){.id-not}, Aniruddha Kembhavi[, [Ali Farhadi](https://en.wikipedia.org/wiki/Ali_Farhadi "Ali Farhadi"){.id-not}, Mohammad Rastegari]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F1911.13299.html "Directory-tag link-bibliography for link https://arxiv.org/abs/1911.13299"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

59. `https://arxiv.org/abs/1804.00222#google`:
    [`“Meta-Learning Update Rules for Unsupervised Representation Learning”`{=html}](https://arxiv.org/abs/1804.00222#google "'Meta-Learning Update Rules for Unsupervised Representation Learning', Metz et al 2018"){#metz-et-al-2018-2
    .link-annotated},
    [[Luke Metz](https://scholar.google.com/citations?user=k_u5ULgAAAAJ "Luke Metz"){.id-not}, Niru Maheswaranathan, Brian Cheung, [Jascha Sohl-Dickstein](https://scholar.google.com/citations?user=-3zYIjQAAAAJ "Jascha Sohl-Dickstein"){.id-not}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F1804.00222%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/1804.00222#google"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

60. `2017-sabatelli.pdf#page=3`:
    [`“Learning to Play Chess With Minimal Lookahead and Deep Value Neural Networks”`{=html}](/doc/reinforcement-learning/chess/2017-sabatelli.pdf#page=3 "'Learning to Play Chess with Minimal Lookahead and Deep Value Neural Networks', Sabatelli 2017 (page 3)"){#sabatelli-2017-page-3
    .link-annotated}, [Matthia Sabatelli]{.author .cite-author}

    [[link-bibliography](/metadata/annotation/link-bibliography/%252Fdoc%252Freinforcement-learning%252Fchess%252F2017-sabatelli.pdf%2523page%253D3.html "Directory-tag link-bibliography for link /doc/reinforcement-learning/chess/2017-sabatelli.pdf#page=3"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

61. `https://arxiv.org/abs/1402.1869`:
    [`“On the Number of Linear Regions of Deep Neural Networks”`{=html}](https://arxiv.org/abs/1402.1869 "'On the Number of Linear Regions of Deep Neural Networks', Montúfar et al 2014"){#montúfar-et-al-2014
    .link-annotated},
    [Guido Montúfar, [Razvan Pascanu](https://sites.google.com/view/razp/home){.link-annotated-partial
    .id-not}, [Kyunghyun Cho](https://www.kyunghyuncho.me/ "Kyunghyun Cho"){.link-annotated-partial
    .id-not}, [Yoshua Bengio](https://en.wikipedia.org/wiki/Yoshua_Bengio "Yoshua Bengio"){.id-not}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F1402.1869.html "Directory-tag link-bibliography for link https://arxiv.org/abs/1402.1869"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

62. `2011-collobert.pdf`:
    [`“Natural Language Processing (Almost) from Scratch”`{=html}](/doc/psychology/linguistics/2011-collobert.pdf "'Natural Language Processing (Almost) from Scratch', Collobert et al 2011"){#collobert-et-al-2011
    .link-annotated},
    [Ronan Collobert, [Jason Weston](https://en.wikipedia.org/wiki/Jason_Weston "Jason Weston"){.id-not}, [Leon Bottou](https://en.wikipedia.org/wiki/Leon_Bottou "Leon Bottou"){.id-not}[, Michael Karlen, [Koray Kavukcuoglu](https://scholar.google.com/citations?user=sGFyDIUAAAAJ "Koray Kavukcuoglu"){.id-not}, Pavel Kuksa]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/%252Fdoc%252Fpsychology%252Flinguistics%252F2011-collobert.pdf.html "Directory-tag link-bibliography for link /doc/psychology/linguistics/2011-collobert.pdf"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}