---
title: '‘self-attention’ directory'
description: "Bibliography for directory <code>ai/nn/transformer/attention</code>, most recent first: 11 <a class='icon-not' href='/doc/ai/nn/transformer/attention/index#see-alsos'>related tags</a>, 247 <a class='icon-not' href='/doc/ai/nn/transformer/attention/index#links'>annotations</a>, & 47 <a class='icon-not' href='/doc/ai/nn/transformer/attention/index#miscellaneous'>links</a> (<a href='/doc/ai/nn/transformer/index' class='link-page link-tag directory-indexes-upwards link-annotated' data-link-icon='arrow-up-left' data-link-icon-type='svg' rel='tag' title='Link to parent directory'>parent</a>)."
thumbnail: /doc/ai/nn/transformer/attention/2023-trockman-figure2-attentionmappatternsbyinitializationandleveloftrainingshowpriors.png
thumbnail-text: ''
thumbnail-css: "outline"
created: 2019-12-17
modified: 2026-09-17
status: in progress
previous: /doc/ai/nn/transformer/alphafold/index
next: /doc/ai/nn/transformer/attention/compression/index
confidence: log
importance: 0
css-extension: dropcaps-not
index: True
backlink: False
...

::: {#manual-annotation .abstract-small .abstract-tag-directory}
[\[page summary\]](/doc/ai/nn/transformer/attention/abstract "Transclude link for doc/ai/nn/transformer/attention/ tag-abstract page."){#gwern-doc-ai-nn-transformer-attention-abstract
.link-annotated .include-content-core .include-strict .link-page}
:::

# See Also

::: {#see-alsos .directory-indexes .columns}
-   [Parent ('*`Transformer`{=html}*' tag)](/doc/ai/nn/transformer/index "Link to parent directory 'doc/ai/nn/transformer/' (ascending)"){#_ojWSXRFs
    .link-annotated .link-tag .directory-indexes-upwards rel="tag"}

-   [*`compressed Transformer`{=html}*](/doc/ai/nn/transformer/attention/compression/index "‘compressed Transformer’ directory"){#_zhl07LRT
    .link-annotated .link-tag .directory-indexes-downwards rel="tag"}

-   [*`multi-scale Transformer`{=html}*](/doc/ai/nn/transformer/attention/hierarchical/index "‘multi-scale Transformer’ directory"){#_tNdqy5Q0
    .link-annotated .link-tag .directory-indexes-downwards rel="tag"}

-   [*`Transformer matrix optimizations`{=html}*](/doc/ai/nn/transformer/attention/linear-algebra/index "‘Transformer matrix optimizations’ directory"){#_7tpMQuLg
    .link-annotated .link-tag .directory-indexes-downwards rel="tag"}

-   [*`attention ≈ SGD`{=html}*](/doc/ai/nn/transformer/attention/meta-descent/index "‘attention ≈ SGD’ directory"){#_SYtzjAxb
    .link-annotated .link-tag .directory-indexes-downwards rel="tag"}

-   [*`recurrent Transformer`{=html}*](/doc/ai/nn/transformer/attention/recurrent/index "‘recurrent Transformer’ directory"){#_I7PwEKIA
    .link-annotated .link-tag .directory-indexes-downwards rel="tag"}

-   [*`sparse Transformer`{=html}*](/doc/ai/nn/transformer/attention/sparsity/index "‘sparse Transformer’ directory"){#_C9QXsKvk
    .link-annotated .link-tag .directory-indexes-downwards rel="tag"}

-   [*`retrieval AI`{=html}*](/doc/ai/nn/retrieval/index "‘retrieval AI’ directory"){#_d5MkigZ5
    .link-annotated .link-tag .directory-indexes-sideways rel="tag"}

-   [*`RNN`{=html}*](/doc/ai/nn/rnn/index "‘RNN’ directory"){#_BHyr1fFu
    .link-annotated .link-tag .directory-indexes-sideways rel="tag"}

-   [*`LM tokenization`{=html}*](/doc/ai/nn/tokenization/index "‘LM tokenization’ directory"){#_OVxPkujb
    .link-annotated .link-tag .directory-indexes-sideways rel="tag"}

-   [*`Claude AI`{=html}*](/doc/ai/nn/transformer/gpt/claude/index "‘Claude AI’ directory"){#_tPRtihL6
    .link-annotated .link-tag .directory-indexes-sideways rel="tag"}

-   [*`video generation`{=html}*](/doc/ai/video/generation/index "‘video generation’ directory"){#_FVaGQXtH
    .link-annotated .link-tag .directory-indexes-sideways rel="tag"}
:::

# Gwern {.display-pop-not}

## `“Absolute Unit NNs: Regression-Based MLPs for Everything”, Gwern 2023`{=html} {#gwern-aunn-section}

**[`Absolute Unit NNs: Regression-Based MLPs for Everything`{=html}](/aunn "'Absolute Unit NNs: Regression-Based MLPs for Everything', Gwern 2023"){.link-annotated
.id-not .include-annotation}**

## `“Research Ideas”, Gwern 2017`{=html} {#gwern-idea-section}

**[`Research Ideas`{=html}](/idea "'Research Ideas', Gwern 2017"){.link-annotated
.id-not .include-annotation}**

## `“GPT-3 Creative Fiction”, Gwern 2020`{=html} {#gwern-gpt-3-section}

**[`GPT-3 Creative Fiction`{=html}](/gpt-3 "'GPT-3 Creative Fiction', Gwern 2020"){.link-annotated
.id-not .include-annotation}**

# Links {.display-pop-not}

## `“FlashAccel: Leveraging High-Bandwidth Flash (HBF) for High-Throughput LLM Inference”, Wang et al 2026`{=html} {#wang-et-al-2026-section}

**[`FlashAccel: Leveraging High-Bandwidth Flash (HBF) for High-Throughput LLM Inference`{=html}](https://arxiv.org/abs/2607.10186 "'FlashAccel: Leveraging High-Bandwidth Flash (HBF) for High-Throughput LLM Inference', Wang et al 2026"){.link-modified-recently
.link-annotated .id-not .include-annotation link-icon="𝛘"
link-icon-type="text" link-icon-color="#b31b1b"}**

## `“MLPs Are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers”, Garcia et al 2026`{=html} {#garcia-et-al-2026-section}

**[`MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers`{=html}](https://arxiv.org/abs/2607.10034 "'MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers', Garcia et al 2026"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Pretraining Recurrent Networks without Recurrence”, Kumar & Isola 2026`{=html} {#kumar-isola-2026-section}

**[`Pretraining Recurrent Networks without Recurrence`{=html}](https://arxiv.org/abs/2606.06479 "'Pretraining Recurrent Networks without Recurrence', Kumar & Isola 2026"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Do Transformers Need 3 Projections? Systematic Study of QKV Variants”, Kayyam et al 2026`{=html} {#kayyam-et-al-2026-section}

**[`Do Transformers Need 3 Projections? Systematic Study of QKV Variants`{=html}](https://arxiv.org/abs/2606.04032 "'Do Transformers Need 3 Projections? Systematic Study of QKV Variants', Kayyam et al 2026"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“A Large-Scale Study on the Accuracy vs Cost Trade-Offs of Training and Evaluation Settings in Fine-Grained Image Recognition”, Rios et al 2026`{=html} {#rios-et-al-2026-section}

**[`A Large-Scale Study on the Accuracy vs Cost Trade-offs of Training and Evaluation Settings in Fine-Grained Image Recognition`{=html}](https://arxiv.org/abs/2605.18700 "'A Large-Scale Study on the Accuracy vs Cost Trade-offs of Training and Evaluation Settings in Fine-Grained Image Recognition', Rios et al 2026"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably”, Du et al 2026`{=html} {#du-et-al-2026-section}

**[`RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably`{=html}](https://arxiv.org/abs/2605.15514 "'RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably', Du et al 2026"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Negation Neglect: When Models Fail to Learn Negations in Training”, Mayne et al 2026`{=html} {#mayne-et-al-2026-section}

**[`Negation Neglect: When models fail to learn negations in training`{=html}](https://arxiv.org/abs/2605.13829 "'Negation Neglect: When models fail to learn negations in training', Mayne et al 2026"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Convergent Evolution: How Different Language Models Learn Similar Number Representations”, Fu et al 2026`{=html} {#fu-et-al-2026-section}

**[`Convergent Evolution: How Different Language Models Learn Similar Number Representations`{=html}](https://arxiv.org/abs/2604.20817 "'Convergent Evolution: How Different Language Models Learn Similar Number Representations', Fu et al 2026"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Slot Machines: How LLMs Keep Track of Multiple Entities”, Bogdan & Lindsey 2026`{=html} {#bogdan-lindsey-2026-section}

**[`Slot Machines: How LLMs Keep Track of Multiple Entities`{=html}](https://arxiv.org/abs/2604.21139 "'Slot Machines: How LLMs Keep Track of Multiple Entities', Bogdan & Lindsey 2026"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Current Activation Oracles Are Hard to Use”`{=html} {#_6Go1eVg3-section}

**[`Current activation oracles are hard to use`{=html}](https://www.lesswrong.com/posts/LXQBcztrWKhtcgQfJ/current-activation-oracles-are-hard-to-use "Current activation oracles are hard to use"){.link-annotated-partial
.id-not .include-annotation link-icon="LW" link-icon-type="text"
link-icon-color="#7faf83"}**

## `“Doc-To-LoRA (D2L): Learning to Instantly Internalize Contexts”, Charakorn et al 2026`{=html} {#charakorn-et-al-2026-section}

**[`Doc-to-LoRA (D2L): Learning to Instantly Internalize Contexts`{=html}](https://arxiv.org/abs/2602.15902#sakana "'Doc-to-LoRA (D2L): Learning to Instantly Internalize Contexts', Charakorn et al 2026"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Debunking Claims about Sub-Quadratic Attention”, Ivanov 2026`{=html} {#ivanov-2026-section}

**[`Debunking claims about sub-quadratic attention`{=html}](https://www.lesswrong.com/posts/kpSXeMcthtHgnwMx3/debunking-claims-about-subquadratic-attention "'Debunking claims about sub-quadratic attention', Ivanov 2026"){.link-annotated-partial
.id-not .include-annotation link-icon="LW" link-icon-type="text"
link-icon-color="#7faf83"}**

## `“DeepSeek-V3.2 Is Okay &amp; Cheap But Slow”, Mowshowitz 2025`{=html} {#mowshowitz-2025-section}

**[`DeepSeek-V3.2 Is Okay &amp; Cheap But Slow`{=html}](https://thezvi.wordpress.com/2025/12/05/deepseek-v3-2-is-okay-and-cheap-but-slow/ "'DeepSeek-V3.2 Is Okay &amp; Cheap But Slow', Mowshowitz 2025"){.link-annotated-partial
.id-not .include-annotation link-icon="deepseek" link-icon-type="svg"
link-icon-color="#4d6bfe"}**

## `“Kimi Linear: An Expressive, Efficient Attention Architecture”, Team et al 2025`{=html} {#team-et-al-2025-section}

**[`Kimi Linear: An Expressive, Efficient Attention Architecture`{=html}](https://arxiv.org/abs/2510.26692#kimi "'Kimi Linear: An Expressive, Efficient Attention Architecture', Team et al 2025"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Transformers Are Inherently Succinct”, Bergsträßer et al 2025`{=html} {#bergsträßer-et-al-2025-section}

**[`Transformers are Inherently Succinct`{=html}](https://arxiv.org/abs/2510.19315 "'Transformers are Inherently Succinct', Bergsträßer et al 2025"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“When Models Manipulate Manifolds: The Geometry of a Counting Task”, Gurnee et al 2025`{=html} {#gurnee-et-al-2025-section}

**[`When Models Manipulate Manifolds: The Geometry of a Counting Task`{=html}](https://transformer-circuits.pub/2025/linebreaks/index.html "'When Models Manipulate Manifolds: The Geometry of a Counting Task', Gurnee et al 2025"){.link-annotated-partial
.id-not .include-annotation link-icon="anthropic" link-icon-type="svg"
link-icon-color="#d4a27f"}**

## `“Evaluating Long Context (Reasoning) Ability”`{=html} {#_BFhzE6mK-section}

**[`Evaluating Long Context (Reasoning) Ability`{=html}](https://nrehiew.github.io/blog/long_context/ "Evaluating Long Context (Reasoning) Ability"){.link-annotated-partial
.id-not .include-annotation}**

## `“Language Models [Activations] Are Injective and Hence Invertible”, Nikolaou et al 2025`{=html} {#nikolaou-et-al-2025-section}

**[`Language Models [Activations] are Injective and Hence Invertible`{=html}](https://arxiv.org/abs/2510.15511 "'Language Models [Activations] are Injective and Hence Invertible', Nikolaou et al 2025"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Tensor Logic: The Language of AI”, Domingos 2025`{=html} {#domingos-2025-section}

**[`Tensor Logic: The Language of AI`{=html}](https://arxiv.org/abs/2510.12269 "'Tensor Logic: The Language of AI', Domingos 2025"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Tensor Logic Homepage”, Domingos 2025`{=html} {#domingos-2025-section}

**[`Tensor Logic homepage`{=html}](https://tensor-logic.org/ "'Tensor Logic homepage', Domingos 2025"){.link-annotated-partial
.id-not .include-annotation}**

## `“Why Can’t Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls”, Bai et al 2025`{=html} {#bai-et-al-2025-section}

**[`Why Can’t Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls`{=html}](https://arxiv.org/abs/2510.00184 "'Why Can’t Transformers Learn Multiplication? Reverse-Engineering Reveals Long-Range Dependency Pitfalls', Bai et al 2025"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Reasoning-Finetuning Repurposes Latent Representations in Base Models”, Ward et al 2025`{=html} {#ward-et-al-2025-section}

**[`Reasoning-Finetuning Repurposes Latent Representations in Base Models`{=html}](https://arxiv.org/abs/2507.12638 "'Reasoning-Finetuning Repurposes Latent Representations in Base Models', Ward et al 2025"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Mixture-Of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation”, Bae et al 2025`{=html} {#bae-et-al-2025-section}

**[`Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation`{=html}](https://arxiv.org/abs/2507.10524 "'Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation', Bae et al 2025"){.link-modified-recently
.link-annotated .id-not .include-annotation link-icon="𝛘"
link-icon-type="text" link-icon-color="#b31b1b"}**

## `“Fast and Simplex: 2-Simplicial Attention in Triton”, Roy et al 2025`{=html} {#roy-et-al-2025-section}

**[`Fast and Simplex: 2-Simplicial Attention in Triton`{=html}](https://arxiv.org/abs/2507.02754#facebook "'Fast and Simplex: 2-Simplicial Attention in Triton', Roy et al 2025"){.link-annotated
.id-not .include-annotation link-icon="facebook" link-icon-type="svg"
link-icon-color="#1877f2"}**

## `“Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning”, Shinnick et al 2025`{=html} {#shinnick-et-al-2025-section}

**[`Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning`{=html}](https://arxiv.org/abs/2505.22308 "'Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning', Shinnick et al 2025"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations”, Ji-An et al 2025`{=html} {#ji-an-et-al-2025-section}

**[`Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations`{=html}](https://arxiv.org/abs/2505.13763 "'Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations', Ji-An et al 2025"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“LLMs Get Lost In Multi-Turn Conversation”, Laban et al 2025`{=html} {#laban-et-al-2025-section}

**[`LLMs Get Lost In Multi-Turn Conversation`{=html}](https://arxiv.org/abs/2505.06120 "'LLMs Get Lost In Multi-Turn Conversation', Laban et al 2025"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“The Geometry of Self-Verification in a Task-Specific Reasoning Model”, Lee et al 2025`{=html} {#lee-et-al-2025-section}

**[`The Geometry of Self-Verification in a Task-Specific Reasoning Model`{=html}](https://arxiv.org/abs/2504.14379 "'The Geometry of Self-Verification in a Task-Specific Reasoning Model', Lee et al 2025"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“It’s All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization”, Behrouz et al 2025`{=html} {#behrouz-et-al-2025-section}

**[`It’s All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization`{=html}](https://arxiv.org/abs/2504.13173#google "'It’s All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization', Behrouz et al 2025"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Content-Addressable Memory With a Content-Free Energy Function”, Benoist et al 2025`{=html} {#benoist-et-al-2025-section}

**[`Content-Addressable Memory with a Content-Free Energy Function`{=html}](https://arxiv.org/abs/2503.18822 "'Content-Addressable Memory with a Content-Free Energy Function', Benoist et al 2025"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Dynamic Tanh: Transformers without Normalization”, Zhu et al 2025`{=html} {#zhu-et-al-2025-section}

**[`Dynamic Tanh: Transformers without Normalization`{=html}](https://arxiv.org/abs/2503.10622#facebook "'Dynamic Tanh: Transformers without Normalization', Zhu et al 2025"){.link-annotated
.id-not .include-annotation link-icon="facebook" link-icon-type="svg"
link-icon-color="#1877f2"}**

## `“(How) Do Language Models Track State?”, Li et al 2025`{=html} {#li-et-al-2025-section}

**[`(How) Do Language Models Track State?`{=html}](https://arxiv.org/abs/2503.02854 "'(How) Do Language Models Track State?', Li et al 2025"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Thinking Slow, Fast: Scaling Inference Compute With Distilled Reasoners”, Paliotta et al 2025`{=html} {#paliotta-et-al-2025-section}

**[`Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasoners`{=html}](https://arxiv.org/abs/2502.20339 "'Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasoners', Paliotta et al 2025"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“<em>μ</em>nit Scaling: Simple and Scalable FP8 LLM Training”, Narayan et al 2025`{=html} {#narayan-et-al-2025-section}

**[`<em>μ</em>nit Scaling: Simple and Scalable FP8 LLM Training`{=html}](https://arxiv.org/abs/2502.05967 "'<em>μ</em>nit Scaling: Simple and Scalable FP8 LLM Training', Narayan et al 2025"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Leveraging the True Depth of LLMs”, González et al 2025`{=html} {#gonzález-et-al-2025-section}

**[`Leveraging the true depth of LLMs`{=html}](https://arxiv.org/abs/2502.02790 "'Leveraging the true depth of LLMs', González et al 2025"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Language Models Use Trigonometry to Do Addition”, Kantamneni 2025`{=html} {#kantamneni-2025-section}

**[`Language Models Use Trigonometry to Do Addition`{=html}](https://www.lesswrong.com/posts/E7z89FKLsHk5DkmDL/language-models-use-trigonometry-to-do-addition-1 "'Language Models Use Trigonometry to Do Addition', Kantamneni 2025"){.link-annotated-partial
.id-not .include-annotation link-icon="LW" link-icon-type="text"
link-icon-color="#7faf83"}**

## `“Test-Time Regression: a Unifying Framework for Designing Sequence Models With Associative Memory”, Wang et al 2025`{=html} {#wang-et-al-2025-section}

**[`Test-time regression: a unifying framework for designing sequence models with associative memory`{=html}](https://arxiv.org/abs/2501.12352 "'Test-time regression: a unifying framework for designing sequence models with associative memory', Wang et al 2025"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“How Has DeepSeek Improved the Transformer Architecture?”, Erdil 2025`{=html} {#erdil-2025-section}

**[`How has DeepSeek improved the Transformer architecture?`{=html}](https://epoch.ai/gradient-updates/how-has-deepseek-improved-the-transformer-architecture "'How has DeepSeek improved the Transformer architecture?', Erdil 2025"){.link-annotated-partial
.id-not .include-annotation link-icon="deepseek" link-icon-type="svg"
link-icon-color="#4d6bfe"}**

## `“MiniMax-01: Scaling Foundation Models With Lightning Attention”, MiniMax et al 2025`{=html} {#minimax-et-al-2025-section}

**[`MiniMax-01: Scaling Foundation Models with Lightning Attention`{=html}](https://arxiv.org/abs/2501.08313#minimax "'MiniMax-01: Scaling Foundation Models with Lightning Attention', MiniMax et al 2025"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Emergent Effects of Scaling on the Functional Hierarchies within Large Language Models”, Foop 2025`{=html} {#foop-2025-section}

**[`Emergent effects of scaling on the functional hierarchies within large language models`{=html}](https://www.lesswrong.com/posts/LTxkwMGEJXMjYHoSh/emergent-effects-of-scaling-on-the-functional-hierarchies "'Emergent effects of scaling on the functional hierarchies within large language models', Foop 2025"){.link-annotated-partial
.id-not .include-annotation link-icon="LW" link-icon-type="text"
link-icon-color="#7faf83"}**

## `“ICLR: In-Context Learning of Representations”, Park et al 2024`{=html} {#park-et-al-2024-section}

**[`ICLR: In-Context Learning of Representations`{=html}](https://arxiv.org/abs/2501.00070 "'ICLR: In-Context Learning of Representations', Park et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Hymba: A Hybrid-Head Architecture for Small Language Models”, Dong et al 2024`{=html} {#dong-et-al-2024-2-section}

**[`Hymba: A Hybrid-head Architecture for Small Language Models`{=html}](https://arxiv.org/abs/2411.13676 "'Hymba: A Hybrid-head Architecture for Small Language Models', Dong et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models”, Ruis et al 2024`{=html} {#ruis-et-al-2024-section}

**[`Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models`{=html}](https://arxiv.org/abs/2411.12580 "'Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models', Ruis et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“You Could Have Designed State-Of-The-Art Positional Encoding”, Fleetwood 2024`{=html} {#fleetwood-2024-section}

**[`You could have designed state-of-the-art Positional Encoding`{=html}](https://fleetwood.dev/posts/you-could-have-designed-SOTA-positional-encoding "'You could have designed state-of-the-art Positional Encoding', Fleetwood 2024"){.id-not
.include-annotation}**

## `“Long Context RAG Performance of Large Language Models”, Leng et al 2024`{=html} {#leng-et-al-2024-section}

**[`Long Context RAG Performance of Large Language Models`{=html}](https://arxiv.org/abs/2411.03538v1 "'Long Context RAG Performance of Large Language Models', Leng et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Ask, and It Shall Be Given: Turing Completeness of Prompting”, Qiu et al 2024`{=html} {#qiu-et-al-2024-section}

**[`Ask, and it shall be given: Turing completeness of prompting`{=html}](https://arxiv.org/abs/2411.01992 "'Ask, and it shall be given: Turing completeness of prompting', Qiu et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“The Belief State Transformer”, Hu et al 2024`{=html} {#hu-et-al-2024-section}

**[`The Belief State Transformer`{=html}](https://arxiv.org/abs/2410.23506 "'The Belief State Transformer', Hu et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics”, Nikankin et al 2024`{=html} {#nikankin-et-al-2024-section}

**[`Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics`{=html}](https://arxiv.org/abs/2410.21272 "'Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics', Nikankin et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“ALTA: Compiler-Based Analysis of Transformers”, Shaw et al 2024`{=html} {#shaw-et-al-2024-section}

**[`ALTA: Compiler-Based Analysis of Transformers`{=html}](https://arxiv.org/abs/2410.18077#deepmind "'ALTA: Compiler-Based Analysis of Transformers', Shaw et al 2024"){.link-annotated
.id-not .include-annotation link-icon="deepmind" link-icon-type="svg"
link-icon-color="#4185f4"}**

## `“Tackling the Abstraction and Reasoning Corpus With Vision Transformers: the Importance of 2D Representation, Positions, and Objects”, Li et al 2024`{=html} {#li-et-al-2024-2-section}

**[`Tackling the Abstraction and Reasoning Corpus with Vision Transformers: the Importance of 2D Representation, Positions, and Objects`{=html}](https://arxiv.org/abs/2410.06405 "'Tackling the Abstraction and Reasoning Corpus with Vision Transformers: the Importance of 2D Representation, Positions, and Objects', Li et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Differential Transformer”, Ye et al 2024`{=html} {#ye-et-al-2024-1-section}

**[`Differential Transformer`{=html}](https://arxiv.org/abs/2410.05258 "'Differential Transformer', Ye et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Were RNNs All We Needed?”, Feng et al 2024`{=html} {#feng-et-al-2024-1-section}

**[`Were RNNs All We Needed?`{=html}](https://arxiv.org/abs/2410.01201 "'Were RNNs All We Needed?', Feng et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“NGPT: Normalized Transformer With Representation Learning on the Hypersphere”, Loshchilov et al 2024`{=html} {#loshchilov-et-al-2024-section}

**[`nGPT: Normalized Transformer with Representation Learning on the Hypersphere`{=html}](https://arxiv.org/abs/2410.01131 "'nGPT: Normalized Transformer with Representation Learning on the Hypersphere', Loshchilov et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Many-Shot Jailbreaking”, Anil et al 2024`{=html} {#anil-et-al-2024-section}

**[`Many-shot Jailbreaking`{=html}](https://openreview.net/forum?id=cw5mgd71jW#anthropic "'Many-shot Jailbreaking', Anil et al 2024"){.link-annotated
.id-not .include-annotation link-icon="anthropic" link-icon-type="svg"
link-icon-color="#d4a27f"}**

## `“Prompt Baking”, Bhargava et al 2024`{=html} {#bhargava-et-al-2024-section}

**[`Prompt Baking`{=html}](https://arxiv.org/abs/2409.13697 "'Prompt Baking', Bhargava et al 2024"){.link-modified-recently
.link-annotated .id-not .include-annotation link-icon="𝛘"
link-icon-type="text" link-icon-color="#b31b1b"}**

## `“Masked Mixers for Language Generation and Retrieval”, Badger 2024`{=html} {#badger-2024-section}

**[`Masked Mixers for Language Generation and Retrieval`{=html}](https://arxiv.org/abs/2409.01482 "'Masked Mixers for Language Generation and Retrieval', Badger 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“The Mamba in the Llama: Distilling and Accelerating Hybrid Models”, Wang et al 2024`{=html} {#wang-et-al-2024-02-section}

**[`The Mamba in the Llama: Distilling and Accelerating Hybrid Models`{=html}](https://arxiv.org/abs/2408.15237 "'The Mamba in the Llama: Distilling and Accelerating Hybrid Models', Wang et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“When Can Transformers Count to <em>n</em>?”, Yehudai et al 2024`{=html} {#yehudai-et-al-2024-section}

**[`When Can Transformers Count to <em>n</em>?`{=html}](https://arxiv.org/abs/2407.15160#google "'When Can Transformers Count to <em>n</em>?', Yehudai et al 2024"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Just Read Twice: Closing the Recall Gap for Recurrent Language Models”, Arora et al 2024`{=html} {#arora-et-al-2024-section}

**[`Just read twice: closing the recall gap for recurrent language models`{=html}](https://arxiv.org/abs/2407.05483 "'Just read twice: closing the recall gap for recurrent language models', Arora et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“What Matters in Transformers? Not All Attention Is Needed”, He et al 2024`{=html} {#he-et-al-2024-section}

**[`What Matters in Transformers? Not All Attention is Needed`{=html}](https://arxiv.org/abs/2406.15786 "'What Matters in Transformers? Not All Attention is Needed', He et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?”, Lee et al 2024`{=html} {#lee-et-al-2024-1-section}

**[`Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?`{=html}](https://arxiv.org/abs/2406.13121#google "'Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?', Lee et al 2024"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“An Empirical Study of Mamba-Based Language Models”, Waleffe et al 2024`{=html} {#waleffe-et-al-2024-section}

**[`An Empirical Study of Mamba-based Language Models`{=html}](https://arxiv.org/abs/2406.07887 "'An Empirical Study of Mamba-based Language Models', Waleffe et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Attention As a Hypernetwork”, Schug et al 2024`{=html} {#schug-et-al-2024-section}

**[`Attention as a Hypernetwork`{=html}](https://arxiv.org/abs/2406.05816 "'Attention as a Hypernetwork', Schug et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Automata Extraction from Transformers”, Zhang et al 2024`{=html} {#zhang-et-al-2024-section}

**[`Automata Extraction from Transformers`{=html}](https://arxiv.org/abs/2406.05564 "'Automata Extraction from Transformers', Zhang et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Scalable Matmul-Free Language Modeling”, Zhu et al 2024`{=html} {#zhu-et-al-2024-2-section}

**[`Scalable Matmul-free Language Modeling`{=html}](https://arxiv.org/abs/2406.02528 "'Scalable Matmul-free Language Modeling', Zhu et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“A Theoretical Understanding of Self-Correction through In-Context Alignment”, Wang et al 2024`{=html} {#wang-et-al-2024-07-section}

**[`A Theoretical Understanding of Self-Correction through In-context Alignment`{=html}](https://arxiv.org/abs/2405.18634 "'A Theoretical Understanding of Self-Correction through In-context Alignment', Wang et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Attention As an RNN”, Feng et al 2024`{=html} {#feng-et-al-2024-3-section}

**[`Attention as an RNN`{=html}](https://arxiv.org/abs/2405.13956#borealis "'Attention as an RNN', Feng et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Your Transformer Is Secretly Linear”, Razzhigaev et al 2024`{=html} {#razzhigaev-et-al-2024-section}

**[`Your Transformer is Secretly Linear`{=html}](https://arxiv.org/abs/2405.12250 "'Your Transformer is Secretly Linear', Razzhigaev et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Retrieval Head Mechanistically Explains Long-Context Factuality”, Wu et al 2024`{=html} {#wu-et-al-2024-1-section}

**[`Retrieval Head Mechanistically Explains Long-Context Factuality`{=html}](https://arxiv.org/abs/2404.15574 "'Retrieval Head Mechanistically Explains Long-Context Factuality', Wu et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Let’s Think Dot by Dot: Hidden Computation in Transformer Language Models”, Pfau et al 2024`{=html} {#pfau-et-al-2024-section}

**[`Let’s Think Dot by Dot: Hidden Computation in Transformer Language Models`{=html}](https://arxiv.org/abs/2404.15758 "'Let’s Think Dot by Dot: Hidden Computation in Transformer Language Models', Pfau et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Towards Smaller, Faster Decoder-Only Transformers: Architectural Variants and Their Implications”, Suresh & P 2024`{=html} {#suresh-p-2024-section}

**[`Towards smaller, faster decoder-only transformers: Architectural variants and their implications`{=html}](https://arxiv.org/abs/2404.14462 "'Towards smaller, faster decoder-only transformers: Architectural variants and their implications', Suresh & P 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“RULER: What’s the Real Context Size of Your Long-Context Language Models?”, Hsieh et al 2024`{=html} {#hsieh-et-al-2024-section}

**[`RULER: What’s the Real Context Size of Your Long-Context Language Models?`{=html}](https://arxiv.org/abs/2404.06654#nvidia "'RULER: What’s the Real Context Size of Your Long-Context Language Models?', Hsieh et al 2024"){.link-annotated
.id-not .include-annotation link-icon="n"
link-icon-type="text,sans,italic" link-icon-color="#77ba00"}**

## `“ReFT: Representation Finetuning for Language Models”, Wu et al 2024`{=html} {#wu-et-al-2024-3-section}

**[`ReFT: Representation Finetuning for Language Models`{=html}](https://arxiv.org/abs/2404.03592 "'ReFT: Representation Finetuning for Language Models', Wu et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Many-Shot Jailbreaking”, Anthropic 2024`{=html} {#anthropic-2024-section}

**[`Many-shot jailbreaking`{=html}](https://www.anthropic.com/research/many-shot-jailbreaking "'Many-shot jailbreaking', Anthropic 2024"){.link-annotated-partial
.id-not .include-annotation link-icon="anthropic" link-icon-type="svg"
link-icon-color="#d4a27f"}**

## `“Do Language Models Plan Ahead for Future Tokens?”, Wu et al 2024`{=html} {#wu-et-al-2024-4-section}

**[`Do language models plan ahead for future tokens?`{=html}](https://arxiv.org/abs/2404.00859 "'Do language models plan ahead for future tokens?', Wu et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Streamlining Redundant Layers to Compress Large Language Models”, Chen et al 2024`{=html} {#chen-et-al-2024-6-section}

**[`Streamlining Redundant Layers to Compress Large Language Models`{=html}](https://arxiv.org/abs/2403.19135 "'Streamlining Redundant Layers to Compress Large Language Models', Chen et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Long-Form Factuality in Large Language Models”, Wei et al 2024`{=html} {#wei-et-al-2024-1-section}

**[`Long-form factuality in large language models`{=html}](https://arxiv.org/abs/2403.18802#deepmind "'Long-form factuality in large language models', Wei et al 2024"){.link-annotated
.id-not .include-annotation link-icon="deepmind" link-icon-type="svg"
link-icon-color="#4185f4"}**

## `“Mechanistic Design and Scaling of Hybrid Architectures”, Poli et al 2024`{=html} {#poli-et-al-2024-section}

**[`Mechanistic Design and Scaling of Hybrid Architectures`{=html}](https://arxiv.org/abs/2403.17844 "'Mechanistic Design and Scaling of Hybrid Architectures', Poli et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“8 Google Employees Invented Modern AI. Here’s the Inside Story: They Met by Chance, Got Hooked on an Idea, and Wrote the Transformers Paper—The Most Consequential Tech Breakthrough in Recent History”, Levy 2024`{=html} {#levy-2024-2-section}

**[`8 Google Employees Invented Modern AI. Here’s the Inside Story: They met by chance, got hooked on an idea, and wrote the Transformers paper—the most consequential tech breakthrough in recent history`{=html}](https://www.wired.com/story/eight-google-employees-invented-modern-ai-transformers-paper/ "'8 Google Employees Invented Modern AI. Here’s the Inside Story: They met by chance, got hooked on an idea, and wrote the Transformers paper—the most consequential tech breakthrough in recent history', Levy 2024"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“RNNs Are Not Transformers (Yet): The Key Bottleneck on In-Context Retrieval”, Wen et al 2024`{=html} {#wen-et-al-2024-2-section}

**[`RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval`{=html}](https://arxiv.org/abs/2402.18510 "'RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval', Wen et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“How Transformers Learn Causal Structure With Gradient Descent”, Nichani et al 2024`{=html} {#nichani-et-al-2024-section}

**[`How Transformers Learn Causal Structure with Gradient Descent`{=html}](https://arxiv.org/abs/2402.14735 "'How Transformers Learn Causal Structure with Gradient Descent', Nichani et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“A Phase Transition between Positional and Semantic Learning in a Solvable Model of Dot-Product Attention”, Cui et al 2024`{=html} {#cui-et-al-2024-section}

**[`A phase transition between positional and semantic learning in a solvable model of dot-product attention`{=html}](https://arxiv.org/abs/2402.03902 "'A phase transition between positional and semantic learning in a solvable model of dot-product attention', Cui et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Rethinking Patch Dependence for Masked Autoencoders”, Fu et al 2024`{=html} {#fu-et-al-2024-2-section}

**[`Rethinking Patch Dependence for Masked Autoencoders`{=html}](https://arxiv.org/abs/2401.14391 "'Rethinking Patch Dependence for Masked Autoencoders', Fu et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Attention versus Contrastive Learning of Tabular Data—A Data-Centric Benchmarking”, Rabbani et al 2024`{=html} {#rabbani-et-al-2024-section}

**[`Attention versus Contrastive Learning of Tabular Data—A Data-centric Benchmarking`{=html}](https://arxiv.org/abs/2401.04266 "'Attention versus Contrastive Learning of Tabular Data—A Data-centric Benchmarking', Rabbani et al 2024"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet”`{=html} {#_MN0ws55D-section}

**[`Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet`{=html}](https://transformer-circuits.pub/2024/scaling-monosemanticity/ "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet"){.link-annotated-partial
.id-not .include-annotation link-icon="anthropic" link-icon-type="svg"
link-icon-color="#d4a27f"}**

## `“Successor Heads: Recurring, Interpretable Attention Heads In The Wild”, Gould et al 2023`{=html} {#gould-et-al-2023-section}

**[`Successor Heads: Recurring, Interpretable Attention Heads In The Wild`{=html}](https://arxiv.org/abs/2312.09230 "'Successor Heads: Recurring, Interpretable Attention Heads In The Wild', Gould et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“SwitchHead: Accelerating Transformers With Mixture-Of-Experts Attention”, Csordás et al 2023`{=html} {#csordás-et-al-2023-section}

**[`SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention`{=html}](https://arxiv.org/abs/2312.07987#schmidhuber "'SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention', Csordás et al 2023"){.link-annotated
.id-not .include-annotation link-icon="SMDH"
link-icon-type="text,quad,sans"}**

## `“Look Before You Leap: A Universal Emergent Decomposition of Retrieval Tasks in Language Models”, Variengien & Winsor 2023`{=html} {#variengien-winsor-2023-section}

**[`Look Before You Leap: A Universal Emergent Decomposition of Retrieval Tasks in Language Models`{=html}](https://arxiv.org/abs/2312.10091#conjecture "'Look Before You Leap: A Universal Emergent Decomposition of Retrieval Tasks in Language Models', Variengien & Winsor 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Can a Transformer Represent a Kalman Filter?”, Goel & Bartlett 2023`{=html} {#goel-bartlett-2023-section}

**[`Can a Transformer Represent a Kalman Filter?`{=html}](https://arxiv.org/abs/2312.06937 "'Can a Transformer Represent a Kalman Filter?', Goel & Bartlett 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Long Context Prompting for Claude 2.1: Claude 2.1 Excels at Retrieving Information across Its 200K Context Window, With a Simple Prompt Adjustment Improving Accuracy from 27% to 98%”, Anthropic 2023`{=html} {#anthropic-2023-section}

**[`Long context prompting for Claude 2.1: Claude 2.1 excels at retrieving information across its 200K context window, with a simple prompt adjustment improving accuracy from 27% to 98%`{=html}](https://www.anthropic.com/news/claude-2-1-prompting "'Long context prompting for Claude 2.1: Claude 2.1 excels at retrieving information across its 200K context window, with a simple prompt adjustment improving accuracy from 27% to 98%', Anthropic 2023"){.link-annotated-partial
.id-not .include-annotation link-icon="anthropic" link-icon-type="svg"
link-icon-color="#d4a27f"}**

## `“Efficient Transformer Knowledge Distillation: A Performance Review”, Brown et al 2023`{=html} {#brown-et-al-2023-section}

**[`Efficient Transformer Knowledge Distillation: A Performance Review`{=html}](https://arxiv.org/abs/2311.13657 "'Efficient Transformer Knowledge Distillation: A Performance Review', Brown et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks As an Alternative to Attention Layers in Transformers”, Bozic et al 2023`{=html} {#bozic-et-al-2023-section}

**[`Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers`{=html}](https://arxiv.org/abs/2311.10642 "'Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers', Bozic et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“In-Context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering”, Liu et al 2023`{=html} {#liu-et-al-2023-03-section}

**[`In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering`{=html}](https://arxiv.org/abs/2311.06668 "'In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering', Liu et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“On Prefrontal Working Memory and Hippocampal Episodic Memory: Unifying Memories Stored in Weights and Activation Slots”, Whittington et al 2023`{=html} {#whittington-et-al-2023-section}

**[`On prefrontal working memory and hippocampal episodic memory: Unifying memories stored in weights and activation slots`{=html}](https://www.biorxiv.org/content/10.1101/2023.11.05.565662.full "'On prefrontal working memory and hippocampal episodic memory: Unifying memories stored in weights and activation slots', Whittington et al 2023"){.link-annotated
.id-not .include-annotation link-icon="chi-dna" link-icon-type="svg"
link-icon-color="#bd2736"}**

## `“LSS Transformer: Ultra-Long Sequence Distributed Transformer”, Wang et al 2023`{=html} {#wang-et-al-2023-08-section}

**[`LSS Transformer: Ultra-Long Sequence Distributed Transformer`{=html}](https://arxiv.org/abs/2311.02382 "'LSS Transformer: Ultra-Long Sequence Distributed Transformer', Wang et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Simplifying Transformer Blocks”, He & Hofmann 2023`{=html} {#he-hofmann-2023-section}

**[`Simplifying Transformer Blocks`{=html}](https://arxiv.org/abs/2311.01906 "'Simplifying Transformer Blocks', He & Hofmann 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling”, Katsch 2023`{=html} {#katsch-2023-section}

**[`GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling`{=html}](https://arxiv.org/abs/2311.01927 "'GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling', Katsch 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Not All Layers Are Equally As Important: Every Layer Counts BERT”, Charpentier & Samuel 2023`{=html} {#charpentier-samuel-2023-section}

**[`Not all layers are equally as important: Every Layer Counts BERT`{=html}](https://arxiv.org/abs/2311.02265 "'Not all layers are equally as important: Every Layer Counts BERT', Charpentier & Samuel 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Implicit Chain-Of-Thought Reasoning via Knowledge Distillation”, Deng et al 2023`{=html} {#deng-et-al-2023-2-section}

**[`Implicit Chain-of-Thought Reasoning via Knowledge Distillation`{=html}](https://arxiv.org/abs/2311.01460 "'Implicit Chain-of-Thought Reasoning via Knowledge Distillation', Deng et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Training Dynamics of Contextual N-Grams in Language Models”, Quirke et al 2023`{=html} {#quirke-et-al-2023-section}

**[`Training Dynamics of Contextual N-Grams in Language Models`{=html}](https://arxiv.org/abs/2311.00863 "'Training Dynamics of Contextual N-Grams in Language Models', Quirke et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“The Impact of Depth and Width on Transformer Language Model Generalization”, Petty et al 2023`{=html} {#petty-et-al-2023-section}

**[`The Impact of Depth and Width on Transformer Language Model Generalization`{=html}](https://arxiv.org/abs/2310.19956 "'The Impact of Depth and Width on Transformer Language Model Generalization', Petty et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Characterizing Mechanisms for Factual Recall in Language Models”, Yu et al 2023`{=html} {#yu-et-al-2023-3-section}

**[`Characterizing Mechanisms for Factual Recall in Language Models`{=html}](https://arxiv.org/abs/2310.15910 "'Characterizing Mechanisms for Factual Recall in Language Models', Yu et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Linear Representations of Sentiment in Large Language Models”, Tigges et al 2023`{=html} {#tigges-et-al-2023-section}

**[`Linear Representations of Sentiment in Large Language Models`{=html}](https://arxiv.org/abs/2310.15154 "'Linear Representations of Sentiment in Large Language Models', Tigges et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Masked Hard-Attention Transformers and Boolean RASP Recognize Exactly the Star-Free Languages”, Angluin et al 2023`{=html} {#angluin-et-al-2023-section}

**[`Masked Hard-Attention Transformers and Boolean RASP Recognize Exactly the Star-Free Languages`{=html}](https://arxiv.org/abs/2310.13897 "'Masked Hard-Attention Transformers and Boolean RASP Recognize Exactly the Star-Free Languages', Angluin et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?”, Wu et al 2023`{=html} {#wu-et-al-2023-1-section}

**[`How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?`{=html}](https://arxiv.org/abs/2310.08391 "'How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?', Wu et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors”, Amos et al 2023`{=html} {#amos-et-al-2023-section}

**[`Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors`{=html}](https://arxiv.org/abs/2310.02980 "'Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors', Amos et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Vision Transformers Need Registers”, Darcet et al 2023`{=html} {#darcet-et-al-2023-section}

**[`Vision Transformers Need Registers`{=html}](https://arxiv.org/abs/2309.16588 "'Vision Transformers Need Registers', Darcet et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Interpret Vision Transformers As ConvNets With Dynamic Convolutions”, Zhou et al 2023`{=html} {#zhou-et-al-2023-05-section}

**[`Interpret Vision Transformers as ConvNets with Dynamic Convolutions`{=html}](https://arxiv.org/abs/2309.10713 "'Interpret Vision Transformers as ConvNets with Dynamic Convolutions', Zhou et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Replacing Softmax With ReLU in Vision Transformers”, Wortsman et al 2023`{=html} {#wortsman-et-al-2023-section}

**[`Replacing softmax with ReLU in Vision Transformers`{=html}](https://arxiv.org/abs/2309.08586 "'Replacing softmax with ReLU in Vision Transformers', Wortsman et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Gated Recurrent Neural Networks Discover Attention”, Zucchet et al 2023`{=html} {#zucchet-et-al-2023-section}

**[`Gated recurrent neural networks discover attention`{=html}](https://arxiv.org/abs/2309.01775 "'Gated recurrent neural networks discover attention', Zucchet et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“One Wide Feedforward Is All You Need”, Pires et al 2023`{=html} {#pires-et-al-2023-section}

**[`One Wide Feedforward is All You Need`{=html}](https://arxiv.org/abs/2309.01826#apple "'One Wide Feedforward is All You Need', Pires et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Activation Addition: Steering Language Models Without Optimization”, Turner et al 2023`{=html} {#turner-et-al-2023-section}

**[`Activation Addition: Steering Language Models Without Optimization`{=html}](https://arxiv.org/abs/2308.10248 "'Activation Addition: Steering Language Models Without Optimization', Turner et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Linearity of Relation Decoding in Transformer Language Models”, Hernandez et al 2023`{=html} {#hernandez-et-al-2023-section}

**[`Linearity of Relation Decoding in Transformer Language Models`{=html}](https://arxiv.org/abs/2308.09124 "'Linearity of Relation Decoding in Transformer Language Models', Hernandez et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“The Hydra Effect: Emergent Self-Repair in Language Model Computations”, McGrath et al 2023`{=html} {#mcgrath-et-al-2023-1-section}

**[`The Hydra Effect: Emergent Self-repair in Language Model Computations`{=html}](https://arxiv.org/abs/2307.15771#deepmind "'The Hydra Effect: Emergent Self-repair in Language Model Computations', McGrath et al 2023"){.link-annotated
.id-not .include-annotation link-icon="deepmind" link-icon-type="svg"
link-icon-color="#4185f4"}**

## `“Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla”, Lieberum et al 2023`{=html} {#lieberum-et-al-2023-section}

**[`Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla`{=html}](https://arxiv.org/abs/2307.09458#deepmind "'Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla', Lieberum et al 2023"){.link-annotated
.id-not .include-annotation link-icon="deepmind" link-icon-type="svg"
link-icon-color="#4185f4"}**

## `“FlashAttention-2: Faster Attention With Better Parallelism and Work Partitioning”, Dao 2023`{=html} {#dao-2023-section}

**[`FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning`{=html}](https://arxiv.org/abs/2307.08691 "'FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning', Dao 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Lost in the Middle: How Language Models Use Long Contexts”, Liu et al 2023`{=html} {#liu-et-al-2023-12-section}

**[`Lost in the Middle: How Language Models Use Long Contexts`{=html}](https://arxiv.org/abs/2307.03172 "'Lost in the Middle: How Language Models Use Long Contexts', Liu et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Trainable Transformer in Transformer”, Panigrahi et al 2023`{=html} {#panigrahi-et-al-2023-section}

**[`Trainable Transformer in Transformer`{=html}](https://arxiv.org/abs/2307.01189 "'Trainable Transformer in Transformer', Panigrahi et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Transformers Learn to Implement Preconditioned Gradient Descent for In-Context Learning”, Ahn et al 2023`{=html} {#ahn-et-al-2023-section}

**[`Transformers learn to implement preconditioned gradient descent for in-context learning`{=html}](https://arxiv.org/abs/2306.00297 "'Transformers learn to implement preconditioned gradient descent for in-context learning', Ahn et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“White-Box Transformers via Sparse Rate Reduction”, Yu et al 2023`{=html} {#yu-et-al-2023-5-section}

**[`White-Box Transformers via Sparse Rate Reduction`{=html}](https://arxiv.org/abs/2306.01129 "'White-Box Transformers via Sparse Rate Reduction', Yu et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Blockwise Parallel Transformer for Long Context Large Models”, Liu & Abbeel 2023`{=html} {#liu-abbeel-2023-section}

**[`Blockwise Parallel Transformer for Long Context Large Models`{=html}](https://arxiv.org/abs/2305.19370 "'Blockwise Parallel Transformer for Long Context Large Models', Liu & Abbeel 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“TTT-NN: Test-Time Training on Nearest Neighbors for Large Language Models”, Hardt & Sun 2023`{=html} {#hardt-sun-2023-section}

**[`TTT-NN: Test-Time Training on Nearest Neighbors for Large Language Models`{=html}](https://arxiv.org/abs/2305.18466 "'TTT-NN: Test-Time Training on Nearest Neighbors for Large Language Models', Hardt & Sun 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Brainformers: Trading Simplicity for Efficiency”, Zhou et al 2023`{=html} {#zhou-et-al-2023-08-section}

**[`Brainformers: Trading Simplicity for Efficiency`{=html}](https://arxiv.org/abs/2306.00008#google "'Brainformers: Trading Simplicity for Efficiency', Zhou et al 2023"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints”, Ainslie et al 2023`{=html} {#ainslie-et-al-2023-section}

**[`GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints`{=html}](https://arxiv.org/abs/2305.13245#google "'GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints', Ainslie et al 2023"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Mimetic Initialization of Self-Attention Layers”, Trockman & Kolter 2023`{=html} {#trockman-kolter-2023-section}

**[`Mimetic Initialization of Self-Attention Layers`{=html}](https://arxiv.org/abs/2305.09828 "'Mimetic Initialization of Self-Attention Layers', Trockman & Kolter 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Simplicial Hopfield Networks”, Burns & Fukai 2023`{=html} {#burns-fukai-2023-section}

**[`Simplicial Hopfield networks`{=html}](https://arxiv.org/abs/2305.05179 "'Simplicial Hopfield networks', Burns & Fukai 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Toeplitz Neural Network for Sequence Modeling”, Qin et al 2023`{=html} {#qin-et-al-2023-3-section}

**[`Toeplitz Neural Network for Sequence Modeling`{=html}](https://arxiv.org/abs/2305.04749 "'Toeplitz Neural Network for Sequence Modeling', Qin et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Finding Neurons in a Haystack: Case Studies With Sparse Probing”, Gurnee et al 2023`{=html} {#gurnee-et-al-2023-section}

**[`Finding Neurons in a Haystack: Case Studies with Sparse Probing`{=html}](https://arxiv.org/abs/2305.01610 "'Finding Neurons in a Haystack: Case Studies with Sparse Probing', Gurnee et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“How Does GPT-2 Compute Greater-Than?: Interpreting Mathematical Abilities in a Pre-Trained Language Model”, Hanna et al 2023`{=html} {#hanna-et-al-2023-section}

**[`How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model`{=html}](https://arxiv.org/abs/2305.00586 "'How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model', Hanna et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Theory of Coupled Neuronal-Synaptic Dynamics”, Clark & Abbott 2023`{=html} {#clark-abbott-2023-section}

**[`Theory of coupled neuronal-synaptic dynamics`{=html}](https://arxiv.org/abs/2302.08985 "'Theory of coupled neuronal-synaptic dynamics', Clark & Abbott 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Coinductive Guide to Inductive Transformer Heads”, Nemecek 2023`{=html} {#nemecek-2023-section}

**[`Coinductive guide to inductive transformer heads`{=html}](https://arxiv.org/abs/2302.01834 "'Coinductive guide to inductive transformer heads', Nemecek 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Tighter Bounds on the Expressivity of Transformer Encoders”, Chiang et al 2023`{=html} {#chiang-et-al-2023-2-section}

**[`Tighter Bounds on the Expressivity of Transformer Encoders`{=html}](https://arxiv.org/abs/2301.10743 "'Tighter Bounds on the Expressivity of Transformer Encoders', Chiang et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Tracr: Compiled Transformers As a Laboratory for Interpretability”, Lindner et al 2023`{=html} {#lindner-et-al-2023-section}

**[`Tracr: Compiled Transformers as a Laboratory for Interpretability`{=html}](https://arxiv.org/abs/2301.05062#deepmind "'Tracr: Compiled Transformers as a Laboratory for Interpretability', Lindner et al 2023"){.link-annotated
.id-not .include-annotation link-icon="deepmind" link-icon-type="svg"
link-icon-color="#4185f4"}**

## `“Skip-Attention: Improving Vision Transformers by Paying Less Attention”, Venkataramanan et al 2023`{=html} {#venkataramanan-et-al-2023-section}

**[`Skip-Attention: Improving Vision Transformers by Paying Less Attention`{=html}](https://arxiv.org/abs/2301.02240 "'Skip-Attention: Improving Vision Transformers by Paying Less Attention', Venkataramanan et al 2023"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Hungry Hungry Hippos: Towards Language Modeling With State Space Models”, Fu et al 2022`{=html} {#fu-et-al-2022-1-section}

**[`Hungry Hungry Hippos: Towards Language Modeling with State Space Models`{=html}](https://arxiv.org/abs/2212.14052 "'Hungry Hungry Hippos: Towards Language Modeling with State Space Models', Fu et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Scalable Adaptive Computation for Iterative Generation”, Jabri et al 2022`{=html} {#jabri-et-al-2022-section}

**[`Scalable Adaptive Computation for Iterative Generation`{=html}](https://arxiv.org/abs/2212.11972#google "'Scalable Adaptive Computation for Iterative Generation', Jabri et al 2022"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Pretraining Without Attention”, Wang et al 2022`{=html} {#wang-et-al-2022-05-section}

**[`Pretraining Without Attention`{=html}](https://arxiv.org/abs/2212.10544 "'Pretraining Without Attention', Wang et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Transformers Learn In-Context by Gradient Descent”, Oswald et al 2022`{=html} {#oswald-et-al-2022-section}

**[`Transformers learn in-context by gradient descent`{=html}](https://arxiv.org/abs/2212.07677#google "'Transformers learn in-context by gradient descent', Oswald et al 2022"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Efficiently Scaling Transformer Inference”, Pope et al 2022`{=html} {#pope-et-al-2022-section}

**[`Efficiently Scaling Transformer Inference`{=html}](https://arxiv.org/abs/2211.05102#google "'Efficiently Scaling Transformer Inference', Pope et al 2022"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Transformers Learn Shortcuts to Automata”, Liu et al 2022`{=html} {#liu-et-al-2022-section}

**[`Transformers Learn Shortcuts to Automata`{=html}](https://arxiv.org/abs/2210.10749#microsoft "'Transformers Learn Shortcuts to Automata', Liu et al 2022"){.link-annotated
.id-not .include-annotation link-icon="MS"
link-icon-type="text,sans,italic"}**

## `“Multi-CLS BERT: An Efficient Alternative to Traditional Ensembling”, Chang et al 2022`{=html} {#chang-et-al-2022-3-section}

**[`Multi-CLS BERT: An Efficient Alternative to Traditional Ensembling`{=html}](https://arxiv.org/abs/2210.05043 "'Multi-CLS BERT: An Efficient Alternative to Traditional Ensembling', Chang et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Transformers Implement First-Order Logic With Majority Quantifiers”, Merrill & Sabharwal 2022`{=html} {#merrill-sabharwal-2022-1-section}

**[`Transformers Implement First-Order Logic with Majority Quantifiers`{=html}](https://arxiv.org/abs/2210.02671 "'Transformers Implement First-Order Logic with Majority Quantifiers', Merrill & Sabharwal 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“The Lie Derivative for Measuring Learned Equivariance”, Gruver et al 2022`{=html} {#gruver-et-al-2022-section}

**[`The Lie Derivative for Measuring Learned Equivariance`{=html}](https://arxiv.org/abs/2210.02984 "'The Lie Derivative for Measuring Learned Equivariance', Gruver et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Relaxed Attention for Transformer Models”, Lohrenz et al 2022`{=html} {#lohrenz-et-al-2022-section}

**[`Relaxed Attention for Transformer Models`{=html}](https://arxiv.org/abs/2209.09735 "'Relaxed Attention for Transformer Models', Lohrenz et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Multitrack Music Transformer: Learning Long-Term Dependencies in Music With Diverse Instruments”, Dong et al 2022`{=html} {#dong-et-al-2022-5-section}

**[`Multitrack Music Transformer: Learning Long-Term Dependencies in Music with Diverse Instruments`{=html}](https://arxiv.org/abs/2207.06983 "'Multitrack Music Transformer: Learning Long-Term Dependencies in Music with Diverse Instruments', Dong et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“<em>N</em>-Grammer: Augmenting Transformers With Latent <em>n</em>-Grams”, Roy et al 2022`{=html} {#roy-et-al-2022-section}

**[`<em>N</em>-Grammer: Augmenting Transformers with latent <em>n</em>-grams`{=html}](https://arxiv.org/abs/2207.06366#google "'<em>N</em>-Grammer: Augmenting Transformers with latent <em>n</em>-grams', Roy et al 2022"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Log-Precision Transformers Are Constant-Depth Uniform Threshold Circuits”, Merrill & Sabharwal 2022`{=html} {#merrill-sabharwal-2022-2-section}

**[`Log-Precision Transformers are Constant-Depth Uniform Threshold Circuits`{=html}](https://arxiv.org/abs/2207.00729 "'Log-Precision Transformers are Constant-Depth Uniform Threshold Circuits', Merrill & Sabharwal 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules”, Irie et al 2022`{=html} {#irie-et-al-2022-1-section}

**[`Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules`{=html}](https://arxiv.org/abs/2206.01649#schmidhuber "'Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules', Irie et al 2022"){.link-annotated
.id-not .include-annotation link-icon="SMDH"
link-icon-type="text,quad,sans"}**

## `“FlashAttention: Fast and Memory-Efficient Exact Attention With IO-Awareness”, Dao et al 2022`{=html} {#dao-et-al-2022-1-section}

**[`FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness`{=html}](https://arxiv.org/abs/2205.14135 "'FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness', Dao et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Simplicial Attention Networks”, Goh et al 2022`{=html} {#goh-et-al-2022-section}

**[`Simplicial Attention Networks`{=html}](https://arxiv.org/abs/2204.09455 "'Simplicial Attention Networks', Goh et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“TATS: Long Video Generation With Time-Agnostic VQGAN and Time-Sensitive Transformer”, Ge et al 2022`{=html} {#ge-et-al-2022-2-section}

**[`TATS: Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer`{=html}](https://arxiv.org/abs/2204.03638#facebook "'TATS: Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer', Ge et al 2022"){.link-annotated
.id-not .include-annotation link-icon="facebook" link-icon-type="svg"
link-icon-color="#1877f2"}**

## `“Transformer Language Models without Positional Encodings Still Learn Positional Information”, Haviv et al 2022`{=html} {#haviv-et-al-2022-section}

**[`Transformer Language Models without Positional Encodings Still Learn Positional Information`{=html}](https://arxiv.org/abs/2203.16634 "'Transformer Language Models without Positional Encodings Still Learn Positional Information', Haviv et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Overcoming a Theoretical Limitation of Self-Attention”, Chiang & Cholak 2022`{=html} {#chiang-cholak-2022-section}

**[`Overcoming a Theoretical Limitation of Self-Attention`{=html}](https://arxiv.org/abs/2202.12172 "'Overcoming a Theoretical Limitation of Self-Attention', Chiang & Cholak 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“It’s Raw! Audio Generation With State-Space Models”, Goel et al 2022`{=html} {#goel-et-al-2022-2-section}

**[`It’s Raw! Audio Generation with State-Space Models`{=html}](https://arxiv.org/abs/2202.09729 "'It’s Raw! Audio Generation with State-Space Models', Goel et al 2022"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“General-Purpose, Long-Context Autoregressive Modeling With Perceiver AR”, Hawthorne et al 2022`{=html} {#hawthorne-et-al-2022-section}

**[`General-purpose, long-context autoregressive modeling with Perceiver AR`{=html}](https://arxiv.org/abs/2202.07765#deepmind "'General-purpose, long-context autoregressive modeling with Perceiver AR', Hawthorne et al 2022"){.link-annotated
.id-not .include-annotation link-icon="deepmind" link-icon-type="svg"
link-icon-color="#4185f4"}**

## `“Transformer Memory As a Differentiable Search Index”, Tay et al 2022`{=html} {#tay-et-al-2022-2-section}

**[`Transformer Memory as a Differentiable Search Index`{=html}](https://arxiv.org/abs/2202.06991#google "'Transformer Memory as a Differentiable Search Index', Tay et al 2022"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention”, Irie et al 2022`{=html} {#irie-et-al-2022-2-section}

**[`The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention`{=html}](https://arxiv.org/abs/2202.05798#schmidhuber "'The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention', Irie et al 2022"){.link-annotated
.id-not .include-annotation link-icon="SMDH"
link-icon-type="text,quad,sans"}**

## `“Attention Approximates Sparse Distributed Memory”, Bricken & Pehlevan 2021`{=html} {#bricken-pehlevan-2021-section}

**[`Attention Approximates Sparse Distributed Memory`{=html}](https://arxiv.org/abs/2111.05498 "'Attention Approximates Sparse Distributed Memory', Bricken & Pehlevan 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Long-Range Transformers for Dynamic Spatiotemporal Forecasting”, Grigsby et al 2021`{=html} {#grigsby-et-al-2021-section}

**[`Long-Range Transformers for Dynamic Spatiotemporal Forecasting`{=html}](https://arxiv.org/abs/2109.12218 "'Long-Range Transformers for Dynamic Spatiotemporal Forecasting', Grigsby et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Train Short, Test Long: Attention With Linear Biases (ALiBi) Enables Input Length Extrapolation”, Press et al 2021`{=html} {#press-et-al-2021-section}

**[`Train Short, Test Long: Attention with Linear Biases (ALiBi) Enables Input Length Extrapolation`{=html}](https://arxiv.org/abs/2108.12409#facebook "'Train Short, Test Long: Attention with Linear Biases (ALiBi) Enables Input Length Extrapolation', Press et al 2021"){.link-annotated
.id-not .include-annotation link-icon="facebook" link-icon-type="svg"
link-icon-color="#1877f2"}**

## `“Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-Identification”, Rao et al 2021`{=html} {#rao-et-al-2021-section}

**[`Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification`{=html}](https://arxiv.org/abs/2108.08728 "'Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification', Rao et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Do Vision Transformers See Like Convolutional Neural Networks?”, Raghu et al 2021`{=html} {#raghu-et-al-2021-section}

**[`Do Vision Transformers See Like Convolutional Neural Networks?`{=html}](https://arxiv.org/abs/2108.08810#google "'Do Vision Transformers See Like Convolutional Neural Networks?', Raghu et al 2021"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Stable, Fast and Accurate: Kernelized Attention With Relative Positional Encoding”, Luo et al 2021`{=html} {#luo-et-al-2021-section}

**[`Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding`{=html}](https://arxiv.org/abs/2106.12566 "'Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding', Luo et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“RASP: Thinking Like Transformers”, Weiss et al 2021`{=html} {#weiss-et-al-2021-section}

**[`RASP: Thinking Like Transformers`{=html}](https://arxiv.org/abs/2106.06981 "'RASP: Thinking Like Transformers', Weiss et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“On the Distribution, Sparsity, and Inference-Time Quantization of Attention Values in Transformers”, Ji et al 2021`{=html} {#ji-et-al-2021-section}

**[`On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers`{=html}](https://arxiv.org/abs/2106.01335 "'On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers', Ji et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training”, Somepalli et al 2021`{=html} {#somepalli-et-al-2021-section}

**[`SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training`{=html}](https://arxiv.org/abs/2106.01342 "'SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training', Somepalli et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Not All Images Are Worth 16×16 Words: Dynamic Transformers for Efficient Image Recognition”, Wang et al 2021`{=html} {#wang-et-al-2021-06-section}

**[`Not All Images are Worth 16×16 Words: Dynamic Transformers for Efficient Image Recognition`{=html}](https://arxiv.org/abs/2105.15075 "'Not All Images are Worth 16×16 Words: Dynamic Transformers for Efficient Image Recognition', Wang et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Less Is More: Pay Less Attention in Vision Transformers”, Pan et al 2021`{=html} {#pan-et-al-2021-section}

**[`Less is More: Pay Less Attention in Vision Transformers`{=html}](https://arxiv.org/abs/2105.14217 "'Less is More: Pay Less Attention in Vision Transformers', Pan et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“FNet: Mixing Tokens With Fourier Transforms”, Lee-Thorp et al 2021`{=html} {#lee-thorp-et-al-2021-section}

**[`FNet: Mixing Tokens with Fourier Transforms`{=html}](https://arxiv.org/abs/2105.03824#google "'FNet: Mixing Tokens with Fourier Transforms', Lee-Thorp et al 2021"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet”, Melas-Kyriazi 2021`{=html} {#melas-kyriazi-2021-section}

**[`Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet`{=html}](https://arxiv.org/abs/2105.02723 "'Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet', Melas-Kyriazi 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“RoFormer: Enhanced Transformer With Rotary Position Embedding”, Su et al 2021`{=html} {#su-et-al-2021-2-section}

**[`RoFormer: Enhanced Transformer with Rotary Position Embedding`{=html}](https://arxiv.org/abs/2104.09864 "'RoFormer: Enhanced Transformer with Rotary Position Embedding', Su et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“ALD: Efficient Transformers in Reinforcement Learning Using Actor-Learner Distillation”, Parisotto & Salakhutdinov 2021`{=html} {#parisotto-salakhutdinov-2021-section}

**[`ALD: Efficient Transformers in Reinforcement Learning using Actor-Learner Distillation`{=html}](https://arxiv.org/abs/2104.01655 "'ALD: Efficient Transformers in Reinforcement Learning using Actor-Learner Distillation', Parisotto & Salakhutdinov 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Attention Is Not All You Need: Pure Attention Loses Rank Doubly Exponentially With Depth”, Dong et al 2021`{=html} {#dong-et-al-2021-1-section}

**[`Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth`{=html}](https://arxiv.org/abs/2103.03404 "'Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth', Dong et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Do Transformer Modifications Transfer Across Implementations and Applications?”, Narang et al 2021`{=html} {#narang-et-al-2021-section}

**[`Do Transformer Modifications Transfer Across Implementations and Applications?`{=html}](https://arxiv.org/abs/2102.11972#google "'Do Transformer Modifications Transfer Across Implementations and Applications?', Narang et al 2021"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Linear Transformers Are Secretly Fast Weight Programmers”, Schlag et al 2021`{=html} {#schlag-et-al-2021-section}

**[`Linear Transformers Are Secretly Fast Weight Programmers`{=html}](https://arxiv.org/abs/2102.11174#schmidhuber "'Linear Transformers Are Secretly Fast Weight Programmers', Schlag et al 2021"){.link-annotated
.id-not .include-annotation link-icon="SMDH"
link-icon-type="text,quad,sans"}**

## `“Unlocking Pixels for Reinforcement Learning via Implicit Attention”, Choromanski et al 2021`{=html} {#choromanski-et-al-2021-section}

**[`Unlocking Pixels for Reinforcement Learning via Implicit Attention`{=html}](https://arxiv.org/abs/2102.04353 "'Unlocking Pixels for Reinforcement Learning via Implicit Attention', Choromanski et al 2021"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Transformer Feed-Forward Layers Are Key-Value Memories”, Pipek et al 2020`{=html} {#pipek-et-al-2020-section}

**[`Transformer Feed-Forward Layers Are Key-Value Memories`{=html}](https://arxiv.org/abs/2012.14913 "'Transformer Feed-Forward Layers Are Key-Value Memories', Pipek et al 2020"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“AdnFM: An Attentive DenseNet Based Factorization Machine for CTR Prediction”, Wang et al 2020`{=html} {#wang-et-al-2020-05-section}

**[`AdnFM: An Attentive DenseNet based Factorization Machine for CTR Prediction`{=html}](https://arxiv.org/abs/2012.10820#tencent "'AdnFM: An Attentive DenseNet based Factorization Machine for CTR Prediction', Wang et al 2020"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Inductive Biases for Deep Learning of Higher-Level Cognition”, Goyal & Bengio 2020`{=html} {#goyal-bengio-2020-section}

**[`Inductive Biases for Deep Learning of Higher-Level Cognition`{=html}](https://arxiv.org/abs/2011.15091 "'Inductive Biases for Deep Learning of Higher-Level Cognition', Goyal & Bengio 2020"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Long Range Arena (LRA): A Benchmark for Efficient Transformers”, Tay et al 2020`{=html} {#tay-et-al-2020-1-section}

**[`Long Range Arena (LRA): A Benchmark for Efficient Transformers`{=html}](https://openreview.net/forum?id=qVyeW-grC2k#google "'Long Range Arena (LRA): A Benchmark for Efficient Transformers', Tay et al 2020"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Current Limitations of Language Models: What You Need Is Retrieval”, Komatsuzaki 2020`{=html} {#komatsuzaki-2020-section}

**[`Current Limitations of Language Models: What You Need is Retrieval`{=html}](https://arxiv.org/abs/2009.06857 "'Current Limitations of Language Models: What You Need is Retrieval', Komatsuzaki 2020"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Efficient Transformers: A Survey”, Tay et al 2020`{=html} {#tay-et-al-2020-efficienttransformers-section}

**[`Efficient Transformers: A Survey`{=html}](https://arxiv.org/abs/2009.06732#google "'Efficient Transformers: A Survey', Tay et al 2020"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“HiPPO: Recurrent Memory With Optimal Polynomial Projections”, Gu et al 2020`{=html} {#gu-et-al-2021-hippo-section}

**[`HiPPO: Recurrent Memory with Optimal Polynomial Projections`{=html}](https://arxiv.org/abs/2008.07669 "'HiPPO: Recurrent Memory with Optimal Polynomial Projections', Gu et al 2020"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Efficient Attention: Breaking The Quadratic Transformer Bottleneck”, Gwern 2020`{=html} {#gwern-doc-ai-nn-transformer-attention-abstract-section}

**[`Efficient Attention: Breaking The Quadratic Transformer Bottleneck`{=html}](/doc/ai/nn/transformer/attention/abstract "'Efficient Attention: Breaking The Quadratic Transformer Bottleneck', Gwern 2020"){.link-annotated
.id-not .include-annotation}**

## `“Pre-Training via Paraphrasing”, Lewis et al 2020`{=html} {#lewis-et-al-2020-section}

**[`Pre-training via Paraphrasing`{=html}](https://arxiv.org/abs/2006.15020#facebook "'Pre-training via Paraphrasing', Lewis et al 2020"){.link-annotated
.id-not .include-annotation link-icon="facebook" link-icon-type="svg"
link-icon-color="#1877f2"}**

## `“Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers”, Choromanski et al 2020`{=html} {#choromanski-et-al-2020-section}

**[`Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers`{=html}](https://arxiv.org/abs/2006.03555#google "'Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers', Choromanski et al 2020"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“GPT-3: Language Models Are Few-Shot Learners”, Brown et al 2020`{=html} {#brown-et-al-2020-2-section}

**[`GPT-3: Language Models are Few-Shot Learners`{=html}](https://arxiv.org/abs/2005.14165#openai "'GPT-3: Language Models are Few-Shot Learners', Brown et al 2020"){.link-annotated
.id-not .include-annotation link-icon="openai" link-icon-type="svg"}**

## `“Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, Lewis et al 2020`{=html} {#lewis-et-al-2020-rag-section}

**[`Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks`{=html}](https://arxiv.org/abs/2005.11401#facebook "'Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks', Lewis et al 2020"){.link-annotated
.id-not .include-annotation link-icon="facebook" link-icon-type="svg"
link-icon-color="#1877f2"}**

## `“Synthesizer: Rethinking Self-Attention in Transformer Models”, Tay et al 2020`{=html} {#tay-et-al-2020-2-section}

**[`Synthesizer: Rethinking Self-Attention in Transformer Models`{=html}](https://arxiv.org/abs/2005.00743#google "'Synthesizer: Rethinking Self-Attention in Transformer Models', Tay et al 2020"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“PowerNorm: Rethinking Batch Normalization in Transformers”, Shen et al 2020`{=html} {#shen-et-al-2020-1-section}

**[`PowerNorm: Rethinking Batch Normalization in Transformers`{=html}](https://arxiv.org/abs/2003.07845 "'PowerNorm: Rethinking Batch Normalization in Transformers', Shen et al 2020"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“On Layer Normalization in the Transformer Architecture”, Xiong et al 2020`{=html} {#xiong-et-al-2020-section}

**[`On Layer Normalization in the Transformer Architecture`{=html}](https://arxiv.org/abs/2002.04745#microsoft "'On Layer Normalization in the Transformer Architecture', Xiong et al 2020"){.link-annotated
.id-not .include-annotation link-icon="MS"
link-icon-type="text,sans,italic"}**

## `“REALM: Retrieval-Augmented Language Model Pre-Training”, Guu et al 2020`{=html} {#guu-et-al-2020-section}

**[`REALM: Retrieval-Augmented Language Model Pre-Training`{=html}](https://arxiv.org/abs/2002.08909#google "'REALM: Retrieval-Augmented Language Model Pre-Training', Guu et al 2020"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“BERT’s Output Layer Recognizes All Hidden Layers? Some Intriguing Phenomena and a Simple Way to Boost BERT”, Kao et al 2020`{=html} {#kao-et-al-2020-section}

**[`BERT’s output layer recognizes all hidden layers? Some Intriguing Phenomena and a simple way to boost BERT`{=html}](https://arxiv.org/abs/2001.09309 "'BERT’s output layer recognizes all hidden layers? Some Intriguing Phenomena and a simple way to boost BERT', Kao et al 2020"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Rethinking Attention With Performers”, Choromanski & Colwell 2020`{=html} {#choromanski-colwell-2020-section}

**[`Rethinking Attention with Performers`{=html}](https://research.google/blog/rethinking-attention-with-performers/ "'Rethinking Attention with Performers', Choromanski & Colwell 2020"){.id-not
.include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Are Transformers Universal Approximators of Sequence-To-Sequence Functions?”, Yun et al 2019`{=html} {#yun-et-al-2019-section}

**[`Are Transformers universal approximators of sequence-to-sequence functions?`{=html}](https://arxiv.org/abs/1912.10077 "'Are Transformers universal approximators of sequence-to-sequence functions?', Yun et al 2019"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Dynamic Convolution: Attention over Convolution Kernels”, Chen et al 2019`{=html} {#chen-et-al-2019-section}

**[`Dynamic Convolution: Attention over Convolution Kernels`{=html}](https://arxiv.org/abs/1912.03458#microsoft "'Dynamic Convolution: Attention over Convolution Kernels', Chen et al 2019"){.link-annotated
.id-not .include-annotation link-icon="MS"
link-icon-type="text,sans,italic"}**

## `“Generalization through Memorization: Nearest Neighbor Language Models”, Khandelwal et al 2019`{=html} {#khandelwal-et-al-2019-section}

**[`Generalization through Memorization: Nearest Neighbor Language Models`{=html}](https://arxiv.org/abs/1911.00172 "'Generalization through Memorization: Nearest Neighbor Language Models', Khandelwal et al 2019"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Multiplicative Interactions and Where to Find Them”, Jayakumar et al 2019`{=html} {#jayakumar-et-al-2019-section}

**[`Multiplicative Interactions and Where to Find Them`{=html}](https://openreview.net/forum?id=rylnK6VtDH#google "'Multiplicative Interactions and Where to Find Them', Jayakumar et al 2019"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“The Bottom-Up Evolution of Representations in the Transformer: A Study With Machine Translation and Language Modeling Objectives”, Voita et al 2019`{=html} {#voita-et-al-2019-1-section}

**[`The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives`{=html}](https://arxiv.org/abs/1909.01380 "'The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives', Voita et al 2019"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Logic and the 2-Simplicial Transformer”, Clift et al 2019`{=html} {#clift-et-al-2019-section}

**[`Logic and the 2-Simplicial Transformer`{=html}](https://arxiv.org/abs/1909.00668 "'Logic and the 2-Simplicial Transformer', Clift et al 2019"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“PKM: Large Memory Layers With Product Keys”, Lample et al 2019`{=html} {#lample-et-al-2019-section}

**[`PKM: Large Memory Layers with Product Keys`{=html}](https://arxiv.org/abs/1907.05242#facebook "'PKM: Large Memory Layers with Product Keys', Lample et al 2019"){.link-annotated
.id-not .include-annotation link-icon="facebook" link-icon-type="svg"
link-icon-color="#1877f2"}**

## `“What Does BERT Look At? An Analysis of BERT’s Attention”, Clark et al 2019`{=html} {#clark-et-al-2019-2-section}

**[`What Does BERT Look At? An Analysis of BERT’s Attention`{=html}](https://arxiv.org/abs/1906.04341 "'What Does BERT Look At? An Analysis of BERT’s Attention', Clark et al 2019"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Are 16 Heads Really Better Than One?”, Michel et al 2019`{=html} {#michel-et-al-2019-section}

**[`Are 16 Heads Really Better than One?`{=html}](https://arxiv.org/abs/1905.10650 "'Are 16 Heads Really Better than One?', Michel et al 2019"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Paper Review: Set Transformers. A Framework for Attention-Based…”`{=html} {#_Kzwhs2cV-section}

**[`Paper Review: Set Transformers. A Framework for Attention-based…`{=html}](https://medium.com/correll-lab/paper-review-set-transformers-6ce59c2ad88f "Paper Review: Set Transformers. A Framework for Attention-based…"){.id-not
.include-annotation link-icon="𝐌" link-icon-type="text"}**

## `“Pay Less Attention With Lightweight and Dynamic Convolutions”, Wu et al 2019`{=html} {#wu-et-al-2019-1-section}

**[`Pay Less Attention with Lightweight and Dynamic Convolutions`{=html}](https://arxiv.org/abs/1901.10430#facebook "'Pay Less Attention with Lightweight and Dynamic Convolutions', Wu et al 2019"){.link-annotated
.id-not .include-annotation link-icon="facebook" link-icon-type="svg"
link-icon-color="#1877f2"}**

## `“On the Turing Completeness of Modern Neural Network Architectures”, Pérez et al 2019`{=html} {#pérez-et-al-2019-section}

**[`On the Turing Completeness of Modern Neural Network Architectures`{=html}](https://arxiv.org/abs/1901.03429 "'On the Turing Completeness of Modern Neural Network Architectures', Pérez et al 2019"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Music Transformer”, Huang et al 2018`{=html} {#huang-et-al-2018-4-section}

**[`Music Transformer`{=html}](https://arxiv.org/abs/1809.04281#google "'Music Transformer', Huang et al 2018"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Character-Level Language Modeling With Deeper Self-Attention”, Al-Rfou et al 2018`{=html} {#al-rfou-et-al-2018-section}

**[`Character-Level Language Modeling with Deeper Self-Attention`{=html}](https://arxiv.org/abs/1808.04444#google "'Character-Level Language Modeling with Deeper Self-Attention', Al-Rfou et al 2018"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“Attention Is All You Need”, Vaswani et al 2017`{=html} {#vaswani-et-al-2017-section}

**[`Attention Is All You Need`{=html}](https://arxiv.org/abs/1706.03762#google "'Attention Is All You Need', Vaswani et al 2017"){.link-annotated
.id-not .include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“A Deep Reinforced Model for Abstractive Summarization”, Paulus et al 2017`{=html} {#paulus-et-al-2017-section}

**[`A Deep Reinforced Model for Abstractive Summarization`{=html}](https://arxiv.org/abs/1705.04304#salesforce "'A Deep Reinforced Model for Abstractive Summarization', Paulus et al 2017"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Get To The Point: Summarization With Pointer-Generator Networks”, See et al 2017`{=html} {#see-et-al-2017-section}

**[`Get To The Point: Summarization with Pointer-Generator Networks`{=html}](https://arxiv.org/abs/1704.04368 "'Get To The Point: Summarization with Pointer-Generator Networks', See et al 2017"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“RAM: Dynamic Computational Time for Visual Attention”, Li et al 2017`{=html} {#li-et-al-2017-5-section}

**[`RAM: Dynamic Computational Time for Visual Attention`{=html}](https://arxiv.org/abs/1703.10332 "'RAM: Dynamic Computational Time for Visual Attention', Li et al 2017"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Hybrid Computing Using a Neural Network With Dynamic External Memory”, Graves et al 2016`{=html} {#graves-et-al-2016-section}

**[`Hybrid computing using a neural network with dynamic external memory`{=html}](/doc/reinforcement-learning/model-free/2016-graves.pdf#deepmind "'Hybrid computing using a neural network with dynamic external memory', Graves et al 2016"){.link-annotated
.id-not .include-annotation link-icon="deepmind" link-icon-type="svg"
link-icon-color="#4185f4"}**

## `“Scaling Memory-Augmented Neural Networks With Sparse Reads and Writes”, Rae et al 2016`{=html} {#rae-et-al-2016-section}

**[`Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes`{=html}](https://arxiv.org/abs/1610.09027#deepmind "'Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes', Rae et al 2016"){.link-annotated
.id-not .include-annotation link-icon="deepmind" link-icon-type="svg"
link-icon-color="#4185f4"}**

## `“Modeling Human Reading With Neural Attention”, Hahn & Keller 2016`{=html} {#hahn-keller-2016-section}

**[`Modeling Human Reading with Neural Attention`{=html}](https://arxiv.org/abs/1608.05604 "'Modeling Human Reading with Neural Attention', Hahn & Keller 2016"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Iterative Alternating Neural Attention for Machine Reading”, Sordoni et al 2016`{=html} {#sordoni-et-al-2016-section}

**[`Iterative Alternating Neural Attention for Machine Reading`{=html}](https://arxiv.org/abs/1606.02245 "'Iterative Alternating Neural Attention for Machine Reading', Sordoni et al 2016"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Adaptive Computation Time for Recurrent Neural Networks”, Graves 2016`{=html} {#graves-2016-section}

**[`Adaptive Computation Time for Recurrent Neural Networks`{=html}](https://arxiv.org/abs/1603.08983#deepmind "'Adaptive Computation Time for Recurrent Neural Networks', Graves 2016"){.link-annotated
.id-not .include-annotation link-icon="deepmind" link-icon-type="svg"
link-icon-color="#4185f4"}**

## `“Foveation-Based Mechanisms Alleviate Adversarial Examples”, Luo et al 2015`{=html} {#luo-et-al-2015-section}

**[`Foveation-based Mechanisms Alleviate Adversarial Examples`{=html}](https://arxiv.org/abs/1511.06292 "'Foveation-based Mechanisms Alleviate Adversarial Examples', Luo et al 2015"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Generating Images from Captions With Attention”, Mansimov et al 2015`{=html} {#mansimov-et-al-2015-section}

**[`Generating Images from Captions with Attention`{=html}](https://arxiv.org/abs/1511.02793 "'Generating Images from Captions with Attention', Mansimov et al 2015"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“DRAW: A Recurrent Neural Network For Image Generation”, Gregor et al 2015`{=html} {#gregor-et-al-2015-section}

**[`DRAW: A Recurrent Neural Network For Image Generation`{=html}](https://arxiv.org/abs/1502.04623 "'DRAW: A Recurrent Neural Network For Image Generation', Gregor et al 2015"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Neural Turing Machines”, Graves et al 2014`{=html} {#graves-et-al-2014-section}

**[`Neural Turing Machines`{=html}](https://arxiv.org/abs/1410.5401#deepmind "'Neural Turing Machines', Graves et al 2014"){.link-annotated
.id-not .include-annotation link-icon="deepmind" link-icon-type="svg"
link-icon-color="#4185f4"}**

## `“Neural Machine Translation by Jointly Learning to Align and Translate”, Bahdanau et al 2014`{=html} {#bahdanau-et-al-2014-section}

**[`Neural Machine Translation by Jointly Learning to Align and Translate`{=html}](https://arxiv.org/abs/1409.0473 "'Neural Machine Translation by Jointly Learning to Align and Translate', Bahdanau et al 2014"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“On Learning Where To Look”, Ranzato 2014`{=html} {#ranzato-2014-section}

**[`On Learning Where To Look`{=html}](https://arxiv.org/abs/1405.5488 "'On Learning Where To Look', Ranzato 2014"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Generating Sequences With Recurrent Neural Networks”, Graves 2013`{=html} {#graves-2013-section}

**[`Generating Sequences With Recurrent Neural Networks`{=html}](https://arxiv.org/abs/1308.0850 "'Generating Sequences With Recurrent Neural Networks', Graves 2013"){.link-annotated
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Building a Minimal Transformer for 10-Digit Addition”, Litzenberger 2026`{=html} {#_PYtd7NB4-section}

**[`Building a minimal Transformer for 10-digit addition`{=html}](https://alexlitzenberger.com/blog/building_a_minimal_transformer_for_10_digit_addition "'Building a minimal Transformer for 10-digit addition', Litzenberger 2026"){.link-annotated-partial
.id-not .include-annotation}**

## `“Efficient Transformers: A Survey § Table 1”`{=html} {#_zB3dr3RB-section}

**[`Efficient Transformers: A Survey § Table 1`{=html}](https://arxiv.org/pdf/2009.06732#page=6&org=google "Efficient Transformers: A Survey § Table 1"){.id-not
.include-annotation link-icon="alphabet" link-icon-type="svg"
link-icon-color="#4285f4"}**

## `“1M Context Is Now Generally Available for Opus 4.6 and Sonnet 4.6”`{=html} {#_n4KMzftH-section}

**[`1M context is now generally available for Opus 4.6 and Sonnet 4.6`{=html}](https://claude.com/blog/1m-context-ga "1M context is now generally available for Opus 4.6 and Sonnet 4.6"){.link-annotated-partial
.id-not .include-annotation}**

## `“Attention and Augmented Recurrent Neural Networks”`{=html} {#_PxtbI-aq-section}

**[`Attention and Augmented Recurrent Neural Networks`{=html}](https://distill.pub/2016/augmented-rnns/ "Attention and Augmented Recurrent Neural Networks"){.link-annotated-partial
.id-not .include-annotation link-icon="distillpub"
link-icon-type="svg"}**

## `“AdderBoard: Contest for Smallest Transformer That Can Add Two 10-Digit Numbers”`{=html} {#_4olwZWTw-section}

**[`AdderBoard: contest for smallest Transformer that can add two 10-digit numbers`{=html}](https://github.com/anadim/AdderBoard "AdderBoard: contest for smallest Transformer that can add two 10-digit numbers"){.link-annotated-partial
.id-not .include-annotation link-icon="github" link-icon-type="svg"}**

## `“Digit-Addition-311p: a Grokked 311-Parameter Transformer for 10-Digit Addition”`{=html} {#_yrSJQYVL-section}

**[`digit-addition-311p: a grokked 311-parameter Transformer for 10-digit addition`{=html}](https://github.com/rezabyt/digit-addition-311p "digit-addition-311p: a grokked 311-parameter Transformer for 10-digit addition"){.link-annotated-partial
.id-not .include-annotation link-icon="github" link-icon-type="svg"}**

## `“Technion-Cs-Nlp/llm-Arithmetic-Heuristics”`{=html} {#_WrubH-Yy-section}

**[`technion-cs-nlp/llm-arithmetic-heuristics`{=html}](https://github.com/technion-cs-nlp/llm-arithmetic-heuristics "technion-cs-nlp/llm-arithmetic-heuristics"){.link-annotated-partial
.id-not .include-annotation link-icon="github" link-icon-type="svg"}**

## `“Hierarchical Object Detection With Deep Reinforcement Learning”`{=html} {#_YgvwcCt7-section}

**[`Hierarchical Object Detection with Deep Reinforcement Learning`{=html}](https://imatge-upc.github.io/detection-2016-nipsws/ "Hierarchical Object Detection with Deep Reinforcement Learning"){.link-annotated-partial
.id-not .include-annotation}**

## `“The Transformer Family: Attention and Self-Attention • Multi-Head Self-Attention • Transformer • Adaptive Computation Time (ACT) • Improved Attention Span: (Longer Attention Span (Transformer-XL) / Adaptive Attention Span / Localized Attention Span (Image Transformer)) • Less Time and Memory Cost: (Sparse Attention Matrix Factorization (Sparse Transformers) / Locality-Sensitive Hashing (Reformer)) • Make It Recurrent (Universal Transformer) • Stabilization for RL (GTrXL)”`{=html} {#_WHCrsBro-section}

**[`The Transformer Family: Attention and Self-Attention • Multi-Head Self-Attention • Transformer • Adaptive Computation Time (ACT) • Improved Attention Span: (Longer Attention Span (Transformer-XL) / Adaptive Attention Span / Localized Attention Span (Image Transformer)) • Less Time and Memory Cost: (Sparse Attention Matrix Factorization (Sparse Transformers) / Locality-Sensitive Hashing (Reformer)) • Make it Recurrent (Universal Transformer) • Stabilization for RL (GTrXL)`{=html}](https://lilianweng.github.io/lil-log/2020/04/07/the-transformer-family.html#openai "The Transformer Family: Attention and Self-Attention • Multi-Head Self-Attention • Transformer • Adaptive Computation Time (ACT) • Improved Attention Span: (Longer Attention Span (Transformer-XL) / Adaptive Attention Span / Localized Attention Span (Image Transformer)) • Less Time and Memory Cost: (Sparse Attention Matrix Factorization (Sparse Transformers) / Locality-Sensitive Hashing (Reformer)) • Make it Recurrent (Universal Transformer) • Stabilization for RL (GTrXL)"){.id-not
.include-annotation link-icon="openai" link-icon-type="svg"}**

::: aux-links-transclude-file
::: collapse
**View HTML**:

[`/doc/www/lilianweng.github.io/8428b81f97ec6ee2f1a32d095f038a405e743f80.html#openai`](/doc/www/lilianweng.github.io/8428b81f97ec6ee2f1a32d095f038a405e743f80.html#openai){.id-not
.link-annotated-not .include-content .include-lazy link-icon="openai"
link-icon-type="svg"}
:::
:::

## `“100M Token Context Windows”`{=html} {#_J5RWhIQT-section}

**[`100M Token Context Windows`{=html}](https://magic.dev/blog/100m-token-context-windows "100M Token Context Windows"){.link-annotated-partial
.id-not .include-annotation}**

## `“Learning to Combine Foveal Glimpses With a Third-Order Boltzmann Machine”`{=html} {#_Pv5-FySP-section}

**[`Learning to combine foveal glimpses with a third-order Boltzmann machine`{=html}](https://pdfs.semanticscholar.org/6da2/445037118c8ab72be1a319dd9f2a2116b305.pdf "Learning to combine foveal glimpses with a third-order Boltzmann machine"){.link-annotated-partial
.id-not .include-annotation link-icon="pdf" link-icon-type="svg"
link-icon-color="#f40f02"}**

## `“Show, Attend and Tell: Neural Image Caption Generation With Visual Attention”`{=html} {#_4lMGuF4C-section}

**[`Show, attend and tell: Neural image caption generation with visual attention`{=html}](https://proceedings.mlr.press/v37/xuc15.pdf "Show, attend and tell: Neural image caption generation with visual attention"){.link-annotated-partial
.id-not .include-annotation link-icon="𝛘" link-icon-type="text"
link-icon-color="#b31b1b"}**

## `“Recurrent Models of Visual Attention”`{=html} {#_QJWNuYF0-section}

**[`Recurrent models of visual attention`{=html}](https://proceedings.neurips.cc/paper/2014/hash/09c6c3783b4a70054da74f2538ed47c6-Abstract.html "Recurrent models of visual attention"){.link-annotated-partial
.id-not .include-annotation link-icon="NIPS"
link-icon-type="text,sans,quad" link-icon-color="#0d6efd"}**

## `“Can Active Memory Replace Attention?”`{=html} {#_h9Irb9mg-section}

**[`Can Active Memory Replace Attention?`{=html}](https://proceedings.neurips.cc/paper/2016/hash/fb8feff253bb6c834deb61ec76baa893-Abstract.html "Can Active Memory Replace Attention?"){.link-annotated-partial
.id-not .include-annotation link-icon="NIPS"
link-icon-type="text,sans,quad" link-icon-color="#0d6efd"}**

## `“Dzmitry Bahdanau”`{=html} {#_2uJhZFOy-section}

**[`Dzmitry Bahdanau`{=html}](https://rizar.github.io/ "Dzmitry Bahdanau"){.link-annotated-partial
.id-not .include-annotation}**

## `“Scaling Automatic Neuron Description”`{=html} {#_aVYFLmbm-section}

**[`Scaling Automatic Neuron Description`{=html}](https://transluce.org/neuron-descriptions "Scaling Automatic Neuron Description"){.link-annotated-partial
.id-not .include-annotation}**

## `“Monitor: An AI-Driven Observability Interface”`{=html} {#_KDoYv_J0-section}

**[`Monitor: An AI-Driven Observability Interface`{=html}](https://transluce.org/observability-interface "Monitor: An AI-Driven Observability Interface"){.link-annotated-partial
.id-not .include-annotation}**

## `“Interpreting GPT: the Logit Lens”`{=html} {#_qpzo4H82-section}

**[`interpreting GPT: the logit lens`{=html}](https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens "interpreting GPT: the logit lens"){.id-not
.include-annotation link-icon="LW" link-icon-type="text"
link-icon-color="#7faf83"}**

::: aux-links-transclude-file
::: collapse
**View External Link**:

[`https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens`](https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens){.id-not
.link-annotated-not .include-content .include-lazy link-icon="LW"
link-icon-type="text" link-icon-color="#7faf83"}
:::
:::

## `“A Sober Look at Steering Vectors for LLMs”`{=html} {#_t9y7Q2lF-section}

**[`A Sober Look at Steering Vectors for LLMs`{=html}](https://www.lesswrong.com/posts/QQP4nq7TXg89CJGBh/a-sober-look-at-steering-vectors-for-llms "A Sober Look at Steering Vectors for LLMs"){.link-annotated-partial
.id-not .include-annotation link-icon="LW" link-icon-type="text"
link-icon-color="#7faf83"}**

## `“Antonym Heads Predict Semantic Opposites in Language Models”`{=html} {#_6t2R_aIq-section}

**[`Antonym Heads Predict Semantic Opposites in Language Models`{=html}](https://www.lesswrong.com/posts/XXK2T4EcbHRkRTBce/antonym-heads-predict-semantic-opposites-in-language-models "Antonym Heads Predict Semantic Opposites in Language Models"){.link-annotated-partial
.id-not .include-annotation link-icon="LW" link-icon-type="text"
link-icon-color="#7faf83"}**

## `“One-Shot Steering Vectors Cause Emergent Misalignment, Too”`{=html} {#_MixkRpej-section}

**[`One-shot steering vectors cause emergent misalignment, too`{=html}](https://www.lesswrong.com/posts/kcKnKHTHycHeRhcHF/one-shot-steering-vectors-cause-emergent-misalignment-too "One-shot steering vectors cause emergent misalignment, too"){.link-annotated-partial
.id-not .include-annotation link-icon="LW" link-icon-type="text"
link-icon-color="#7faf83"}**

## `“A Survey of Long-Term Context in Transformers: Sparse Transformers • Adaptive Span Transformers • Transformer-XL • Compressive Transformers • Reformer • Routing Transformer • Sinkhorn Transformer • Linformer • Efficient Attention: Attention With Linear Complexities • Transformers Are RNNs • ETC • Longformer”`{=html} {#_cNr_PjHH-section}

**[`A Survey of Long-Term Context in Transformers: Sparse Transformers • Adaptive Span Transformers • Transformer-XL • Compressive Transformers • Reformer • Routing Transformer • Sinkhorn Transformer • Linformer • Efficient Attention: Attention with Linear Complexities • Transformers are RNNs • ETC • Longformer`{=html}](https://www.pragmatic.ml/a-survey-of-methods-for-incorporating-long-term-context/ "A Survey of Long-Term Context in Transformers: Sparse Transformers • Adaptive Span Transformers • Transformer-XL • Compressive Transformers • Reformer • Routing Transformer • Sinkhorn Transformer • Linformer • Efficient Attention: Attention with Linear Complexities • Transformers are RNNs • ETC • Longformer"){.link-annotated-partial
.id-not .include-annotation link-icon="𝕄" link-icon-type="text"}**

## `“FlashAttention-3: Fast and Accurate Attention With Asynchrony and Low-Precision”`{=html} {#_rAsKiX1f-section}

**[`FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision`{=html}](https://www.together.ai/blog/flashattention-3 "FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision"){.link-annotated-partial
.id-not .include-annotation}**

## Sort By Magic

::: {demo-type="sort-by-magic-preface"}
Annotations sorted by machine learning into [inferred \'tags\'](/design#future-tag-features){#gwern-design--future-tag-features
.link-annotated-partial}. This provides an alternative way to browse: instead of by *date* order, one can browse in *topic* order. The \'sorted\' list has been automatically clustered into multiple sections & auto-labeled for easier browsing.

Beginning with the newest annotation, it uses the embedding of each annotation to attempt to create a list of nearest-neighbor annotations, creating a progression of topics. For more details, see the link.
:::

### `scaling-optimization` {.collapse title="Machine-generated tag name for the following cluster of links."}

[\[see previous entry\]](#minimax-et-al-2025-section){.include
.include-even-when-collapsed}

### `optimal-projections` {.collapse title="Machine-generated tag name for the following cluster of links."}

[\[see previous entry\]](#gwern-doc-ai-nn-transformer-attention-abstract-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#gu-et-al-2021-hippo-section){.include
.include-even-when-collapsed}

### `foveation-adversarial` {.collapse title="Machine-generated tag name for the following cluster of links."}

[\[see previous entry\]](#luo-et-al-2015-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#li-et-al-2017-5-section){.include
.include-even-when-collapsed}

### `in-context-learning` {.collapse title="Machine-generated tag name for the following cluster of links."}

[\[see previous entry\]](#wu-et-al-2023-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#wang-et-al-2020-05-section){.include
.include-even-when-collapsed}

### `sentiment-representation language-activation invertibility linear-models invertible-sentiment representation` {.collapse title="Machine-generated tag name for the following cluster of links."}

[\[see previous entry\]](#nikolaou-et-al-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#tigges-et-al-2023-section){.include
.include-even-when-collapsed}

### `prompt-engineering` {.collapse title="Machine-generated tag name for the following cluster of links."}

[\[see previous entry\]](#turner-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#bhargava-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#qiu-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#nikankin-et-al-2024-section){.include
.include-even-when-collapsed}

### `belief-state masked-mixers language-gen retrieval` {.collapse title="Machine-generated tag name for the following cluster of links."}

[\[see previous entry\]](#hu-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#badger-2024-section){.include
.include-even-when-collapsed}

### `memory-optimization` {.collapse title="Machine-generated tag name for the following cluster of links."}

[\[see previous entry\]](#lieberum-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#gurnee-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#pipek-et-al-2020-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#pires-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#narayan-et-al-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#wang-et-al-2026-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#chen-et-al-2024-6-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#lample-et-al-2019-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#tay-et-al-2022-2-section){.include
.include-even-when-collapsed}

### `tabular-interactions` {.collapse title="Machine-generated tag name for the following cluster of links."}

[\[see previous entry\]](#rabbani-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#somepalli-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#goh-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#burns-fukai-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#benoist-et-al-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#clark-abbott-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#whittington-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#goyal-bengio-2020-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#jayakumar-et-al-2019-section){.include
.include-even-when-collapsed}

### `automatic-summarization` {.collapse title="Machine-generated tag name for the following cluster of links."}

[\[see previous entry\]](#jabri-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#panigrahi-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#gould-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#hanna-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#bai-et-al-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#fu-et-al-2026-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#yehudai-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#quirke-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#roy-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#al-rfou-et-al-2018-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#choromanski-et-al-2020-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#graves-2016-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#rae-et-al-2016-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#graves-et-al-2016-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#graves-et-al-2014-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#bahdanau-et-al-2014-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#paulus-et-al-2017-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#see-et-al-2017-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#lewis-et-al-2020-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#lewis-et-al-2020-rag-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#guu-et-al-2020-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#wei-et-al-2024-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#mayne-et-al-2026-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#anil-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#brown-et-al-2020-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#hardt-sun-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#khandelwal-et-al-2019-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#wu-et-al-2024-3-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#chang-et-al-2022-3-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#ainslie-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#fu-et-al-2024-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#rao-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#rios-et-al-2026-section){.include
.include-even-when-collapsed}

### `context-learning` {.collapse title="Machine-generated tag name for the following cluster of links."}

[\[see previous entry\]](#graves-2013-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#gregor-et-al-2015-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#mansimov-et-al-2015-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#ranzato-2014-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#choromanski-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#venkataramanan-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#pan-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#melas-kyriazi-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#bozic-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#dong-et-al-2021-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#cui-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#park-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#liu-et-al-2023-03-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#oswald-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#ahn-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#nichani-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#voita-et-al-2019-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#mcgrath-et-al-2023-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#wang-et-al-2024-07-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#ji-an-et-al-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#yu-et-al-2023-3-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#wu-et-al-2024-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#komatsuzaki-2020-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#variengien-winsor-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#lee-et-al-2024-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#leng-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#hsieh-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#tay-et-al-2020-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#tay-et-al-2020-efficienttransformers-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#zhou-et-al-2023-08-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#he-hofmann-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#he-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#michel-et-al-2019-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#ji-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#paliotta-et-al-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#wang-et-al-2024-02-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#waleffe-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#dong-et-al-2024-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#fu-et-al-2022-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#wang-et-al-2022-05-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#amos-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#kumar-isola-2026-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#feng-et-al-2024-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#feng-et-al-2024-3-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#zucchet-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#irie-et-al-2022-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#schlag-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#irie-et-al-2022-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#katsch-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#wang-et-al-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#behrouz-et-al-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#schug-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#bricken-pehlevan-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#hahn-keller-2016-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#sordoni-et-al-2016-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#clark-et-al-2019-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#kao-et-al-2020-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#charpentier-samuel-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#petty-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#bae-et-al-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#csordás-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#dao-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#dao-et-al-2022-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#liu-abbeel-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#wang-et-al-2023-08-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#pope-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#gonzález-et-al-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#zhu-et-al-2024-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#garcia-et-al-2026-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#shinnick-et-al-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#ruis-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#ward-et-al-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#lee-et-al-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#deng-et-al-2023-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#pfau-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#wu-et-al-2024-4-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#li-et-al-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#bogdan-lindsey-2026-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#hernandez-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#razzhigaev-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#yun-et-al-2019-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#pérez-et-al-2019-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#liu-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#zhang-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#lindner-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#shaw-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#weiss-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#angluin-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#chiang-cholak-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#chiang-et-al-2023-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#merrill-sabharwal-2022-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#bergsträßer-et-al-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#merrill-sabharwal-2022-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#domingos-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#clift-et-al-2019-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#roy-et-al-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#poli-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#suresh-p-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#brown-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#parisotto-salakhutdinov-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#team-et-al-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#wu-et-al-2019-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#chen-et-al-2019-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#zhou-et-al-2023-05-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#wortsman-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#raghu-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#wang-et-al-2021-06-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#li-et-al-2024-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#darcet-et-al-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#trockman-kolter-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#levy-2024-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#vaswani-et-al-2017-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#tay-et-al-2020-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#lee-thorp-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#luo-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#qin-et-al-2023-3-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#su-et-al-2021-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#du-et-al-2026-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#wen-et-al-2024-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#arora-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#charakorn-et-al-2026-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#liu-et-al-2023-12-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#laban-et-al-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#haviv-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#press-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#lohrenz-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#ye-et-al-2024-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#kayyam-et-al-2026-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#narang-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#xiong-et-al-2020-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#shen-et-al-2020-1-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#zhu-et-al-2025-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#loshchilov-et-al-2024-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#yu-et-al-2023-5-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#nemecek-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#gruver-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#goel-bartlett-2023-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#grigsby-et-al-2021-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#ge-et-al-2022-2-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#hawthorne-et-al-2022-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#huang-et-al-2018-4-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#dong-et-al-2022-5-section){.include
.include-even-when-collapsed}

[\[see previous entry\]](#goel-et-al-2022-2-section){.include
.include-even-when-collapsed}

# Miscellaneous

-   **[`<code>/doc/ai/nn/transformer/attention/2023-09-08-charlesfoster-aunn-variantwithcausaldecoderattention.jpg</code>`{=html}](/doc/ai/nn/transformer/attention/2023-09-08-charlesfoster-aunn-variantwithcausaldecoderattention.jpg){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/transformer/attention/2023-trockman-figure2-attentionmappatternsbyinitializationandleveloftrainingshowpriors.png</code>`{=html}](/doc/ai/nn/transformer/attention/2023-trockman-figure2-attentionmappatternsbyinitializationandleveloftrainingshowpriors.png){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/transformer/attention/2023-trockman-figure7-gpt2attentionmatrixpatterns.png</code>`{=html}](/doc/ai/nn/transformer/attention/2023-trockman-figure7-gpt2attentionmatrixpatterns.png){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/transformer/attention/2022-hassid-figure3-largertransformersmakemoreuseofattentionwhennablatedtomlbenchmarkperformance.jpg</code>`{=html}](/doc/ai/nn/transformer/attention/2022-hassid-figure3-largertransformersmakemoreuseofattentionwhennablatedtomlbenchmarkperformance.jpg){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/transformer/attention/2022-tay-figure4-scalingofmodelbydepth.jpg</code>`{=html}](/doc/ai/nn/transformer/attention/2022-tay-figure4-scalingofmodelbydepth.jpg){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/transformer/attention/2022-tay-figure5-scalingofmodelbymlpfeedforwardparameters.jpg</code>`{=html}](/doc/ai/nn/transformer/attention/2022-tay-figure5-scalingofmodelbymlpfeedforwardparameters.jpg){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/transformer/attention/2020-08-11-gwern-meme-twoastronauts-hopfieldnetworksareallyouneed.jpg</code>`{=html}](/doc/ai/nn/transformer/attention/2020-08-11-gwern-meme-twoastronauts-hopfieldnetworksareallyouneed.jpg "Gwern 2020"){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/transformer/attention/2020-longrangearena-figure3-performancefrontier.jpg</code>`{=html}](/doc/ai/nn/transformer/attention/2020-longrangearena-figure3-performancefrontier.jpg){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/transformer/attention/2020-tay-figure2-efficientattentiontaxonomy.png</code>`{=html}](/doc/ai/nn/transformer/attention/2020-tay-figure2-efficientattentiontaxonomy.png){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>/doc/ai/nn/transformer/attention/2020-tay-table1-efficienttransformermodels.png</code>`{=html}](/doc/ai/nn/transformer/attention/2020-tay-table1-efficienttransformermodels.png){.link-annotated-partial
    .id-not .include-annotation link-icon="image"
    link-icon-type="svg"}**

-   **[`<code>https://bbycroft.net/llm</code>`{=html}](https://bbycroft.net/llm){.id-not
    .include-annotation}**

-   **[`<code>https://bclarkson-code.github.io/posts/llm-from-scratch-scalar-autograd/post.html</code>`{=html}](https://bclarkson-code.github.io/posts/llm-from-scratch-scalar-autograd/post.html){.link-annotated-partial
    .id-not .include-annotation}**

-   **[`<code>https://e2eml.school/transformers.html</code>`{=html}](https://e2eml.school/transformers.html){.link-annotated-partial
    .id-not .include-annotation}**

-   **[`<code>https://github.com/haizelabs/thorn-in-haizestack</code>`{=html}](https://github.com/haizelabs/thorn-in-haizestack){.link-annotated-partial
    .id-not .include-annotation link-icon="github"
    link-icon-type="svg"}**

-   **[`<code>https://github.com/montemac/activation_additions</code>`{=html}](https://github.com/montemac/activation_additions){.id-not
    .include-annotation link-icon="github" link-icon-type="svg"}**

-   **[`<code>https://lilianweng.github.io/posts/2018-06-24-attention/</code>`{=html}](https://lilianweng.github.io/posts/2018-06-24-attention/){.link-annotated-partial
    .id-not .include-annotation}**

-   **[`<code>https://magazine.sebastianraschka.com/p/understanding-and-coding-self-attention</code>`{=html}](https://magazine.sebastianraschka.com/p/understanding-and-coding-self-attention){.link-annotated-partial
    .id-not .include-annotation}**

-   **[`<code>https://mehta-rohan.com/writings/blog_posts/attention.html</code>`{=html}](https://mehta-rohan.com/writings/blog_posts/attention.html){.link-annotated-partial
    .id-not .include-annotation}**

-   **[`<code>https://nian.llmonpy.ai/</code>`{=html}](https://nian.llmonpy.ai/){.link-annotated-partial
    .id-not .include-annotation}**

-   **[`<code>https://nostalgebraist.tumblr.com/post/740164510909890560/information-flow-in-transformers</code>`{=html}](https://nostalgebraist.tumblr.com/post/740164510909890560/information-flow-in-transformers){.link-annotated-partial
    .id-not .include-annotation link-icon="tumblr" link-icon-type="svg"
    link-icon-color="#001935"}**

-   **[`<code>https://shyam.blog/posts/beyond-self-attention/</code>`{=html}](https://shyam.blog/posts/beyond-self-attention/){.link-annotated-partial
    .id-not .include-annotation}**

-   **[`<code>https://vgel.me/posts/handmade-transformer/</code>`{=html}](https://vgel.me/posts/handmade-transformer/){.link-annotated-partial
    .id-not .include-annotation}**

-   **[`<code>https://vgel.me/posts/representation-engineering/</code>`{=html}](https://vgel.me/posts/representation-engineering/){.link-annotated-partial
    .id-not .include-annotation}**

-   **[`<code>https://www.anthropic.com/index/100k-context-windows</code>`{=html}](https://www.anthropic.com/index/100k-context-windows){.link-annotated-partial
    .id-not .include-annotation link-icon="anthropic"
    link-icon-type="svg" link-icon-color="#d4a27f"}**

-   **[`<code>https://www.beren.io/2024-03-03-Linear-Attention-as-Iterated-Hopfield-Networks/</code>`{=html}](https://www.beren.io/2024-03-03-Linear-Attention-as-Iterated-Hopfield-Networks/){.link-annotated-partial
    .id-not .include-annotation}**

-   **[`<code>https://www.dipkumar.dev/becoming-the-unbeatable/posts/gpt-kvcache/</code>`{=html}](https://www.dipkumar.dev/becoming-the-unbeatable/posts/gpt-kvcache/){.link-annotated-partial
    .id-not .include-annotation}**

-   **[`<code>https://www.lesswrong.com/posts/7fxusXdkMNmAhkAfc/finding-sparse-linear-connections-between-features-in-llms</code>`{=html}](https://www.lesswrong.com/posts/7fxusXdkMNmAhkAfc/finding-sparse-linear-connections-between-features-in-llms){.link-annotated-partial
    .id-not .include-annotation link-icon="LW" link-icon-type="text"
    link-icon-color="#7faf83"}**

-   **[`<code>https://www.lesswrong.com/posts/Ei8q37PB3cAky6kaK/takeaways-from-a-mechanistic-interpretability-project-on</code>`{=html}](https://www.lesswrong.com/posts/Ei8q37PB3cAky6kaK/takeaways-from-a-mechanistic-interpretability-project-on){.id-not
    .include-annotation link-icon="LW" link-icon-type="text"
    link-icon-color="#7faf83"}**

    ::: aux-links-transclude-file
    ::: collapse
    **View External Link**:

    [`https://www.lesswrong.com/posts/Ei8q37PB3cAky6kaK/takeaways-from-a-mechanistic-interpretability-project-on`](https://www.lesswrong.com/posts/Ei8q37PB3cAky6kaK/takeaways-from-a-mechanistic-interpretability-project-on){.id-not
    .link-annotated-not .include-content .include-lazy link-icon="LW"
    link-icon-type="text" link-icon-color="#7faf83"}
    :::
    :::

-   **[`<code>https://www.lesswrong.com/posts/K7AyY8LMrcKhwfbyj/no-really-attention-is-all-you-need-attention-can-do</code>`{=html}](https://www.lesswrong.com/posts/K7AyY8LMrcKhwfbyj/no-really-attention-is-all-you-need-attention-can-do){.link-annotated-partial
    .id-not .include-annotation link-icon="LW" link-icon-type="text"
    link-icon-color="#7faf83"}**

-   **[`<code>https://www.lesswrong.com/posts/euam65XjigaCJQkcN/an-analogy-for-understanding-transformers</code>`{=html}](https://www.lesswrong.com/posts/euam65XjigaCJQkcN/an-analogy-for-understanding-transformers){.id-not
    .include-annotation link-icon="LW" link-icon-type="text"
    link-icon-color="#7faf83"}**

    ::: aux-links-transclude-file
    ::: collapse
    **View External Link**:

    [`https://www.lesswrong.com/posts/euam65XjigaCJQkcN/an-analogy-for-understanding-transformers`](https://www.lesswrong.com/posts/euam65XjigaCJQkcN/an-analogy-for-understanding-transformers){.id-not
    .link-annotated-not .include-content .include-lazy link-icon="LW"
    link-icon-type="text" link-icon-color="#7faf83"}
    :::
    :::

-   **[`<code>https://www.lesswrong.com/posts/jGuXSZgv6qfdhMCuJ/refusal-in-llms-is-mediated-by-a-single-direction</code>`{=html}](https://www.lesswrong.com/posts/jGuXSZgv6qfdhMCuJ/refusal-in-llms-is-mediated-by-a-single-direction){.link-annotated-partial
    .id-not .include-annotation link-icon="LW" link-icon-type="text"
    link-icon-color="#7faf83"}**

-   **[`<code>https://www.lesswrong.com/posts/kobJymvvcvhbjWFKe/laying-the-foundations-for-vision-and-multimodal-mechanistic</code>`{=html}](https://www.lesswrong.com/posts/kobJymvvcvhbjWFKe/laying-the-foundations-for-vision-and-multimodal-mechanistic){.id-not
    .include-annotation link-icon="LW" link-icon-type="text"
    link-icon-color="#7faf83"}**

    ::: aux-links-transclude-file
    ::: collapse
    **View External Link**:

    [`https://www.lesswrong.com/posts/kobJymvvcvhbjWFKe/laying-the-foundations-for-vision-and-multimodal-mechanistic`](https://www.lesswrong.com/posts/kobJymvvcvhbjWFKe/laying-the-foundations-for-vision-and-multimodal-mechanistic){.id-not
    .link-annotated-not .include-content .include-lazy link-icon="LW"
    link-icon-type="text" link-icon-color="#7faf83"}
    :::
    :::

-   **[`<code>https://www.lesswrong.com/posts/nuJFTS5iiJKT5G5yh/polysemantic-attention-head-in-a-4-layer-transformer</code>`{=html}](https://www.lesswrong.com/posts/nuJFTS5iiJKT5G5yh/polysemantic-attention-head-in-a-4-layer-transformer){.id-not
    .include-annotation link-icon="LW" link-icon-type="text"
    link-icon-color="#7faf83"}**

    ::: aux-links-transclude-file
    ::: collapse
    **View External Link**:

    [`https://www.lesswrong.com/posts/nuJFTS5iiJKT5G5yh/polysemantic-attention-head-in-a-4-layer-transformer`](https://www.lesswrong.com/posts/nuJFTS5iiJKT5G5yh/polysemantic-attention-head-in-a-4-layer-transformer){.id-not
    .link-annotated-not .include-content .include-lazy link-icon="LW"
    link-icon-type="text" link-icon-color="#7faf83"}
    :::
    :::

-   **[`<code>https://www.lesswrong.com/posts/thePw6qdyabD8XR4y/interpreting-openai-s-whisper</code>`{=html}](https://www.lesswrong.com/posts/thePw6qdyabD8XR4y/interpreting-openai-s-whisper){.link-annotated-partial
    .id-not .include-annotation link-icon="LW" link-icon-type="text"
    link-icon-color="#7faf83"}**

-   **[`<code>https://www.lesswrong.com/posts/u6KXXmKFbXfWzoAXn/a-circuit-for-python-docstrings-in-a-4-layer-attention-only</code>`{=html}](https://www.lesswrong.com/posts/u6KXXmKFbXfWzoAXn/a-circuit-for-python-docstrings-in-a-4-layer-attention-only){.id-not
    .include-annotation link-icon="LW" link-icon-type="text"
    link-icon-color="#7faf83"}**

    ::: aux-links-transclude-file
    ::: collapse
    **View External Link**:

    [`https://www.lesswrong.com/posts/u6KXXmKFbXfWzoAXn/a-circuit-for-python-docstrings-in-a-4-layer-attention-only`](https://www.lesswrong.com/posts/u6KXXmKFbXfWzoAXn/a-circuit-for-python-docstrings-in-a-4-layer-attention-only){.id-not
    .link-annotated-not .include-content .include-lazy link-icon="LW"
    link-icon-type="text" link-icon-color="#7faf83"}
    :::
    :::

-   **[`<code>https://www.lesswrong.com/postsiGuwZTHWb6DFY3sKB/fact-finding-attempting-to-reverse-engineer-factual-recall</code>`{=html}](https://www.lesswrong.com/postsiGuwZTHWb6DFY3sKB/fact-finding-attempting-to-reverse-engineer-factual-recall){.link-annotated-partial
    .id-not .include-annotation link-icon="LW" link-icon-type="text"
    link-icon-color="#7faf83"}**

-   **[`<code>https://www.perfectlynormal.co.uk/blog-induction-heads-illustrated</code>`{=html}](https://www.perfectlynormal.co.uk/blog-induction-heads-illustrated){.id-not
    .include-annotation}**

-   **[`<code>https://x.com/BrendanBycroft/status/1731042957149827140</code>`{=html}](https://x.com/BrendanBycroft/status/1731042957149827140 "Bycroft 2026"){.link-annotated-partial
    .id-not .include-annotation link-icon="twitter" link-icon-type="svg"
    link-icon-color="#1da1f2"}**

-   **[`<code>https://x.com/GregKamradt/status/1722386725635580292</code>`{=html}](https://x.com/GregKamradt/status/1722386725635580292 "Kamradt 2026"){.link-annotated-partial
    .id-not .include-annotation link-icon="twitter" link-icon-type="svg"
    link-icon-color="#1da1f2"}**

-   **[`<code>https://x.com/LouisKnightWebb/status/1724510794514157668</code>`{=html}](https://x.com/LouisKnightWebb/status/1724510794514157668 "Knight-Webb 2026"){.link-annotated-partial
    .id-not .include-annotation link-icon="twitter" link-icon-type="svg"
    link-icon-color="#1da1f2"}**

-   **[`<code>https://x.com/arankomatsuzaki/status/1622666312219598864</code>`{=html}](https://x.com/arankomatsuzaki/status/1622666312219598864 "Komatsuzaki 2026"){.link-annotated-partial
    .id-not .include-annotation link-icon="twitter" link-icon-type="svg"
    link-icon-color="#1da1f2"}**

-   **[`<code>https://x.com/bycloudai/status/2002347892535930934</code>`{=html}](https://x.com/bycloudai/status/2002347892535930934 "bycloudai 2026"){.id-not
    .include-annotation link-icon="twitter" link-icon-type="svg"
    link-icon-color="#1da1f2"}**

    ::: aux-links-transclude-file
    [`https://x.com/bycloudai/status/2002347892535930934`](https://x.com/bycloudai/status/2002347892535930934 "bycloudai 2026"){.id-not
    .include-content}
    :::

-   **[`<code>https://x.com/davisblalock/status/1939956579698094166</code>`{=html}](https://x.com/davisblalock/status/1939956579698094166 "Blalock 2026"){.id-not
    .include-annotation link-icon="twitter" link-icon-type="svg"
    link-icon-color="#1da1f2"}**

    ::: aux-links-transclude-file
    [`https://x.com/davisblalock/status/1939956579698094166`](https://x.com/davisblalock/status/1939956579698094166 "Blalock 2026"){.id-not
    .include-content}
    :::

-   **[`<code>https://x.com/karpathy/status/1864023344435380613</code>`{=html}](https://x.com/karpathy/status/1864023344435380613 "Karpathy 2026"){.id-not
    .include-annotation link-icon="twitter" link-icon-type="svg"
    link-icon-color="#1da1f2"}**

    ::: aux-links-transclude-file
    [`https://x.com/karpathy/status/1864023344435380613`](https://x.com/karpathy/status/1864023344435380613 "Karpathy 2026"){.id-not
    .include-content}
    :::

-   **[`<code>https://x.com/mathemagic1an/status/1636121914849792000</code>`{=html}](https://x.com/mathemagic1an/status/1636121914849792000 "Hack 2026"){.link-annotated-partial
    .id-not .include-annotation link-icon="twitter" link-icon-type="svg"
    link-icon-color="#1da1f2"}**

-   **[`<code>https://x.com/sand7one/status/1912012664269930896</code>`{=html}](https://x.com/sand7one/status/1912012664269930896 "sand7one 2026"){.link-annotated-partial
    .id-not .include-annotation link-icon="twitter" link-icon-type="svg"
    link-icon-color="#1da1f2"}**

-   **[`<code>https://x.com/swyx/status/1722441535235768372</code>`{=html}](https://x.com/swyx/status/1722441535235768372 "Wang 2026"){.link-annotated-partial
    .id-not .include-annotation link-icon="twitter" link-icon-type="svg"
    link-icon-color="#1da1f2"}**

# Bibliography {#link-bibliography-section}

1.  `https://arxiv.org/abs/2507.02754#facebook`:
    [`“Fast and Simplex: 2-Simplicial Attention in Triton”`{=html}](https://arxiv.org/abs/2507.02754#facebook "'Fast and Simplex: 2-Simplicial Attention in Triton', Roy et al 2025"){#roy-et-al-2025
    .link-annotated},
    [Aurko Roy, Timothy Chou, Sai Surya Duvvuri[, Sijia Chen, Jiecao Yu, [Xiaodong Wang](https://en.wikipedia.org/wiki/Xiaodong_Wang_(electrical_engineer) "Xiaodong Wang (electrical engineer)"){.id-not}, Manzil Zaheer, Rohan Anil]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2507.02754%2523facebook.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2507.02754#facebook"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

2.  `https://arxiv.org/abs/2504.14379`:
    [`“The Geometry of Self-Verification in a Task-Specific Reasoning Model”`{=html}](https://arxiv.org/abs/2504.14379 "'The Geometry of Self-Verification in a Task-Specific Reasoning Model', Lee et al 2025"){#lee-et-al-2025
    .link-annotated},
    [Andrew Lee, Lihao Sun, Chris Wendler[, [Fernanda Viégas](https://en.wikipedia.org/wiki/Fernanda_Vi%C3%A9gas "Fernanda Viégas"){.id-not}, [Martin M. Wattenberg](https://en.wikipedia.org/wiki/Martin_M._Wattenberg "Martin M. Wattenberg"){.id-not}]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2504.14379.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2504.14379"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

3.  `https://arxiv.org/abs/2503.10622#facebook`:
    [`“Dynamic Tanh: Transformers without Normalization”`{=html}](https://arxiv.org/abs/2503.10622#facebook "'Dynamic Tanh: Transformers without Normalization', Zhu et al 2025"){#zhu-et-al-2025
    .link-annotated},
    [Jiachen Zhu, [Xinlei Chen](https://xinleic.xyz/ "Xinlei Chen"){.link-annotated-partial
    .id-not}, [Kaiming He](https://en.wikipedia.org/wiki/Kaiming_He "Kaiming He"){.id-not}[, [Yann LeCun](https://en.wikipedia.org/wiki/Yann_LeCun "Yann LeCun"){.id-not}, [Zhuang Liu](https://liuzhuang13.github.io/ "Zhuang Liu’s Homepage"){.link-annotated-partial
    .id-not}]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2503.10622%2523facebook.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2503.10622#facebook"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

4.  `https://arxiv.org/abs/2502.20339`:
    [`“Thinking Slow, Fast: Scaling Inference Compute With Distilled Reasoners”`{=html}](https://arxiv.org/abs/2502.20339 "'Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasoners', Paliotta et al 2025"){#paliotta-et-al-2025
    .link-annotated},
    [Daniele Paliotta, [Junxiong Wang](https://www.cs.cornell.edu/~junxiong/ "Junxiong Wang"){.link-annotated-partial
    .id-not}, Matteo Pagliardini[, Kevin Y. Li, Aviv Bick, [J. Zico Kolter](https://en.wikipedia.org/wiki/Zico_Kolter "Zico Kolter"){.id-not}, [Albert Gu](https://scholar.google.com/citations?user=DVCHv1kAAAAJ "Albert Gu"){.id-not}, François Fleuret, [Tri Dao](https://tridao.me/ "Tri Dao"){.link-annotated-partial
    .id-not}]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2502.20339.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2502.20339"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

5.  `https://arxiv.org/abs/2501.08313#minimax`:
    [`“MiniMax-01: Scaling Foundation Models With Lightning Attention”`{=html}](https://arxiv.org/abs/2501.08313#minimax "'MiniMax-01: Scaling Foundation Models with Lightning Attention', MiniMax et al 2025"){#minimax-et-al-2025
    .link-annotated},
    [MiniMax, Aonian Li, Bangwei Gong[, [Bo Yang](https://en.wikipedia.org/wiki/Bo_Yang "Bo Yang"){.id-not}, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, Enwei Jiao, Gengxin Li, Guojun Zhang, Haohai Sun, Houze Dong, Jiadai Zhu, Jiaqi Zhuang, Jiayuan Song, Jin Zhu, Jingtao Han, Jingyang Li, Junbin Xie, Junhao Xu, Junjie Yan, Kaishun Zhang, Kecheng Xiao, Kexi Kang, Le Han, Leyang Wang, Lianfei Yu, Liheng Feng, Lin Zheng, Linbo Chai, Long Xing, Meizhi Ju, Mingyuan Chi, Mozhi Zhang, Peikai Huang, Pengcheng Niu, Pengfei Li, Pengyu Zhao, Qi Yang, Qidi Xu, Qiexiang Wang, Qin Wang, Qiuhui Li, Ruitao Leng, Shengmin Shi, Shuqi Yu, Sichen Li, Songquan Zhu, Tao Huang, Tianrun Liang, Weigao Sun, Weixuan Sun, Weiyu Cheng, Wenkai Li, Xiangjun Song, Xiao Su, Xiaodong Han, Xinjie Zhang, Xinzhu Hou, Xu Min, Xun Zou, Xuyang Shen, Yan Gong, Yingjie Zhu, Yipeng Zhou, Yiran Zhong, Yongyi Hu, Yuanxiang Fan, Yue Yu, Yufeng Yang, Yuhao Li, Yunan Huang, Yunji Li, Yunpeng Huang, Yunzhi Xu, Yuxin Mao, Zehan Li, Zekang Li, Zewei Tao, Zewen Ying, Zhaoyang Cong, Zhen Qin, Zhenhua Fan, Zhihang Yu, Zhuo Jiang, Zijia Wu]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2501.08313%2523minimax.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2501.08313#minimax"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

6.  `https://arxiv.org/abs/2410.18077#deepmind`:
    [`“ALTA: Compiler-Based Analysis of Transformers”`{=html}](https://arxiv.org/abs/2410.18077#deepmind "'ALTA: Compiler-Based Analysis of Transformers', Shaw et al 2024"){#shaw-et-al-2024
    .link-annotated},
    [Peter Shaw, James Cohan, Jacob Eisenstein[, Kenton Lee, [Jonathan Berant](https://www.cs.tau.ac.il//~joberant/ "Jonathan Berant"){.link-annotated-partial
    .id-not}, Kristina Toutanova]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2410.18077%2523deepmind.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2410.18077#deepmind"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

7.  `https://arxiv.org/abs/2410.06405`:
    [`“Tackling the Abstraction and Reasoning Corpus With Vision Transformers: the Importance of 2D Representation, Positions, and Objects”`{=html}](https://arxiv.org/abs/2410.06405 "'Tackling the Abstraction and Reasoning Corpus with Vision Transformers: the Importance of 2D Representation, Positions, and Objects', Li et al 2024"){#li-et-al-2024-2
    .link-annotated},
    [Wenhao Li, Yudong Xu, Scott Sanner, Elias Boutros Khalil]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2410.06405.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2410.06405"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

8.  `https://arxiv.org/abs/2410.01201`:
    [`“Were RNNs All We Needed?”`{=html}](https://arxiv.org/abs/2410.01201 "'Were RNNs All We Needed?', Feng et al 2024"){#feng-et-al-2024-1
    .link-annotated},
    [Leo Feng, Frederick Tung, Mohamed Osama Ahmed[, [Yoshua Bengio](https://en.wikipedia.org/wiki/Yoshua_Bengio "Yoshua Bengio"){.id-not}, Hossein Hajimirsadegh]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2410.01201.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2410.01201"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

9.  `https://arxiv.org/abs/2408.15237`:
    [`“The Mamba in the Llama: Distilling and Accelerating Hybrid Models”`{=html}](https://arxiv.org/abs/2408.15237 "'The Mamba in the Llama: Distilling and Accelerating Hybrid Models', Wang et al 2024"){#wang-et-al-2024-02
    .link-annotated},
    [[Junxiong Wang](https://www.cs.cornell.edu/~junxiong/ "Junxiong Wang"){.link-annotated-partial
    .id-not}, Daniele Paliotta, Avner May[, [Alexander M. Rush](https://rush-nlp.com/){.link-annotated-partial
    .id-not}, [Tri Dao](https://tridao.me/ "Tri Dao"){.link-annotated-partial
    .id-not}]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2408.15237.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2408.15237"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

10. `https://arxiv.org/abs/2406.15786`:
    [`“What Matters in Transformers? Not All Attention Is Needed”`{=html}](https://arxiv.org/abs/2406.15786 "'What Matters in Transformers? Not All Attention is Needed', He et al 2024"){#he-et-al-2024
    .link-annotated},
    [Shwai He, Guoheng Sun, Zheyu Shen, Ang Li]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2406.15786.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2406.15786"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

11. `https://arxiv.org/abs/2406.13121#google`:
    [`“Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?”`{=html}](https://arxiv.org/abs/2406.13121#google "'Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?', Lee et al 2024"){#lee-et-al-2024-1
    .link-annotated},
    [Jinhyuk Lee, [Anthony Chen](https://en.wikipedia.org/wiki/Anthony_Chen "Anthony Chen"){.id-not}, Zhuyun Dai[, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Sébastien M. R. Arnold, Vincent Perot, Siddharth Dalmia, Hexiang Hu, Xudong Lin, Panupong Pasupat, Aida Amini, Jeremy R. Cole, [Sebastian Riedel](http://www.riedelcastro.org/ "'Sebastian Riedel homepage', Riedel 2026"){.link-annotated-partial
    .id-not}, Iftekhar Naim, Ming-Wei Chang, Kelvin Guu]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2406.13121%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2406.13121#google"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

12. `https://arxiv.org/abs/2406.07887`:
    [`“An Empirical Study of Mamba-Based Language Models”`{=html}](https://arxiv.org/abs/2406.07887 "'An Empirical Study of Mamba-based Language Models', Waleffe et al 2024"){#waleffe-et-al-2024
    .link-annotated},
    [Roger Waleffe, Wonmin Byeon, Duncan Riach[, Brandon Norick, Vijay Korthikanti, [Tri Dao](https://tridao.me/ "Tri Dao"){.link-annotated-partial
    .id-not}, [Albert Gu](https://scholar.google.com/citations?user=DVCHv1kAAAAJ "Albert Gu"){.id-not}, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, Garvit Kulshreshtha, Vartika Singh, Jared Casper, Jan Kautz, [Mohammad Shoeybi](https://scholar.google.com/citations?user=62ElavIAAAAJ "Mohammad Shoeybi"){.id-not}, [Bryan Catanzaro](https://scholar.google.com/citations?user=UZ6kI2AAAAAJ "Bryan Catanzaro"){.id-not}]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2406.07887.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2406.07887"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

13. `https://arxiv.org/abs/2406.05564`:
    [`“Automata Extraction from Transformers”`{=html}](https://arxiv.org/abs/2406.05564 "'Automata Extraction from Transformers', Zhang et al 2024"){#zhang-et-al-2024
    .link-annotated}, [Yihao Zhang, Zeming Wei, Meng Sun]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2406.05564.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2406.05564"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

14. `https://arxiv.org/abs/2404.15574`:
    [`“Retrieval Head Mechanistically Explains Long-Context Factuality”`{=html}](https://arxiv.org/abs/2404.15574 "'Retrieval Head Mechanistically Explains Long-Context Factuality', Wu et al 2024"){#wu-et-al-2024-1
    .link-annotated},
    [Wenhao Wu, [Yizhong Wang](https://homes.cs.washington.edu/~yizhongw/ "Yizhong Wang—University of Washington"){.link-annotated-partial
    .id-not}, Guangxuan Xiao[, Hao Peng, Yao Fu]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2404.15574.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2404.15574"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

15. `https://arxiv.org/abs/2404.15758`:
    [`“Let’s Think Dot by Dot: Hidden Computation in Transformer Language Models”`{=html}](https://arxiv.org/abs/2404.15758 "'Let’s Think Dot by Dot: Hidden Computation in Transformer Language Models', Pfau et al 2024"){#pfau-et-al-2024
    .link-annotated},
    [Jacob Pfau, William Merrill, [Samuel R. Bowman](https://cims.nyu.edu/~sbowman/ "Sam Bowman"){.link-annotated-partial
    .id-not}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2404.15758.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2404.15758"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

16. `https://arxiv.org/abs/2403.18802#deepmind`:
    [`“Long-Form Factuality in Large Language Models”`{=html}](https://arxiv.org/abs/2403.18802#deepmind "'Long-form factuality in large language models', Wei et al 2024"){#wei-et-al-2024-1
    .link-annotated},
    [Jerry Wei, Chengrun Yang, Xinying Song[, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, [Quoc V. Le](https://en.wikipedia.org/wiki/Quoc_V._Le "Quoc V. Le"){.id-not}]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2403.18802%2523deepmind.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2403.18802#deepmind"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

17. `https://arxiv.org/abs/2403.17844`:
    [`“Mechanistic Design and Scaling of Hybrid Architectures”`{=html}](https://arxiv.org/abs/2403.17844 "'Mechanistic Design and Scaling of Hybrid Architectures', Poli et al 2024"){#poli-et-al-2024
    .link-annotated},
    [Michael Poli, Armin W. Thomas, Eric Nguyen[, Pragaash Ponnusamy, Björn Deiseroth, [Kristian Kersting](https://en.wikipedia.org/wiki/Kristian_Kersting "Kristian Kersting"){.id-not}, Taiji Suzuki, Brian Hie, [Stefano Ermon](https://scholar.google.com/citations?user=ogXTOZ4AAAAJ "Stefano Ermon"){.id-not}, [Christopher Ré](https://en.wikipedia.org/wiki/Christopher_R%C3%A9 "Christopher Ré"){.id-not}, Ce Zhang, Stefano Massaroli]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2403.17844.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2403.17844"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

18. `https://www.wired.com/story/eight-google-employees-invented-modern-ai-transformers-paper/`:
    [`“8 Google Employees Invented Modern AI. Here’s the Inside Story: They Met by Chance, Got Hooked on an Idea, and Wrote the Transformers Paper—The Most Consequential Tech Breakthrough in Recent History”`{=html}](https://www.wired.com/story/eight-google-employees-invented-modern-ai-transformers-paper/ "'8 Google Employees Invented Modern AI. Here’s the Inside Story: They met by chance, got hooked on an idea, and wrote the Transformers paper—the most consequential tech breakthrough in recent history', Levy 2024"){#levy-2024-2
    .link-annotated},
    [[Steven Levy](https://en.wikipedia.org/wiki/Steven_Levy "Steven Levy"){.link-annotated-partial
    .id-not}]{.author .cite-author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fwww.wired.com%252Fstory%252Feight-google-employees-invented-modern-ai-transformers-paper%252F.html "Directory-tag link-bibliography for link https://www.wired.com/story/eight-google-employees-invented-modern-ai-transformers-paper/"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

19. `https://arxiv.org/abs/2401.14391`:
    [`“Rethinking Patch Dependence for Masked Autoencoders”`{=html}](https://arxiv.org/abs/2401.14391 "'Rethinking Patch Dependence for Masked Autoencoders', Fu et al 2024"){#fu-et-al-2024-2
    .link-annotated},
    [Letian Fu, Long Lian, Renhao Wang[, Baifeng Shi, Xudong Wang, Adam Yala, [Trevor Darrell](https://en.wikipedia.org/wiki/Trevor_Darrell "Trevor Darrell"){.id-not}, [Alexei A. Efros](https://en.wikipedia.org/wiki/Alexei_A._Efros "Alexei A. Efros"){.id-not}, Ken Goldberg]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2401.14391.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2401.14391"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

20. `https://arxiv.org/abs/2312.09230`:
    [`“Successor Heads: Recurring, Interpretable Attention Heads In The Wild”`{=html}](https://arxiv.org/abs/2312.09230 "'Successor Heads: Recurring, Interpretable Attention Heads In The Wild', Gould et al 2023"){#gould-et-al-2023
    .link-annotated},
    [Rhys Gould, Euan Ong, George Ogden, Arthur Conmy]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2312.09230.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2312.09230"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

21. `https://arxiv.org/abs/2311.13657`:
    [`“Efficient Transformer Knowledge Distillation: A Performance Review”`{=html}](https://arxiv.org/abs/2311.13657 "'Efficient Transformer Knowledge Distillation: A Performance Review', Brown et al 2023"){#brown-et-al-2023
    .link-annotated},
    [Nathan Brown, Ashton Williamson, Tahj Anderson, Logan Lawrence]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2311.13657.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2311.13657"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

22. `https://arxiv.org/abs/2311.02265`:
    [`“Not All Layers Are Equally As Important: Every Layer Counts BERT”`{=html}](https://arxiv.org/abs/2311.02265 "'Not all layers are equally as important: Every Layer Counts BERT', Charpentier & Samuel 2023"){#charpentier-samuel-2023
    .link-annotated},
    [Lucas Georges Gabriel Charpentier, David Samuel]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2311.02265.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2311.02265"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

23. `https://arxiv.org/abs/2310.15154`:
    [`“Linear Representations of Sentiment in Large Language Models”`{=html}](https://arxiv.org/abs/2310.15154 "'Linear Representations of Sentiment in Large Language Models', Tigges et al 2023"){#tigges-et-al-2023
    .link-annotated},
    [Curt Tigges, Oskar John Hollinsworth, Atticus Geiger, [Neel Nanda](https://www.neelnanda.io/about){.id-not}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2310.15154.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2310.15154"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

24. `https://arxiv.org/abs/2310.02980`:
    [`“Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors”`{=html}](https://arxiv.org/abs/2310.02980 "'Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors', Amos et al 2023"){#amos-et-al-2023
    .link-annotated},
    [Ido Amos, [Jonathan Berant](https://www.cs.tau.ac.il//~joberant/ "Jonathan Berant"){.link-annotated-partial
    .id-not}, Ankit Gupta]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2310.02980.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2310.02980"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

25. `https://arxiv.org/abs/2309.10713`:
    [`“Interpret Vision Transformers As ConvNets With Dynamic Convolutions”`{=html}](https://arxiv.org/abs/2309.10713 "'Interpret Vision Transformers as ConvNets with Dynamic Convolutions', Zhou et al 2023"){#zhou-et-al-2023-05
    .link-annotated},
    [Chong Zhou, [Chen Change Loy](https://www.mmlab-ntu.com/person/ccloy/){.id-not}, [Bo Dai](https://en.wikipedia.org/wiki/Bo_Dai "Bo Dai"){.id-not}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2309.10713.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2309.10713"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

26. `https://arxiv.org/abs/2309.08586`:
    [`“Replacing Softmax With ReLU in Vision Transformers”`{=html}](https://arxiv.org/abs/2309.08586 "'Replacing softmax with ReLU in Vision Transformers', Wortsman et al 2023"){#wortsman-et-al-2023
    .link-annotated},
    [[Mitchell Wortsman](https://mitchellnw.github.io/ "Mitchell Wortsman"){.id-not}, Jaehoon Lee, Justin Gilmer, [Simon Kornblith](https://www.zotero.org/simon/cv "'Simon Kornblith CV', Kornblith 2026"){.id-not}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2309.08586.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2309.08586"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

27. `https://arxiv.org/abs/2308.10248`:
    [`“Activation Addition: Steering Language Models Without Optimization”`{=html}](https://arxiv.org/abs/2308.10248 "'Activation Addition: Steering Language Models Without Optimization', Turner et al 2023"){#turner-et-al-2023
    .link-annotated},
    [Alexander Matt Turner, Lisa Thiergart, Gavin Leech[, David Udell, Juan J. Vazquez, Ulisse Mini, Monte MacDiarmid]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2308.10248.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2308.10248"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

28. `https://arxiv.org/abs/2305.18466`:
    [`“TTT-NN: Test-Time Training on Nearest Neighbors for Large Language Models”`{=html}](https://arxiv.org/abs/2305.18466 "'TTT-NN: Test-Time Training on Nearest Neighbors for Large Language Models', Hardt & Sun 2023"){#hardt-sun-2023
    .link-annotated},
    [Moritz Hardt, [Yu Sun](https://yueatsprograms.github.io/ "Yu Sun"){.link-annotated-partial
    .id-not}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2305.18466.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2305.18466"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

29. `https://arxiv.org/abs/2306.00008#google`:
    [`“Brainformers: Trading Simplicity for Efficiency”`{=html}](https://arxiv.org/abs/2306.00008#google "'Brainformers: Trading Simplicity for Efficiency', Zhou et al 2023"){#zhou-et-al-2023-08
    .link-annotated},
    [Yanqi Zhou, Nan Du, [Yanping Huang](https://scholar.google.com/citations?user=uEtBQScAAAAJ "Yanping Huang"){.id-not}[, Daiyi Peng, Chang Lan, Da Huang, Siamak Shakeri, David So, Andrew Dai, Yifeng Lu, [Zhifeng Chen](https://scholar.google.com/citations?user=EemUE4gAAAAJ "Zhifeng Chen"){.id-not}, [Quoc V. Le](https://en.wikipedia.org/wiki/Quoc_V._Le "Quoc V. Le"){.id-not}, Claire Cui, James Laundon, [Jeff Dean](https://en.wikipedia.org/wiki/Jeff_Dean "Jeff Dean"){.link-annotated-partial
    .id-not}]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2306.00008%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2306.00008#google"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

30. `https://arxiv.org/abs/2305.09828`:
    [`“Mimetic Initialization of Self-Attention Layers”`{=html}](https://arxiv.org/abs/2305.09828 "'Mimetic Initialization of Self-Attention Layers', Trockman & Kolter 2023"){#trockman-kolter-2023
    .link-annotated},
    [Asher Trockman, [J. Zico Kolter](https://en.wikipedia.org/wiki/Zico_Kolter "Zico Kolter"){.id-not}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2305.09828.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2305.09828"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

31. `https://arxiv.org/abs/2301.02240`:
    [`“Skip-Attention: Improving Vision Transformers by Paying Less Attention”`{=html}](https://arxiv.org/abs/2301.02240 "'Skip-Attention: Improving Vision Transformers by Paying Less Attention', Venkataramanan et al 2023"){#venkataramanan-et-al-2023
    .link-annotated},
    [Shashanka Venkataramanan, Amir Ghodrati, Yuki M. Asano[, [Fatih Porikli](https://en.wikipedia.org/wiki/Fatih_Porikli "Fatih Porikli"){.id-not}, Amirhossein Habibian]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2301.02240.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2301.02240"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

32. `https://arxiv.org/abs/2212.14052`:
    [`“Hungry Hungry Hippos: Towards Language Modeling With State Space Models”`{=html}](https://arxiv.org/abs/2212.14052 "'Hungry Hungry Hippos: Towards Language Modeling with State Space Models', Fu et al 2022"){#fu-et-al-2022-1
    .link-annotated},
    [Daniel Y. Fu, [Tri Dao](https://tridao.me/ "Tri Dao"){.link-annotated-partial
    .id-not}, Khaled K. Saab[, Armin W. Thomas, [Atri Rudra](https://cse.buffalo.edu/faculty/atri/ "Atri Rudra"){.id-not}, [Christopher Ré](https://en.wikipedia.org/wiki/Christopher_R%C3%A9 "Christopher Ré"){.id-not}]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2212.14052.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2212.14052"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

33. `https://arxiv.org/abs/2212.10544`:
    [`“Pretraining Without Attention”`{=html}](https://arxiv.org/abs/2212.10544 "'Pretraining Without Attention', Wang et al 2022"){#wang-et-al-2022-05
    .link-annotated},
    [[Junxiong Wang](https://www.cs.cornell.edu/~junxiong/ "Junxiong Wang"){.link-annotated-partial
    .id-not}, Jing Nathan Yan, [Albert Gu](https://scholar.google.com/citations?user=DVCHv1kAAAAJ "Albert Gu"){.id-not}, [Alexander M. Rush](https://rush-nlp.com/){.link-annotated-partial
    .id-not}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2212.10544.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2212.10544"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

34. `https://arxiv.org/abs/2212.07677#google`:
    [`“Transformers Learn In-Context by Gradient Descent”`{=html}](https://arxiv.org/abs/2212.07677#google "'Transformers learn in-context by gradient descent', Oswald et al 2022"){#oswald-et-al-2022
    .link-annotated},
    [Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo[, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, Max Vladymyrov]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2212.07677%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2212.07677#google"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

35. `https://arxiv.org/abs/2211.05102#google`:
    [`“Efficiently Scaling Transformer Inference”`{=html}](https://arxiv.org/abs/2211.05102#google "'Efficiently Scaling Transformer Inference', Pope et al 2022"){#pope-et-al-2022
    .link-annotated},
    [Reiner Pope, [Sholto Douglas](https://en.wikipedia.org/wiki/Sholto_Douglas "Sholto Douglas"){.id-not}, Aakanksha Chowdhery[, Jacob Devlin, [James Bradbury](https://x.com/jekbradbury "Bradbury 2026"){.id-not}, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, [Jeff Dean](https://en.wikipedia.org/wiki/Jeff_Dean "Jeff Dean"){.link-annotated-partial
    .id-not}]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2211.05102%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2211.05102#google"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

36. `https://arxiv.org/abs/2210.05043`:
    [`“Multi-CLS BERT: An Efficient Alternative to Traditional Ensembling”`{=html}](https://arxiv.org/abs/2210.05043 "'Multi-CLS BERT: An Efficient Alternative to Traditional Ensembling', Chang et al 2022"){#chang-et-al-2022-3
    .link-annotated},
    [Haw-Shiuan Chang, Ruei-Yao Sun, Kathryn Ricci, [Andrew McCallum](https://en.wikipedia.org/wiki/Andrew_McCallum "Andrew McCallum"){.id-not}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2210.05043.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2210.05043"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

37. `https://arxiv.org/abs/2206.01649#schmidhuber`:
    [`“Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules”`{=html}](https://arxiv.org/abs/2206.01649#schmidhuber "'Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules', Irie et al 2022"){#irie-et-al-2022-1
    .link-annotated},
    [Kazuki Irie, Francesco Faccio, [Jürgen Schmidhuber](https://en.wikipedia.org/wiki/J%C3%BCrgen_Schmidhuber "Jürgen Schmidhuber"){.link-annotated-partial
    .id-not}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2206.01649%2523schmidhuber.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2206.01649#schmidhuber"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

38. `https://arxiv.org/abs/2205.14135`:
    [`“FlashAttention: Fast and Memory-Efficient Exact Attention With IO-Awareness”`{=html}](https://arxiv.org/abs/2205.14135 "'FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness', Dao et al 2022"){#dao-et-al-2022-1
    .link-annotated},
    [[Tri Dao](https://tridao.me/ "Tri Dao"){.link-annotated-partial
    .id-not}, Daniel Y. Fu, [Stefano Ermon](https://scholar.google.com/citations?user=ogXTOZ4AAAAJ "Stefano Ermon"){.id-not}[, [Atri Rudra](https://cse.buffalo.edu/faculty/atri/ "Atri Rudra"){.id-not}, [Christopher Ré](https://en.wikipedia.org/wiki/Christopher_R%C3%A9 "Christopher Ré"){.id-not}]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2205.14135.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2205.14135"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

39. `https://arxiv.org/abs/2204.03638#facebook`:
    [`“TATS: Long Video Generation With Time-Agnostic VQGAN and Time-Sensitive Transformer”`{=html}](https://arxiv.org/abs/2204.03638#facebook "'TATS: Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer', Ge et al 2022"){#ge-et-al-2022-2
    .link-annotated},
    [Songwei Ge, Thomas Hayes, Harry Yang[, [Xi Yin](https://en.wikipedia.org/wiki/Xi_Yin "Xi Yin"){.id-not}, Guan Pang, David Jacobs, Jia-Bin Huang, [Devi Parikh](https://en.wikipedia.org/wiki/Devi_Parikh "Devi Parikh"){.id-not}]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2204.03638%2523facebook.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2204.03638#facebook"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

40. `https://arxiv.org/abs/2202.09729`:
    [`“It’s Raw! Audio Generation With State-Space Models”`{=html}](https://arxiv.org/abs/2202.09729 "'It’s Raw! Audio Generation with State-Space Models', Goel et al 2022"){#goel-et-al-2022-2
    .link-annotated},
    [[Karan Goel](https://en.wikipedia.org/wiki/Karan_Goel "Karan Goel"){.id-not}, [Albert Gu](https://scholar.google.com/citations?user=DVCHv1kAAAAJ "Albert Gu"){.id-not}, [Chris Donahue](https://en.wikipedia.org/wiki/Chris_Donahue "Chris Donahue"){.id-not}, [Christopher Ré](https://en.wikipedia.org/wiki/Christopher_R%C3%A9 "Christopher Ré"){.id-not}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2202.09729.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2202.09729"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

41. `https://arxiv.org/abs/2202.07765#deepmind`:
    [`“General-Purpose, Long-Context Autoregressive Modeling With Perceiver AR”`{=html}](https://arxiv.org/abs/2202.07765#deepmind "'General-purpose, long-context autoregressive modeling with Perceiver AR', Hawthorne et al 2022"){#hawthorne-et-al-2022
    .link-annotated},
    [Curtis Hawthorne, Andrew Jaegle, Cătălina Cangea[, Sebastian Borgeaud, Charlie Nash, [Mateusz Malinowski](https://en.wikipedia.org/wiki/Mateusz_Malinowski "Mateusz Malinowski"){.id-not}, [Sander Dieleman](https://sander.ai/about/ "About me—Sander Dieleman"){.id-not}, [Oriol Vinyals](https://en.wikipedia.org/wiki/Oriol_Vinyals "Oriol Vinyals"){.id-not}, [Matthew Botvinick](https://scholar.google.com/citations?user=eM916YMAAAAJ "Matthew Botvinick"){.id-not}, Ian Simon, Hannah Sheahan, Neil Zeghidour, Jean-Baptiste Alayrac, João Carreira, Jesse Engel]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2202.07765%2523deepmind.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2202.07765#deepmind"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

42. `https://arxiv.org/abs/2108.12409#facebook`:
    [`“Train Short, Test Long: Attention With Linear Biases (ALiBi) Enables Input Length Extrapolation”`{=html}](https://arxiv.org/abs/2108.12409#facebook "'Train Short, Test Long: Attention with Linear Biases (ALiBi) Enables Input Length Extrapolation', Press et al 2021"){#press-et-al-2021
    .link-annotated},
    [Ofir Press, [Noah Smith](https://en.wikipedia.org/wiki/Noah_Smith_(writer) "Noah Smith (writer)"){.id-not}, [Mike Lewis](https://scholar.google.com/citations?user=SnQnQicAAAAJ "Mike Lewis"){.id-not}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2108.12409%2523facebook.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2108.12409#facebook"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

43. `https://arxiv.org/abs/2108.08810#google`:
    [`“Do Vision Transformers See Like Convolutional Neural Networks?”`{=html}](https://arxiv.org/abs/2108.08810#google "'Do Vision Transformers See Like Convolutional Neural Networks?', Raghu et al 2021"){#raghu-et-al-2021
    .link-annotated},
    [Maithra Raghu, Thomas Unterthiner, [Simon Kornblith](https://www.zotero.org/simon/cv "'Simon Kornblith CV', Kornblith 2026"){.id-not}[, Chiyuan Zhang, Alexey Dosovitskiy]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2108.08810%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2108.08810#google"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

44. `https://arxiv.org/abs/2106.06981`:
    [`“RASP: Thinking Like Transformers”`{=html}](https://arxiv.org/abs/2106.06981 "'RASP: Thinking Like Transformers', Weiss et al 2021"){#weiss-et-al-2021
    .link-annotated},
    [Gail Weiss, [Yoav Goldberg](https://scholar.google.com/citations?user=0rskDKgAAAAJ "Yoav Goldberg"){.id-not}, Eran Yahav]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2106.06981.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2106.06981"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

45. `https://arxiv.org/abs/2105.15075`:
    [`“Not All Images Are Worth 16×16 Words: Dynamic Transformers for Efficient Image Recognition”`{=html}](https://arxiv.org/abs/2105.15075 "'Not All Images are Worth 16×16 Words: Dynamic Transformers for Efficient Image Recognition', Wang et al 2021"){#wang-et-al-2021-06
    .link-annotated},
    [Yulin Wang, Rui Huang, Shiji Song[, Zeyi Huang, [Gao Huang](https://en.wikipedia.org/wiki/Gao_Huang "Gao Huang"){.id-not}]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2105.15075.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2105.15075"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

46. `https://arxiv.org/abs/2105.14217`:
    [`“Less Is More: Pay Less Attention in Vision Transformers”`{=html}](https://arxiv.org/abs/2105.14217 "'Less is More: Pay Less Attention in Vision Transformers', Pan et al 2021"){#pan-et-al-2021
    .link-annotated},
    [Zizheng Pan, Bohan Zhuang, Haoyu He[, Jing Liu, Jianfei Cai]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2105.14217.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2105.14217"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

47. `https://arxiv.org/abs/2105.03824#google`:
    [`“FNet: Mixing Tokens With Fourier Transforms”`{=html}](https://arxiv.org/abs/2105.03824#google "'FNet: Mixing Tokens with Fourier Transforms', Lee-Thorp et al 2021"){#lee-thorp-et-al-2021
    .link-annotated},
    [James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, Santiago Ontanon]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2105.03824%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2105.03824#google"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

48. `https://arxiv.org/abs/2105.02723`:
    [`“Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet”`{=html}](https://arxiv.org/abs/2105.02723 "'Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet', Melas-Kyriazi 2021"){#melas-kyriazi-2021
    .link-annotated}, [Luke Melas-Kyriazi]{.author .cite-author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2105.02723.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2105.02723"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

49. `https://openreview.net/forum?id=qVyeW-grC2k#google`:
    [`“Long Range Arena (LRA): A Benchmark for Efficient Transformers”`{=html}](https://openreview.net/forum?id=qVyeW-grC2k#google "'Long Range Arena (LRA): A Benchmark for Efficient Transformers', Tay et al 2020"){#tay-et-al-2020-1
    .link-annotated},
    [[Yi Tay](https://www.yitay.net/ "Yi Tay"){.link-annotated-partial
    .id-not}, [Mostafa Dehghani](https://scholar.google.com/citations?user=MiHOX3QAAAAJ "Mostafa Dehghani"){.id-not}, Samira Abnar[, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, [Donald Metzler](https://www.don-metzler.net/ "About—Donald Metzler"){.link-annotated-partial
    .id-not}]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Fopenreview.net%252Fforum%253Fid%253DqVyeW-grC2k%2523google.html "Directory-tag link-bibliography for link https://openreview.net/forum?id=qVyeW-grC2k#google"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

50. `https://arxiv.org/abs/2009.06732#google`:
    [`“Efficient Transformers: A Survey”`{=html}](https://arxiv.org/abs/2009.06732#google "'Efficient Transformers: A Survey', Tay et al 2020"){#tay-et-al-2020-efficienttransformers
    .link-annotated},
    [[Yi Tay](https://www.yitay.net/ "Yi Tay"){.link-annotated-partial
    .id-not}, [Mostafa Dehghani](https://scholar.google.com/citations?user=MiHOX3QAAAAJ "Mostafa Dehghani"){.id-not}, Dara Bahri, [Donald Metzler](https://www.don-metzler.net/ "About—Donald Metzler"){.link-annotated-partial
    .id-not}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2009.06732%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2009.06732#google"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

51. `https://arxiv.org/abs/2008.07669`:
    [`“HiPPO: Recurrent Memory With Optimal Polynomial Projections”`{=html}](https://arxiv.org/abs/2008.07669 "'HiPPO: Recurrent Memory with Optimal Polynomial Projections', Gu et al 2020"){#gu-et-al-2021-hippo
    .link-annotated},
    [[Albert Gu](https://scholar.google.com/citations?user=DVCHv1kAAAAJ "Albert Gu"){.id-not}, [Tri Dao](https://tridao.me/ "Tri Dao"){.link-annotated-partial
    .id-not}, [Stefano Ermon](https://scholar.google.com/citations?user=ogXTOZ4AAAAJ "Stefano Ermon"){.id-not}[, [Atri Rudra](https://cse.buffalo.edu/faculty/atri/ "Atri Rudra"){.id-not}, [Christopher Re](https://en.wikipedia.org/wiki/Christopher_Re "Christopher Re"){.id-not}]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2008.07669.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2008.07669"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

52. `abstract`:
    [`“Efficient Attention: Breaking The Quadratic Transformer Bottleneck”`{=html}](/doc/ai/nn/transformer/attention/abstract "'Efficient Attention: Breaking The Quadratic Transformer Bottleneck', Gwern 2020"){#gwern-doc-ai-nn-transformer-attention-abstract-2
    .link-annotated},
    [[Gwern](/index#abstract "'Gwern Bibliography', Gwern 2010"){.link-annotated-partial
    .id-not}]{.author .cite-author}

    [[link-bibliography](/metadata/annotation/link-bibliography/%252Fdoc%252Fai%252Fnn%252Ftransformer%252Fattention%252Fabstract.html "Directory-tag link-bibliography for link /doc/ai/nn/transformer/attention/abstract"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

53. `https://arxiv.org/abs/2005.00743#google`:
    [`“Synthesizer: Rethinking Self-Attention in Transformer Models”`{=html}](https://arxiv.org/abs/2005.00743#google "'Synthesizer: Rethinking Self-Attention in Transformer Models', Tay et al 2020"){#tay-et-al-2020-2
    .link-annotated},
    [[Yi Tay](https://www.yitay.net/ "Yi Tay"){.link-annotated-partial
    .id-not}, Dara Bahri, [Donald Metzler](https://www.don-metzler.net/ "About—Donald Metzler"){.link-annotated-partial
    .id-not}[, Da-Cheng Juan, Zhe Zhao, Che Zheng]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2005.00743%2523google.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2005.00743#google"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

54. `https://arxiv.org/abs/2003.07845`:
    [`“PowerNorm: Rethinking Batch Normalization in Transformers”`{=html}](https://arxiv.org/abs/2003.07845 "'PowerNorm: Rethinking Batch Normalization in Transformers', Shen et al 2020"){#shen-et-al-2020-1
    .link-annotated},
    [[Sheng Shen](https://sincerass.github.io/ "Sheng Shen’s Homepage"){.link-annotated-partial
    .id-not}, Zhewei Yao, Amir Gholami[, Michael W. Mahoney, [Kurt Keutzer](https://en.wikipedia.org/wiki/Kurt_Keutzer "Kurt Keutzer"){.id-not}]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2003.07845.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2003.07845"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

55. `https://arxiv.org/abs/2001.09309`:
    [`“BERT’s Output Layer Recognizes All Hidden Layers? Some Intriguing Phenomena and a Simple Way to Boost BERT”`{=html}](https://arxiv.org/abs/2001.09309 "'BERT’s output layer recognizes all hidden layers? Some Intriguing Phenomena and a simple way to boost BERT', Kao et al 2020"){#kao-et-al-2020
    .link-annotated},
    [Wei-Tsung Kao, Tsung-Han Wu, Po-Han Chi[, Chun-Cheng Hsieh, Hung-Yi Lee]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F2001.09309.html "Directory-tag link-bibliography for link https://arxiv.org/abs/2001.09309"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}

56. `https://arxiv.org/abs/1912.03458#microsoft`:
    [`“Dynamic Convolution: Attention over Convolution Kernels”`{=html}](https://arxiv.org/abs/1912.03458#microsoft "'Dynamic Convolution: Attention over Convolution Kernels', Chen et al 2019"){#chen-et-al-2019
    .link-annotated},
    [Yinpeng Chen, Xiyang Dai, Mengchen Liu[, Dongdong Chen, Lu Yuan, [Zicheng Liu](https://en.wikipedia.org/wiki/Zicheng_Liu "Zicheng Liu"){.id-not}]{.collapse}]{.author}

    [[link-bibliography](/metadata/annotation/link-bibliography/https%253A%252F%252Farxiv.org%252Fabs%252F1912.03458%2523microsoft.html "Directory-tag link-bibliography for link https://arxiv.org/abs/1912.03458#microsoft"){.id-not
    .include-even-when-collapsed}]{.collapse
    .tag-index-link-bibliography-block}