self-attention tag

Gwern Branwen

See Also
Gwern
Links
Miscellaneous
Link Bibliography

[Warning: JavaScript Disabled!]

[For support of key website features (link annotation popups/popovers & transclusions, collapsible sections, backlinks, tablesorting, image zooming, sidenotes etc), you must enable JavaScript.]

[page summary]

Gwern

“Absolute Unit NNs: Regression-Based MLPs for Everything”, Gwern 2023

Absolute Unit NNs: Regression-Based MLPs for Everything

“GPT-3 Creative Fiction”, Gwern 2020

GPT-3 Creative Fiction

“Efficient Attention: Breaking The Quadratic Transformer Bottleneck”, Gwern 2020

Efficient Attention: Breaking The Quadratic Transformer Bottleneck

Links

“ReFT: Representation Finetuning for Language Models”, Wu et al 2024

ReFT: Representation Finetuning for Language Models

“How Well Can Transformers Emulate In-Context Newton’s Method?”, Giannou et al 2024

How Well Can Transformers Emulate In-context Newton’s Method?

“RNNs Are Not Transformers (Yet): The Key Bottleneck on In-Context Retrieval”, Wen et al 2024

RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval

“Rethinking Patch Dependence for Masked Autoencoders”, Fu et al 2024

Rethinking Patch Dependence for Masked Autoencoders

“Attention versus Contrastive Learning of Tabular Data—A Data-Centric Benchmarking”, Rabbani et al 2024

Attention versus Contrastive Learning of Tabular Data—A Data-centric Benchmarking

“SwitchHead: Accelerating Transformers With Mixture-Of-Experts Attention”, Csordás et al 2023

SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention

“Look Before You Leap: A Universal Emergent Decomposition of Retrieval Tasks in Language Models”, Variengien & Winsor 2023

Look Before You Leap: A Universal Emergent Decomposition of Retrieval Tasks in Language Models

“Can a Transformer Represent a Kalman Filter?”, Goel & Bartlett 2023

Can a Transformer Represent a Kalman Filter?

“Efficient Transformer Knowledge Distillation: A Performance Review”, Brown et al 2023

Efficient Transformer Knowledge Distillation: A Performance Review

“Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks As an Alternative to Attention Layers in Transformers”, Bozic et al 2023

Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers

“In-Context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering”, Liu et al 2023

In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering

“On Prefrontal Working Memory and Hippocampal Episodic Memory: Unifying Memories Stored in Weights and Activation Slots”, Whittington et al 2023

On prefrontal working memory and hippocampal episodic memory: Unifying memories stored in weights and activation slots

“LSS Transformer: Ultra-Long Sequence Distributed Transformer”, Wang et al 2023

LSS Transformer: Ultra-Long Sequence Distributed Transformer

“Simplifying Transformer Blocks”, He & Hofmann 2023

Simplifying Transformer Blocks

“GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling”, Katsch 2023

GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modeling

“Not All Layers Are Equally As Important: Every Layer Counts BERT”, Charpentier & Samuel 2023

Not all layers are equally as important: Every Layer Counts BERT

“Implicit Chain-Of-Thought Reasoning via Knowledge Distillation”, Deng et al 2023

Implicit Chain-of-Thought Reasoning via Knowledge Distillation

“Training Dynamics of Contextual N-Grams in Language Models”, Quirke et al 2023

Training Dynamics of Contextual N-Grams in Language Models

“The Impact of Depth and Width on Transformer Language Model Generalization”, Petty et al 2023

The Impact of Depth and Width on Transformer Language Model Generalization

“Transformers Learn Higher-Order Optimization Methods for In-Context Learning: A Study With Linear Models”, Fu et al 2023

Transformers Learn Higher-Order Optimization Methods for In-Context Learning: A Study with Linear Models

“Linear Representations of Sentiment in Large Language Models”, Tigges et al 2023

Linear Representations of Sentiment in Large Language Models

“Masked Hard-Attention Transformers and Boolean RASP Recognize Exactly the Star-Free Languages”, Angluin et al 2023

Masked Hard-Attention Transformers and Boolean RASP Recognize Exactly the Star-Free Languages

“How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?”, Wu et al 2023

How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?

“Vision Transformers Need Registers”, Darcet et al 2023

Vision Transformers Need Registers

“Interpret Vision Transformers As ConvNets With Dynamic Convolutions”, Zhou et al 2023

Interpret Vision Transformers as ConvNets with Dynamic Convolutions

“Replacing Softmax With ReLU in Vision Transformers”, Wortsman et al 2023

Replacing softmax with ReLU in Vision Transformers

“Linearity of Relation Decoding in Transformer Language Models”, Hernandez et al 2023

Linearity of Relation Decoding in Transformer Language Models

“Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla”, Lieberum et al 2023

Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla

“One Step of Gradient Descent Is Provably the Optimal In-Context Learner With One Layer of Linear Self-Attention”, Mahankali et al 2023

One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention

“Lost in the Middle: How Language Models Use Long Contexts”, Liu et al 2023

Lost in the Middle: How Language Models Use Long Contexts

“Trainable Transformer in Transformer”, Panigrahi et al 2023

Trainable Transformer in Transformer

“Transformers Learn to Implement Preconditioned Gradient Descent for In-Context Learning”, Ahn et al 2023

Transformers learn to implement preconditioned gradient descent for in-context learning

“White-Box Transformers via Sparse Rate Reduction”, Yu et al 2023

White-Box Transformers via Sparse Rate Reduction

“Blockwise Parallel Transformer for Long Context Large Models”, Liu & Abbeel 2023

Blockwise Parallel Transformer for Long Context Large Models

“TTT-NN: Test-Time Training on Nearest Neighbors for Large Language Models”, Hardt & Sun 2023

TTT-NN: Test-Time Training on Nearest Neighbors for Large Language Models

“Brainformers: Trading Simplicity for Efficiency”, Zhou et al 2023

Brainformers: Trading Simplicity for Efficiency

“Toeplitz Neural Network for Sequence Modeling”, Qin et al 2023

Toeplitz Neural Network for Sequence Modeling

“Finding Neurons in a Haystack: Case Studies With Sparse Probing”, Gurnee et al 2023

Finding Neurons in a Haystack: Case Studies with Sparse Probing

“How Does GPT-2 Compute Greater-Than?: Interpreting Mathematical Abilities in a Pre-Trained Language Model”, Hanna et al 2023

How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model

“Coinductive Guide to Inductive Transformer Heads”, Nemecek 2023

Coinductive guide to inductive transformer heads

“Tighter Bounds on the Expressivity of Transformer Encoders”, Chiang et al 2023

Tighter Bounds on the Expressivity of Transformer Encoders

“Tracr: Compiled Transformers As a Laboratory for Interpretability”, Lindner et al 2023

Tracr: Compiled Transformers as a Laboratory for Interpretability

“Hungry Hungry Hippos: Towards Language Modeling With State Space Models”, Fu et al 2022

Hungry Hungry Hippos: Towards Language Modeling with State Space Models

“Pretraining Without Attention”, Wang et al 2022

Pretraining Without Attention

“Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent As Meta-Optimizers”, Dai et al 2022

Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers

“Transformers Learn In-Context by Gradient Descent”, Oswald et al 2022

Transformers learn in-context by gradient descent

“What Learning Algorithm Is In-Context Learning? Investigations With Linear Models”, Akyürek et al 2022

What learning algorithm is in-context learning? Investigations with linear models

“Efficiently Scaling Transformer Inference”, Pope et al 2022

Efficiently Scaling Transformer Inference

“Transformers Learn Shortcuts to Automata”, Liu et al 2022

Transformers Learn Shortcuts to Automata

“Transformers Implement First-Order Logic With Majority Quantifiers”, Merrill & Sabharwal 2022

Transformers Implement First-Order Logic with Majority Quantifiers

“Relaxed Attention for Transformer Models”, Lohrenz et al 2022

Relaxed Attention for Transformer Models

“What Can Transformers Learn In-Context? A Case Study of Simple Function Classes”, Garg et al 2022

What Can Transformers Learn In-Context? A Case Study of Simple Function Classes

“Multitrack Music Transformer: Learning Long-Term Dependencies in Music With Diverse Instruments”, Dong et al 2022

Multitrack Music Transformer: Learning Long-Term Dependencies in Music with Diverse Instruments

“N-Grammer: Augmenting Transformers With Latent n-Grams”, Roy et al 2022

N-Grammer: Augmenting Transformers with latent n-grams

“Log-Precision Transformers Are Constant-Depth Uniform Threshold Circuits”, Merrill & Sabharwal 2022

Log-Precision Transformers are Constant-Depth Uniform Threshold Circuits

“Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules”, Irie et al 2022

Neural Differential Equations for Learning to Program Neural Nets Through Continuous Learning Rules

“FlashAttention: Fast and Memory-Efficient Exact Attention With IO-Awareness”, Dao et al 2022

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness

“TATS: Long Video Generation With Time-Agnostic VQGAN and Time-Sensitive Transformer”, Ge et al 2022

TATS: Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer

“Overcoming a Theoretical Limitation of Self-Attention”, Chiang & Cholak 2022

Overcoming a Theoretical Limitation of Self-Attention

“It’s Raw! Audio Generation With State-Space Models”, Goel et al 2022

It’s Raw! Audio Generation with State-Space Models

“General-Purpose, Long-Context Autoregressive Modeling With Perceiver AR”, Hawthorne et al 2022

General-purpose, long-context autoregressive modeling with Perceiver AR

“Transformer Memory As a Differentiable Search Index”, Tay et al 2022

Transformer Memory as a Differentiable Search Index

“The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention”, Irie et al 2022

The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention

“Attention Approximates Sparse Distributed Memory”, Bricken & Pehlevan 2021

Attention Approximates Sparse Distributed Memory

“An Explanation of In-Context Learning As Implicit Bayesian Inference”, Xie et al 2021

An Explanation of In-context Learning as Implicit Bayesian Inference

“Long-Range Transformers for Dynamic Spatiotemporal Forecasting”, Grigsby et al 2021

Long-Range Transformers for Dynamic Spatiotemporal Forecasting

“Train Short, Test Long: Attention With Linear Biases (ALiBi) Enables Input Length Extrapolation”, Press et al 2021

Train Short, Test Long: Attention with Linear Biases (ALiBi) Enables Input Length Extrapolation

“Do Vision Transformers See Like Convolutional Neural Networks?”, Raghu et al 2021

Do Vision Transformers See Like Convolutional Neural Networks?

“Stable, Fast and Accurate: Kernelized Attention With Relative Positional Encoding”, Luo et al 2021

Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding

“RASP: Thinking Like Transformers”, Weiss et al 2021

RASP: Thinking Like Transformers

“On the Distribution, Sparsity, and Inference-Time Quantization of Attention Values in Transformers”, Ji et al 2021

On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers

“SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training”, Somepalli et al 2021

SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training

“Not All Images Are Worth 16×16 Words: Dynamic Transformers for Efficient Image Recognition”, Wang et al 2021

Not All Images are Worth 16×16 Words: Dynamic Transformers for Efficient Image Recognition

“Less Is More: Pay Less Attention in Vision Transformers”, Pan et al 2021

Less is More: Pay Less Attention in Vision Transformers

“FNet: Mixing Tokens With Fourier Transforms”, Lee-Thorp et al 2021

FNet: Mixing Tokens with Fourier Transforms

“Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet”, Melas-Kyriazi 2021

Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet

“RoFormer: Enhanced Transformer With Rotary Position Embedding”, Su et al 2021

RoFormer: Enhanced Transformer with Rotary Position Embedding

“ALD: Efficient Transformers in Reinforcement Learning Using Actor-Learner Distillation”, Parisotto & Salakhutdinov 2021

ALD: Efficient Transformers in Reinforcement Learning using Actor-Learner Distillation

“Attention Is Not All You Need: Pure Attention Loses Rank Doubly Exponentially With Depth”, Dong et al 2021

Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth

“Do Transformer Modifications Transfer Across Implementations and Applications?”, Narang et al 2021

Do Transformer Modifications Transfer Across Implementations and Applications?

“Linear Transformers Are Secretly Fast Weight Programmers”, Schlag et al 2021

Linear Transformers Are Secretly Fast Weight Programmers

“Unlocking Pixels for Reinforcement Learning via Implicit Attention”, Choromanski et al 2021

Unlocking Pixels for Reinforcement Learning via Implicit Attention

“AdnFM: An Attentive DenseNet Based Factorization Machine for CTR Prediction”, Wang et al 2020

AdnFM: An Attentive DenseNet based Factorization Machine for CTR Prediction

“Long Range Arena (LRA): A Benchmark for Efficient Transformers”, Tay et al 2020

Long Range Arena (LRA): A Benchmark for Efficient Transformers

“Current Limitations of Language Models: What You Need Is Retrieval”, Komatsuzaki 2020

Current Limitations of Language Models: What You Need is Retrieval

“Efficient Transformers: A Survey”, Tay et al 2020

Efficient Transformers: A Survey

“HiPPO: Recurrent Memory With Optimal Polynomial Projections”, Gu et al 2020

HiPPO: Recurrent Memory with Optimal Polynomial Projections

“Pre-Training via Paraphrasing”, Lewis et al 2020

Pre-training via Paraphrasing

“Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers”, Choromanski et al 2020

Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers

“GPT-3: Language Models Are Few-Shot Learners”, Brown et al 2020

GPT-3: Language Models are Few-Shot Learners

“Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, Lewis et al 2020

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

“Synthesizer: Rethinking Self-Attention in Transformer Models”, Tay et al 2020

Synthesizer: Rethinking Self-Attention in Transformer Models

“PowerNorm: Rethinking Batch Normalization in Transformers”, Shen et al 2020

PowerNorm: Rethinking Batch Normalization in Transformers

“REALM: Retrieval-Augmented Language Model Pre-Training”, Guu et al 2020

REALM: Retrieval-Augmented Language Model Pre-Training

“Rethinking Attention With Performers”, Choromanski & Colwell 2020

Rethinking Attention with Performers

“Generalization through Memorization: Nearest Neighbor Language Models”, Khandelwal et al 2019

Generalization through Memorization: Nearest Neighbor Language Models

“Multiplicative Interactions and Where to Find Them”, Jayakumar et al 2019

Multiplicative Interactions and Where to Find Them

“The Bottom-Up Evolution of Representations in the Transformer: A Study With Machine Translation and Language Modeling Objectives”, Voita et al 2019

The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives

“Large Memory Layers With Product Keys”, Lample et al 2019

Large Memory Layers with Product Keys

“What Does BERT Look At? An Analysis of BERT’s Attention”, Clark et al 2019

What Does BERT Look At? An Analysis of BERT’s Attention

“Pay Less Attention With Lightweight and Dynamic Convolutions”, Wu et al 2019

Pay Less Attention with Lightweight and Dynamic Convolutions

“On the Turing Completeness of Modern Neural Network Architectures”, Pérez et al 2019

On the Turing Completeness of Modern Neural Network Architectures

“Music Transformer”, Huang et al 2018

Music Transformer

“Character-Level Language Modeling With Deeper Self-Attention”, Al-Rfou et al 2018

Character-Level Language Modeling with Deeper Self-Attention

“Attention Is All You Need”, Vaswani et al 2017

Attention Is All You Need

“A Deep Reinforced Model for Abstractive Summarization”, Paulus et al 2017

A Deep Reinforced Model for Abstractive Summarization

“Get To The Point: Summarization With Pointer-Generator Networks”, See et al 2017

Get To The Point: Summarization with Pointer-Generator Networks

“RAM: Dynamic Computational Time for Visual Attention”, Li et al 2017

RAM: Dynamic Computational Time for Visual Attention

“Hybrid Computing Using a Neural Network With Dynamic External Memory”, Graves et al 2016

Hybrid computing using a neural network with dynamic external memory

“Scaling Memory-Augmented Neural Networks With Sparse Reads and Writes”, Rae et al 2016

Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes

“Modeling Human Reading With Neural Attention”, Hahn & Keller 2016

Modeling Human Reading with Neural Attention

“Iterative Alternating Neural Attention for Machine Reading”, Sordoni et al 2016

Iterative Alternating Neural Attention for Machine Reading

“Adaptive Computation Time for Recurrent Neural Networks”, Graves 2016

Adaptive Computation Time for Recurrent Neural Networks

“Foveation-Based Mechanisms Alleviate Adversarial Examples”, Luo et al 2015

Foveation-based Mechanisms Alleviate Adversarial Examples

“Generating Images from Captions With Attention”, Mansimov et al 2015

Generating Images from Captions with Attention

“DRAW: A Recurrent Neural Network For Image Generation”, Gregor et al 2015

DRAW: A Recurrent Neural Network For Image Generation

“Neural Turing Machines”, Graves et al 2014

Neural Turing Machines

“Neural Machine Translation by Jointly Learning to Align and Translate”, Bahdanau et al 2014

Neural Machine Translation by Jointly Learning to Align and Translate

“On Learning Where To Look”, Ranzato 2014

On Learning Where To Look

“Generating Sequences With Recurrent Neural Networks”, Graves 2013

Generating Sequences With Recurrent Neural Networks

“Efficient Transformers: A Survey § Table 1”

Efficient Transformers: A Survey § Table 1⁠:

View PDF:

/doc/www/arxiv.org/4b545d261b12d222fe71f312cc32d13bfa3e25cc.pdf#org=google&page=6

“Attention and Augmented Recurrent Neural Networks”

Attention and Augmented Recurrent Neural Networks

“Hierarchical Object Detection With Deep Reinforcement Learning”

Hierarchical Object Detection with Deep Reinforcement Learning

“The Transformer Family: Attention and Self-Attention · Multi-Head Self-Attention · Transformer · Adaptive Computation Time (ACT) · Improved Attention Span: (Longer Attention Span (Transformer-XL) / Adaptive Attention Span / Localized Attention Span (Image Transformer)) · Less Time and Memory Cost: (Sparse Attention Matrix Factorization (Sparse Transformers) / Locality-Sensitive Hashing (Reformer)) · Make It Recurrent (Universal Transformer) · Stabilization for RL (GTrXL)”

The Transformer Family: Attention and Self-Attention · Multi-Head Self-Attention · Transformer · Adaptive Computation Time (ACT) · Improved Attention Span: (Longer Attention Span (Transformer-XL) / Adaptive Attention Span / Localized Attention Span (Image Transformer)) · Less Time and Memory Cost: (Sparse Attention Matrix Factorization (Sparse Transformers) / Locality-Sensitive Hashing (Reformer)) · Make it Recurrent (Universal Transformer) · Stabilization for RL (GTrXL)⁠:

View HTML:

/doc/www/lilianweng.github.io/8428b81f97ec6ee2f1a32d095f038a405e743f80.html#openai

“Learning to Combine Foveal Glimpses With a Third-Order Boltzmann Machine”

Learning to combine foveal glimpses with a third-order Boltzmann machine

“Show, Attend and Tell: Neural Image Caption Generation With Visual Attention”

Show, attend and tell: Neural image caption generation with visual attention

“Recurrent Models of Visual Attention”

Recurrent models of visual attention

“Can Active Memory Replace Attention?”

Can Active Memory Replace Attention?

“A Survey of Long-Term Context in Transformers: Sparse Transformers · Adaptive Span Transformers · Transformer-XL · Compressive Transformers · Reformer · Routing Transformer · Sinkhorn Transformer · Linformer · Efficient Attention: Attention With Linear Complexities · Transformers Are RNNs · ETC · Longformer”

A Survey of Long-Term Context in Transformers: Sparse Transformers · Adaptive Span Transformers · Transformer-XL · Compressive Transformers · Reformer · Routing Transformer · Sinkhorn Transformer · Linformer · Efficient Attention: Attention with Linear Complexities · Transformers are RNNs · ETC · Longformer

Annotations sorted by machine learning into inferred 'tags'. This provides an alternative way to browse: instead of by date order, one can browse in topic order. The 'sorted' list has been automatically clustered into multiple sections & auto-labeled for easier browsing.

Beginning with the newest annotation, it uses the embedding of each annotation to attempt to create a list of nearest-neighbor annotations, creating a progression of topics. For more details, see the link.