“CosFormer: Rethinking Softmax in Attention”, Qin Et Al 2022

“cosFormer: Rethinking Softmax in Attention”⁠, Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, Yiran Zhong2022-02-17 (similar)

“Self-attention Does Not Need 𝒪 (N²) Memory”, Rabe & Staats2021

“Self-attention Does Not Need 𝒪 (n²) Memory”⁠, Markus N. Rabe, Charles Staats2021-12-10 (similar)

“Skyformer: Remodel Self-Attention With Gaussian Kernel and Nyström Method”, Chen Et Al 2021

“Skyformer: Remodel Self-Attention with Gaussian Kernel and Nyström Method”⁠, Yifan Chen, Qi Zeng, Heng Ji, Yun Yang2021-10-29 (backlinks; similar)

“On Learning the Transformer Kernel”, Chowdhury Et Al 2021

“On Learning the Transformer Kernel”⁠, Sankalan Pal Chowdhury, Adamos Solomou, Avinava Dubey, Mrinmaya Sachan2021-10-15 (backlinks; similar)

“A Dot Product Attention Free Transformer”, Zhai Et Al 2021

“A Dot Product Attention Free Transformer”⁠, Shuangfei Zhai, Walter Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Ruixiang ZHANG, Joshua M. Susskind2021-10-05 (backlinks; similar)

“Luna: Linear Unified Nested Attention”, Ma Et Al 2021

“Luna: Linear Unified Nested Attention”⁠, Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, Luke Zettlemoyer2021-06-03 (backlinks; similar)

“Beyond Self-attention: External Attention Using Two Linear Layers for Visual Tasks (EAMLP)”, Guo Et Al 2021

“Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks (EAMLP)”⁠, Meng-Hao Guo, Zheng-Ning Liu, Tai-Jiang Mu, Shi-Min Hu2021-05-05 (backlinks; similar)

“Sub-Linear Memory: How to Make Performers SLiM”, Likhosherstov Et Al 2020

“Sub-Linear Memory: How to Make Performers SLiM”⁠, Valerii Likhosherstov, Krzysztof Choromanski, Jared Davis, Xingyou Song, Adrian Weller2020-12-21 (backlinks; similar)

“LambdaNetworks: Modeling Long-range Interactions without Attention”, Bello2020

“LambdaNetworks: Modeling long-range Interactions without Attention”⁠, Irwan Bello2020-09-28 (backlinks; similar)

“AFT: An Attention Free Transformer”, Anonymous2020

“AFT: An Attention Free Transformer”⁠, Anonymous2020-09-28 (MLP NN⁠, RNN; similar)

“Transformers Are RNNs: Fast Autoregressive Transformers With Linear Attention”, Katharopoulos Et Al 2020

“Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention”⁠, Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, François Fleuret2020-06-29 (RNN⁠, GPT; backlinks; similar)

“Linformer: Self-Attention With Linear Complexity”, Wang Et Al 2020

“Linformer: Self-Attention with Linear Complexity”⁠, Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, Hao Ma2020-06-08 (MLP NN; similar)

“Efficient Attention: Attention With Linear Complexities”, Shen Et Al 2018

“Efficient Attention: Attention with Linear Complexities”⁠, Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, Hongsheng Li2018-12-04 (similar)

“Efficient Attention: Attention With Linear Complexities [blog]”