“Fine-Tuning Pre-trained Transformers into Decaying Fast Weights”, Mao2022

“Fine-Tuning Pre-trained Transformers into Decaying Fast Weights”⁠, Huanru Henry Mao2022-10-09 (GPT; similar; bibliography)

“Simple Recurrence Improves Masked Language Models”, Lei Et Al 2022

“Simple Recurrence Improves Masked Language Models”⁠, Tao Lei, Ran Tian, Jasmijn Bastings, Ankur P. Parikh2022-05-23 (RNN; similar)

“Block-Recurrent Transformers”, Hutchins Et Al 2022

“Block-Recurrent Transformers”⁠, DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, Behnam Neyshabur2022-03-11 (RNN; backlinks; similar; bibliography)

“S4: Efficiently Modeling Long Sequences With Structured State Spaces”, Gu Et Al 2021

“S4: Efficiently Modeling Long Sequences with Structured State Spaces”⁠, Albert Gu, Karan Goel, Christopher Ré2021-10-31 (RNN; backlinks; similar; bibliography)

“LSSL: Combining Recurrent, Convolutional, and Continuous-time Models With Linear State-Space Layers”, Gu Et Al 2021

“LSSL: Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers”⁠, Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, Christopher Ré2021-10-26 (RNN; backlinks; similar)

“Do Long-Range Language Models Actually Use Long-Range Context?”, Sun Et Al 2021

“Do Long-Range Language Models Actually Use Long-Range Context?”⁠, Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Micke, Mohit Iyyer2021-09-19 (similar; bibliography)

“Finetuning Pretrained Transformers into RNNs”, Kasai Et Al 2021

“Finetuning Pretrained Transformers into RNNs”⁠, Jungo Kasai, Hao Peng, Yizhe Zhang, Dani Yogatama, Gabriel Ilharco, Nikolaos Pappas, Yi Mao, Weizhu Chen2021-03-24 (RNN; backlinks; similar)

“When Attention Meets Fast Recurrence: Training SRU++ Language Models With Reduced Compute”, Lei2021

“When Attention Meets Fast Recurrence: Training SRU++ Language Models with Reduced Compute”⁠, Tao Lei2021-02-24 (RNN; backlinks; similar)

“Shortformer: Better Language Modeling Using Shorter Inputs”, Press Et Al 2020

“Shortformer: Better Language Modeling using Shorter Inputs”⁠, Ofir Press, Noah A. Smith, Mike Lewis2020-12-31 (backlinks; similar)

“Untangling Tradeoffs between Recurrence and Self-attention in Neural Networks”, Kerg Et Al 2020

“Untangling tradeoffs between recurrence and self-attention in neural networks”⁠, Giancarlo Kerg, Bhargav Kanuparthi, Anirudh Goyal, Kyle Goyette, Yoshua Bengio, Guillaume Lajoie2020-06-16 (RNN; backlinks; similar)

“Addressing Some Limitations of Transformers With Feedback Memory”, Fan Et Al 2020

“Addressing Some Limitations of Transformers with Feedback Memory”⁠, Angela Fan, Thibaut Lavril, Edouard Grave, Arm, Joulin, Sainbayar Sukhbaatar2020-02-21 (similar)

“DEQ: Deep Equilibrium Models”, Bai Et Al 2019

“DEQ: Deep Equilibrium Models”⁠, Shaojie Bai, J. Zico Kolter, Vladlen Koltun2019-09-03 (backlinks; similar)

“XLNet: Generalized Autoregressive Pretraining for Language Understanding”, Yang Et Al 2019

“XLNet: Generalized Autoregressive Pretraining for Language Understanding”⁠, Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le2019-06-19 (RNN; backlinks; similar)

“Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context”, Dai Et Al 2019

“Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context”⁠, Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov2019-01-09 (RNN⁠, GPT; backlinks; similar)

“Transformer-XL—Combining Transformers and RNNs Into a State-of-the-art Language Model”, Horev2019

“Universal Transformers”, Dehghani Et Al 2018

“Universal Transformers”⁠, Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, Łukasz Kaiser2018-07-10 (RNN⁠, GPT; similar)

“So I Tried out GPT-3’s Trick of Conditioning on Training Data With XLNet. While It Doesn’t Do as well as the 175B GPT-3, It Does Much Better Than the Version Which Is the Same Size As XLNet (0.4B). The Visual below Is from Their Paper on WinoGrande—I Added the Squares for XLNet.”