“‘Recurrent Transformers’ Tag”,2020-11-09 (RNN, compressed Transformers; backlinks)⁠:

Bibliography for tag ai/nn/transformer/attention/recurrent, most recent first: 1 related tag, 30 annotations, & 4 links (parent).

• See Also
• Links
  • “RecurrentGemma: Moving Past Transformers for Efficient Open Language Models”, Botev et al 2024
  • “Revisiting Dynamic Evaluation: Online Adaptation for Large Language Models”, Rannen-Triki et al 2024
  • “Transformers Are Multi-State RNNs”, Oren et al 2024
  • “Think Before You Speak: Training Language Models With Pause Tokens”, Goyal et al 2023
  • “Retentive Network: A Successor to Transformer for Large Language Models”, Sun et al 2023
  • “Block-State Transformers”, Fathi et al 2023
  • “Looped Transformers As Programmable Computers”, Giannou et al 2023
  • “FWL: Meta-Learning Fast Weight Language Models”, Clark et al 2022
  • “Fine-Tuning Pre-Trained Transformers into Decaying Fast Weights”, Mao2022
  • “Simple Recurrence Improves Masked Language Models”, Lei et al 2022
  • “Block-Recurrent Transformers”, Hutchins et al 2022
  • “S4: Efficiently Modeling Long Sequences With Structured State Spaces”, Gu et al 2021
  • “LSSL: Combining Recurrent, Convolutional, and Continuous-Time Models With Linear State-Space Layers”, Gu et al 2021
  • “Do Long-Range Language Models Actually Use Long-Range Context?”, Sun et al 2021
  • “Finetuning Pretrained Transformers into RNNs”, Kasai et al 2021
  • “When Attention Meets Fast Recurrence: Training SRU++ Language Models With Reduced Compute”, Lei2021
  • “Mind the Gap: Assessing Temporal Generalization in Neural Language Models § Dynamic Evaluation”, Lazaridou et al 2021 (page 7 org deepmind)
  • “Shortformer: Better Language Modeling Using Shorter Inputs”, Press et al 2020
  • “Untangling Tradeoffs between Recurrence and Self-Attention in Neural Networks”, Kerg et al 2020
  • “Addressing Some Limitations of Transformers With Feedback Memory”, Fan et al 2020
  • “DEQ: Deep Equilibrium Models”, Bai et al 2019
  • “XLNet: Generalized Autoregressive Pretraining for Language Understanding”, Yang et al 2019
  • “Dynamic Evaluation of Transformer Language Models”, Krause et al 2019
  • “Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context”, Dai et al 2019
  • “Transformer-XL—Combining Transformers and RNNs Into a State-Of-The-Art Language Model”, Horev2019
  • “Universal Transformers”, Dehghani et al 2018
  • “Hyperbolic Attention Networks”, Gulcehre et al 2018
  • “Improving Neural Language Models With a Continuous Cache”, Grave et al 2016
  • “Context Caching”
  • joeddav
  • Sort By Magic
    • efficient-modeling
    • long-range-context
    • language-modeling
• Miscellaneous
• Bibliography

Backlinks: