Bibliography:

  1. ​ ‘self-attention’ tag

  2. ​ LoLCATs: On Low-Rank Linearizing of Large Language Models

  3. ​ SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers

  4. ​ Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformers

  5. ​ RWKV: Reinventing RNNs for the Transformer Era

  6. ​ cosFormer: Rethinking Softmax in Attention

  7. ​ Self-attention Does Not Need 𝒪(n2) Memory

  8. ​ Skyformer: Remodel Self-Attention with Gaussian Kernel and Nyström Method

  9. ​ On Learning the Transformer Kernel

  10. ​ A Dot Product Attention Free Transformer

  11. ​ Luna: Linear Unified Nested Attention

  12. ​ Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks (EAMLP)

  13. ​ Sub-Linear Memory: How to Make Performers SLiM

  14. ​ AFT: An Attention Free Transformer

  15. ​ LambdaNetworks: Modeling long-range Interactions without Attention

  16. ​ Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

  17. ​ Linformer: Self-Attention with Linear Complexity

  18. ​ Efficient Attention: Attention with Linear Complexities

  19. ​ Efficient Attention: Attention With Linear Complexities [Blog]

  20. ​ design#future-tag-features

    [Transclude the forward-link's context]

  21. ​ https://github.com/idiap/fast-transformers

  22. ​ https://manifestai.com/blogposts/faster-after-all/

  23. ​ eaed3ff82cffc0550ba85e7c9c89febbdd547d4e.html

  24. ​ SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers

  25. ​ https%253A%252F%252Farxiv.org%252Fabs%252F2410.10629%2523nvidia.html

  26. ​ RWKV: Reinventing RNNs for the Transformer Era

  27. ​ https%253A%252F%252Farxiv.org%252Fabs%252F2305.13048.html

  28. ​ A Dot Product Attention Free Transformer

  29. ​ https%253A%252F%252Fopenreview.net%252Fforum%253Fid%253DJVR4JswsEM.html

  30. ​ Efficient Attention: Attention with Linear Complexities

  31. ​ https%253A%252F%252Farxiv.org%252Fabs%252F1812.01243%2523sensetime.html

  32. Wikipedia Bibliography:

    1. ​ Bo Peng

    2. ​ Michael Chung

    3. ​ Peng Zhou