“‘Transformer Matrix Optimizations’ Tag”,2021-04-17 (math)⁠:

Bibliography for tag ai/nn/transformer/attention/linear-algebra, most recent first: 18 annotations & 2 links (parent).

• See Also
• Links
  • “LoLCATs: On Low-Rank Linearizing of Large Language Models”, Zhang et al 2024
  • “SANA: Efficient High-Resolution Image Synthesis With Linear Diffusion Transformers”, Xie et al 2024
  • “Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformers”, Gu et al 2024
  • “RWKV: Reinventing RNNs for the Transformer Era”, Peng et al 2023
  • “CosFormer: Rethinking Softmax in Attention”, Qin et al 2022
  • “Self-Attention Does Not Need 𝒪(n²) Memory”, Rabe & Staats2021
  • “Skyformer: Remodel Self-Attention With Gaussian Kernel and Nyström Method”, Chen et al 2021
  • “On Learning the Transformer Kernel”, Chowdhury et al 2021
  • “A Dot Product Attention Free Transformer”, Zhai et al 2021
  • “Luna: Linear Unified Nested Attention”, Ma et al 2021
  • “Beyond Self-Attention: External Attention Using Two Linear Layers for Visual Tasks (EAMLP)”, Guo et al 2021
  • “Sub-Linear Memory: How to Make Performers SLiM”, Likhosherstov et al 2020
  • “AFT: An Attention Free Transformer”, Anonymous2020
  • “LambdaNetworks: Modeling Long-Range Interactions without Attention”, Bello2020
  • “Transformers Are RNNs: Fast Autoregressive Transformers With Linear Attention”, Katharopoulos et al 2020
  • “Linformer: Self-Attention With Linear Complexity”, Wang et al 2020
  • “Efficient Attention: Attention With Linear Complexities”, Shen et al 2018
  • “Efficient Attention: Attention With Linear Complexities [Blog]”
  • Sort By Magic
    • efficient-attention
    • attention-free
    • linear-attention
• Miscellaneous
• Bibliography