“‘Sparse Transformers’ Tag”,2020-11-29 (NN sparsity)⁠:

Bibliography for tag ai/nn/transformer/attention/sparsity, most recent first: 2 related tags, 40 annotations, & 4 links (parent).

[page summary]

• See Also
• Links
  • “When Parts Are Greater Than Sums: Individual LLM Components Can Outperform Full Models”, Chang et al 2024
  • “AI Is a Black Box. Anthropic Figured Out a Way to Look Inside: What Goes on in Artificial Neural Networks Work Is Largely a Mystery, Even to Their Creators. But Researchers from Anthropic Have Caught a Glimpse”, Levy2024
  • “Revisiting the Equivalence of In-Context Learning and Gradient Descent: The Impact of Data Distribution”, Mahdavi et al 2024
  • “Zoology: Measuring and Improving Recall in Efficient Language Models”, Arora et al 2023
  • “HyperAttention: Long-Context Attention in Near-Linear Time”, Han et al 2023
  • “LongLoRA: Efficient Fine-Tuning of Long-Context Large Language Models”, Chen et al 2023
  • “H₂O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models”, Zhang et al 2023
  • “Unlimiformer: Long-Range Transformers With Unlimited Length Input”, Bertsch et al 2023
  • “How Much Does Attention Actually Attend? Questioning the Importance of Attention in Pretrained Transformers”, Hassid et al 2022
  • “Scaling Laws vs Model Architectures: How Does Inductive Bias Influence Scaling?”, Tay et al 2022
  • “Random Feature Attention”, Peng et al 2022
  • “Sparse Is Enough in Scaling Transformers”, Jaszczur et al 2021
  • “You Only Sample (Almost) Once: Linear Cost Self-Attention Via Bernoulli Sampling”, Zeng et al 2021
  • “Scatterbrain: Unifying Sparse and Low-Rank Attention Approximation”, Chen et al 2021
  • “Combiner: Full Attention Transformer With Sparse Computation Cost”, Ren et al 2021
  • “OmniNet: Omnidirectional Representations from Transformers”, Tay et al 2021
  • “Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention”, Xiong et al 2021
  • “Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting”, Zhou et al 2020
  • “SMYRF: Efficient Attention Using Asymmetric Clustering”, Daras et al 2020
  • “FAVOR+: Rethinking Attention With Performers”, Choromanski et al 2020
  • “Cluster-Former: Clustering-Based Sparse Transformer for Long-Range Dependency Encoding”, Wang et al 2020
  • “DeepSpeed Sparse Attention”, Team2020
  • “BigBird: Transformers for Longer Sequences”, Zaheer et al 2020
  • “Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation”, Wang et al 2020
  • “Efficient Content-Based Sparse Attention With Routing Transformers”, Roy et al 2020
  • “Sparse Sinkhorn Attention”, Tay et al 2020
  • “Reformer: The Efficient Transformer”, Kitaev et al 2020
  • “The Reformer—Pushing the Limits of Language Modeling”, Platen2020
  • “Axial Attention in Multidimensional Transformers”, Ho et al 2019
  • “Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting”, Li et al 2019
  • “Scaling Autoregressive Video Models”, Weissenborn et al 2019
  • “Adaptive Attention Span in Transformers”, Sukhbaatar et al 2019
  • “Generating Long Sequences With Sparse Transformers”, Child et al 2019
  • “Generative Modeling With Sparse Transformers: We’ve Developed the Sparse Transformer, a Deep Neural Network Which Sets New Records at Predicting What Comes next in a Sequence—Whether Text, Images, or Sound. It Uses an Algorithmic Improvement of the attention Mechanism to Extract Patterns from Sequences 30× Longer Than Possible Previously”, Child & Gray2019
  • “Star-Transformer”, Guo et al 2019
  • “CCNet: Criss-Cross Attention for Semantic Segmentation”, Huang et al 2018
  • “Image Transformer”, Parmar et al 2018
  • “Constructing Transformers For Longer Sequences With Sparse Attention Methods”
  • “A Deep Dive into the Reformer”
  • “Optimal Transport and the Sinkhorn Transformer”
  • Sort By Magic
    • sparse-transformer clustering efficiency inductive-bias long-range attentive-architectures
    • efficient-sequences
    • sparse-attention lightweight-attention efficient-transformers routing-attention long-context attention-optimization
    • image-transformer
• Miscellaneous
• Bibliography