“‘Multi-Scale Transformers’ Tag”,2021-03-01 (CNN)⁠:

Bibliography for tag ai/nn/transformer/attention/hierarchical, most recent first: 61 annotations & 4 links (parent).

• See Also
• Gwern
  • “Fully-Connected Neural Nets”, Gwern2021
• Links
  • “State-Space Models Can Learn In-Context by Gradient Descent”, Sushma et al 2024
  • “XT: Nested Tokenization for Larger Context in Large Images”, Gupta et al 2024
  • “A Long-Context Language Model for the Generation of Bacteriophage Genomes”, Shao2023
  • “HGRN: Hierarchically Gated Recurrent Neural Network for Sequence Modeling”, Qin et al 2023
  • “Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layer”, Zhang et al 2023
  • “LongNet: Scaling Transformers to 1,000,000,000 Tokens”, Ding et al 2023
  • “Bytes Are All You Need: Transformers Operating Directly On File Bytes”, Horton et al 2023
  • “Landmark Attention: Random-Access Infinite Context Length for Transformers”, Mohtashami & Jaggi2023
  • “MEGABYTE: Predicting Million-Byte Sequences With Multiscale Transformers”, Yu et al 2023
  • “Parallel Context Windows Improve In-Context Learning of Large Language Models”, Ratner et al 2022
  • “Structured Prompting: Scaling In-Context Learning to 1,000 Examples”, Hao et al 2022
  • “Efficient Transformers With Dynamic Token Pooling”, Nawrot et al 2022
  • “Accurate Image Restoration With Attention Retractable Transformer (ART)”, Zhang et al 2022
  • “Co-Writing Screenplays and Theatre Scripts With Language Models (Dramatron): An Evaluation by Industry Professionals”, Mirowski et al 2022
  • “DiNAT: Dilated Neighborhood Attention Transformer”, Hassani & Shi2022
  • “Mega: Moving Average Equipped Gated Attention”, Ma et al 2022
  • “Investigating Efficiently Extending Transformers for Long Input Summarization”, Phang et al 2022
  • “ChordMixer: A Scalable Neural Attention Model for Sequences With Different Lengths”, Khalitov et al 2022
  • “Paramixer: Parameterizing Mixing Links in Sparse Factors Works Better Than Dot-Product Self-Attention”, Yu et al 2022
  • “NAT: Neighborhood Attention Transformer”, Hassani et al 2022
  • “ViS4mer: Long Movie Clip Classification With State-Space Video Models”, Islam & Bertasius2022
  • “MaxViT: Multi-Axis Vision Transformer”, Tu et al 2022
  • “Hierarchical Perceiver”, Carreira et al 2022
  • “Transformer Quality in Linear Time”, Hua et al 2022
  • “LongT5: Efficient Text-To-Text Transformer for Long Sequences”, Guo et al 2021
  • “Simple Local Attentions Remain Competitive for Long-Context Tasks”, Xiong et al 2021
  • “Restormer: Efficient Transformer for High-Resolution Image Restoration”, Zamir et al 2021
  • “Swin Transformer V2: Scaling Up Capacity and Resolution”, Liu et al 2021
  • “Hourglass: Hierarchical Transformers Are More Efficient Language Models”, Nawrot et al 2021
  • “Fastformer: Additive Attention Can Be All You Need”, Wu et al 2021
  • “AdaMRA: Adaptive Multi-Resolution Attention With Linear Complexity”, Zhang et al 2021
  • “Long-Short Transformer (Transformer-LS): Efficient Transformers for Language and Vision”, Zhu et al 2021
  • “Global Filter Networks for Image Classification”, Rao et al 2021
  • “HiT: Improved Transformer for High-Resolution GANs”, Zhao et al 2021
  • “A Multi-Level Attention Model for Evidence-Based Fact Checking”, Kruengkrai et al 2021
  • “Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document Modeling”, Wu et al 2021
  • “Aggregating Nested Transformers”, Zhang et al 2021
  • “Pay Attention to MLPs”, Liu et al 2021
  • “MViT: Multiscale Vision Transformers”, Fan et al 2021
  • “Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows”, Liu et al 2021
  • “Coordination Among Neural Modules Through a Shared Global Workspace”, Goyal et al 2021
  • “Generative Adversarial Transformers”, Hudson & Zitnick2021
  • “LazyFormer: Self Attention With Lazy Update”, Ying et al 2021
  • “CDLM: Cross-Document Language Modeling”, Caciularu et al 2021
  • “Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition”, Zhang et al 2020
  • “Summarize, Outline, and Elaborate: Long-Text Generation via Hierarchical Supervision from Extractive Summaries”, Sun et al 2020
  • “Transformer-QL: A Step Towards Making Transformer Network Quadratically Large”, Hajra2020
  • “Adding Recurrence to Pretrained Transformers for Improved Efficiency and Context Size”, Yoshida et al 2020
  • “Progressive Generation of Long Text”, Tan et al 2020
  • “Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing”, Dai et al 2020
  • “Conformer: Convolution-Augmented Transformer for Speech Recognition”, Gulati et al 2020
  • “Multi-Scale Transformer Language Models”, Subramanian et al 2020
  • “Beyond 512 Tokens: Siamese Multi-Depth Transformer-Based Hierarchical Encoder for Long-Form Document Matching”, Yang et al 2020
  • “Lite Transformer With Long-Short Range Attention”, Wu et al 2020
  • “ETC: Encoding Long and Structured Inputs in Transformers”, Ainslie et al 2020
  • “Longformer: The Long-Document Transformer”, Beltagy et al 2020
  • “BP-Transformer: Modeling Long-Range Context via Binary Partitioning”, Ye et al 2019
  • “Blockwise Self-Attention for Long Document Understanding”, Qiu et al 2019
  • “Hierarchical Transformers for Multi-Document Summarization”, Liu & Lapata2019
  • “Hierarchical Multiscale Recurrent Neural Networks”, Chung et al 2016
  • “A Clockwork RNN”, Koutník et al 2014
• Miscellaneous
• Bibliography