“Blockwise Parallel Transformer for Long Context Large Models”, Hao Liu, Pieter Abbeel2023-05-30 (self-attention, algorithms)⁠:

Transformers have emerged as the cornerstone of state-of-the-art natural language processing models, showcasing exceptional performance across a wide range of AI applications. However, the memory demands posed by the self-attention mechanism and the large feedforward network in Transformers limit their ability to handle long sequences, thereby creating challenges for tasks involving multiple long sequences or long-term dependencies.

We present a distinct approach, Blockwise Parallel Transformer (BPT), that leverages blockwise computation of self-attention and feedforward network fusion to minimize memory costs.

By processing longer input sequences while maintaining memory efficiency, BPT enables training sequences up to 32× longer than vanilla Transformers and 2 to 4× longer than previous memory-efficient methods. Extensive experiments on language modeling and reinforcement learning tasks demonstrate the effectiveness of BPT in reducing memory requirements and improving performance.