“Were RNNs All We Needed?”, Leo Feng, Frederick Tung, Mohamed Osama Ahmed, Yoshua Bengio, Hossein Hajimirsadegh2024-10-02 (RNN, self-attention; similar)⁠:

The scalability limitations of Transformers regarding sequence length have renewed interest in recurrent sequence models that are parallelizable during training. As a result, many novel recurrent architectures, such as S4, Mamba, and Aaren [cf. QRNNs, xLSTM], have been proposed that achieve comparable performance.

In this work, we revisit traditional recurrent neural networks (RNNs) from over a decade ago: LSTMs (1997_27ya) and GRUs (2014). While these models were slow due to requiring backpropagation through time (BPTT), we show that by removing their hidden state dependencies from their input, forget, and update gates, LSTMs and GRUs no longer need to BPTT and can be efficiently trained in parallel.

Building on this, we introduce minimal versions (minLSTMs & minGRUs) that (1) use fewer parameters than their traditional counterparts and (2) are fully parallelizable during training (175× faster for a sequence of length 512).

Lastly, we show that these stripped-down versions of decade-old RNNs match the empirical performance of recent sequence models.

Similar Links: