“SequenceMatch: Imitation Learning for Autoregressive Sequence Modeling With Backtracking”, Chris Cundy, Stefano Ermon2023-06-08 (Transformer, imitation learning)⁠:

[cf. LaserTagger, PEER, Reid & Neubig2022] In many domains, autoregressive models can attain high likelihood on the task of predicting the next observation. However, this maximum-likelihood (MLE) objective does not necessarily match a downstream use-case of autoregressively generating high-quality sequences. The MLE objective weights sequences proportionally to their frequency under the data distribution, with no guidance for the model’s behavior out of distribution (OOD): leading to compounding error during autoregressive generation.

In order to address this compounding error problem, we formulate sequence generation as an imitation learning (IL) problem. This allows us to minimize a variety of divergences between the distribution of sequences generated by an autoregressive model and sequences from a dataset, including divergences with weight on OOD generated sequences. The IL framework also allows us to incorporate backtracking by introducing a backspace action into the generation process. This further mitigates the compounding error problem by allowing the model to revert a sampled token if it takes the sequence OOD.

Our resulting method, SequenceMatch, can be implemented without adversarial training or major architectural changes. We identify the SequenceMatch-χ² divergence as a more suitable training objective for autoregressive models which are used for generation.

We show that empirically, SequenceMatch training leads to improvements over MLE on text generation with language models.