“EdiT5: Semi-Autoregressive Text-Editing With T5 Warm-Start”, Jonathan Mallinson, Jakub Adamek, Eric Malmi, Aliaksei Severyn2022-05-24 (T5 Transformer)⁠:

[code] We present EdiT5—a novel semi-autoregressive text-editing model designed to combine the strengths of non-autoregressive text-editing and autoregressive decoding. EdiT5 is faster during inference than conventional sequence-to-sequence (seq2seq) models, while being capable of modeling flexible input-output transformations.

This is achieved by decomposing the generation process into 3 sub-tasks: (1) tagging to decide on the subset of input tokens to be preserved in the output, (2) re-ordering [using pointer networks] to define their order in the output text, and (3) insertion to infill the missing tokens that are not present in the input. The tagging and re-ordering steps, which are responsible for generating the largest portion of the output, are non-autoregressive, while the insertion step uses an autoregressive decoder.

Depending on the task, EdiT5 on average requires fewer autoregressive steps, demonstrating speedups of up to 25× when compared to seq2seq models. Quality-wise, EdiT5 is initialized with a pre-trained T5 checkpoint yielding comparable performance to T5 in high-resource settings when evaluated on 3 NLG tasks: Sentence Fusion, Grammatical Error Correction, and Decontextualization while clearly outperforming T5 in low-resource settings.