“Learning to Compress Prompts With Gist Tokens”, Jesse Mu, Xiang Lisa Li, Noah Goodman2023-04-17 (compressed Transformers, T5 Transformer)⁠:

Prompting is now the primary way to use the multitask capabilities of language models (LMs), but prompts occupy valuable space in the input context window, and re-encoding the same prompt is computationally inefficient. Finetuning and distillation methods allow for specialization of LMs without prompting, but require retraining the model for each task.

To avoid this trade-off entirely, we present gisting, which trains an LM to compress prompts into smaller sets of “gist” tokens which can be reused for compute efficiency. Gist models can be easily trained as part of instruction finetuning via a restricted attention mask that encourages prompt compression.

On decoder (LLaMa-7B) and encoder-decoder (FLAN-T5-XXL) LMs, gisting enables up to 26× compression of prompts, resulting in up to 40% FLOPs reductions, 4.2% wall time speedups, storage savings, and minimal loss in output quality.