“A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity”, Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K. Kummerfeld, Rada Mihalcea2024-01-03 (GPT-2, preference learning)⁠:

While alignment algorithms are now commonly used to tune pre-trained language models towards a user’s preferences, we lack explanations for the underlying mechanisms in which models become “aligned”, thus making it difficult to explain phenomena like jailbreaks.

In this work we study a popular algorithm, direct preference optimization (DPO), and the mechanisms by which it reduces toxicity. Namely, we first study how toxicity is represented and elicited in a pre-trained language model, GPT-2-medium. We then apply DPO with a carefully crafted pairwise dataset to reduce toxicity.

We examine how the resulting model averts toxic outputs, and find that capabilities learned from pre-training are not removed, but rather bypassed.

We use this insight to demonstrate a simple method to un-align the model, reverting it back to its toxic behavior.