“TRACT: Denoising Diffusion Models With Transitive Closure Time-Distillation”, David Berthelot, Arnaud Autef, Jierui Lin, Dian Ang Yap, Shuangfei Zhai, Siyuan Hu, Daniel Zheng, Walter Talbott, Eric Gu2023-03-07 (diffusion model, knowledge distillation; backlinks)⁠:

Denoising Diffusion models have demonstrated their proficiency for generative sampling. However, generating good samples often requires many iterations. Consequently, techniques such as binary time-distillation (BTD) have been proposed to reduce the number of network calls for a fixed architecture.

In this paper, we introduce TRAnsitive Closure Time-distillation (TRACT), a new method that extends BTD. For single step diffusion, TRACT improves FID by up to 2.4× on the same architecture, and achieves new single-step Denoising Diffusion Implicit Models (DDIM) state-of-the-art FID (7.4 for ImageNet64, 3.8 for CIFAR-10). Finally we tease apart the method through extended ablations.

The PyTorch implementation will be released soon.

Backlinks: