“Small-GAN: Speeding Up GAN Training Using Core-Sets”, Samarth Sinha, Han Zhang, Anirudh Goyal, Yoshua Bengio, Hugo Larochelle, Augustus Odena2019-10-29 (GAN, active learning; backlinks)⁠:

Recent work by Brock et al 2018 suggests that Generative Adversarial Networks (GANs) benefit disproportionately from large mini-batch sizes. Unfortunately, using large batches is slow and expensive on conventional hardware. Thus, it would be nice if we could generate batches that were effectively large though actually small.

In this work, we propose a method to do this, inspired by the use of Coreset-selection in active learning.

When training a GAN, we draw a large batch of samples from the prior and then compress that batch using Coreset-selection. To create effectively large batches of ‘real’ images, we create a cached dataset of Inception CNN activations of each training image, randomly project them down to a smaller dimension, and then use Coreset-selection on those projected activations at training time.

We conduct experiments showing that this technique substantially reduces training time and memory usage for modern GAN variants, that it reduces the fraction of dropped modes in a synthetic dataset, and that it allows GANs to reach a new state-of-the-art in anomaly detection.

Backlinks: