“Extrapolating from a Single Image to a Thousand Classes Using Distillation”, Yuki M. Asano, Aaqib Saeed2021-12-01 (knowledge distillation; similar)⁠:

What can neural networks learn about the visual world from a single image? While it obviously cannot contain the multitudes of possible objects, scenes and lighting conditions that exist—within the space of all possible 256^3×224×224 224px-sized square images, it might still provide a strong prior for natural images.

To analyze this hypothesis, we develop a framework for training neural networks from scratch using a single image by means of knowledge distillation from a supervised pretrained teacher.

With this, we find that the answer to the above question is: ‘surprisingly, a lot’. In quantitative terms, we find top-1 accuracies of 94%/74% on CIFAR-10/100, 59% on ImageNet and, by extending this method to audio, 84% on SpeechCommands.

In extensive analyses we disentangle the effect of augmentations, choice of source image and network architectures and also discover “panda neurons” in networks that have never seen a panda.

This work shows that one image can be used to extrapolate to thousands of object classes and motivates a renewed research agenda on the fundamental interplay of augmentations and image.

Similar Links: