“Drag Your GAN (DragGAN): Interactive Point-Based Manipulation on the Generative Image Manifold”, Xingang Pan, Ayush Tewari, Thomas Leimkühler, Lingjie Liu, Abhimitra Meka, Christian Theobalt2023-05-18 (StyleGAN)⁠:

[code; cf. Endo2022; videos] Synthesizing visual content that meets users’ needs often requires flexible and precise controllability of the pose, shape, expression, and layout of the generated objects. Existing approaches gain controllability of generative adversarial networks (GANs) via manually annotated training data or a prior 3D model, which often lack flexibility, precision, and generality.

In this work, we study a powerful yet much less explored way of controlling GANs, that is, to “drag” any points of the image to precisely reach target points in a user-interactive manner, as shown in Figure 1. To achieve this, we propose DragGAN, which consists of two main components: (1) a feature-based motion supervision that drives the handle point to move towards the target position, and (2) a new point tracking approach that leverages the discriminative generator features to keep localizing the position of the handle points…Our technique is built on the key insight that the feature space of a GAN is sufficiently discriminative to enable both motion supervision and precise point tracking. Specifically, the motion supervision is achieved via a shifted feature patch loss that optimizes the latent code. Each optimization step leads to the handle points shifting closer to the targets; thus point tracking is then performed through nearest neighbor search in the feature space. This optimization process is repeated until the handle points reach the targets.

…Through DragGAN, anyone can deform an image [in near real-time] with precise control over where pixels go, thus manipulating the pose, shape, expression, and layout of diverse categories such as animals, cars, humans, landscapes, etc. As these manipulations are performed on the learned generative image manifold of a GAN, they tend to produce realistic outputs even for challenging scenarios such as hallucinating occluded content and deforming shapes that consistently follow the object’s rigidity.

Both qualitative and quantitative comparisons demonstrate the advantage of DragGAN over prior approaches in the tasks of image manipulation and point tracking. We also showcase the manipulation of real images through GAN inversion.