“Two-Step Training: Adjustable Sketch Colorization via Reference Image and Text Tag”, Dingkun Yan, Ryogo Ito, Ryo Moriai, Suguru Saito2023-04-05 (Danbooru AI, MLP NN, GAN)⁠:

Automatic sketch colorization is a highly interesting topic in the image-generation field. However, due to the absence of texture in sketch images and the lack of training data, existing reference-based methods are ineffective in generating visually pleasant results and cannot edit the colors using text tags.

Thus, this paper presents a conditional generative adversarial network (cGAN)-based architecture with a pre-trained convolutional neural network (CNN), reference-based channel-wise attention (RBCA) and self-adaptive multi-layer perceptron (MLP) to tackle this problem. We propose two-step training and spatial latent manipulation to achieve high-quality and color-adjustable results using reference images and text tags.

The superiority of our approach in reference-based colorization is demonstrated through qualitative/quantitative comparisons and user studies with existing network-based methods. We also validate the controllability of the proposed model and discuss the details of our latent manipulation on the basis of experimental results of multi-label manipulation.