[Affiliations: Tsinghua University, DAMO Academy, AliBaba Group, BAAI; CogView2] Text-to-Image generation in the general domain has long been an open problem, which requires both a powerful generative model and cross-modal understanding.
We propose CogView, a 4-billion-parameter Transformer with VQ-VAE tokenizer to advance this problem. We also demonstrate the finetuning strategies for various downstream tasks, eg. style learning, super-resolution, text-image ranking and fashion design, and methods to stabilize pretraining, eg. eliminating NaN losses.
CogView (zero-shot) achieves a new state-of-the-art FID on blurred MS-COCO, outperforms previous GAN-based models and the recent similar work DALL·E 1. [cf.: âM6: A Chinese Multimodal Pretrainerâ, et al 2021; screenshots; future model release homepage; WuDaoMM dataset (also for Wenlan); live demo.]
âŠ3.3 Image Captioning and Self-reranking: To finetune CogView for image captioning is straightforward: exchanging the order of text and image tokens in the input sequences [and then finetune-train the model on the new flipped data training corpus ie âanalysis by synthesisâ]. Since the model has already learnt the corresponding relationships between text and images, reversing the generation is not hard. We did not evaluate the performance due to that (1) there is no authoritative Chinese image captioning benchmark (2) image captioning is not the focus of this work. The main purpose of finetuning such a model is for self-reranking. We propose the Caption Score (CapS) to evaluate the correspondence between images and textâŠthis method can be seen as an adaptation of inverse prompting53 for text-to-image generation. Finally, images with the highest CapS are chosen. [cf. self-distillation, CLIP ranking]
