āDALLĀ·E 2 Is Seeing Double: Flaws in Word-To-Concept Mapping in Text2Image Modelsā, 2022-10-19 (; similar)ā :
[see BPEs/CLIP] We study the way DALLĀ·E 2 maps symbols (words) in the prompt to their references (entities or properties of entities in the generated image).
We show that in stark contrast to the way humans process language, DALLĀ·E 2 does not follow the constraint that each word has a single role in the interpretation, and sometimes reuses the same symbol for different purposes.
We collect a set of stimuli that reflect this phenomenon: we show that DALLĀ·E 2 depicts both senses of nouns with multiple senses at once; and that a given word can modify the properties of two distinct entities in the image, or can be depicted as one object while also modifying the properties of another object, creating a semantic leakage of properties between entities.
Taken together, our study highlights the differences between DALLĀ·E 2 and human language processing and opens an avenue for future study on the inductive biases of text-to-image models.