“Semantic Projection Recovers Rich Human Knowledge of Multiple Object Features from Word Embeddings”, Gabriel Grand, Idan Asher Blank, Francisco Pereira, Evelina Fedorenko2022-04-14 (RNN, language)⁠:

How is knowledge about word meaning represented in the mental lexicon? Current computational models infer word meanings from lexical co-occurrence patterns. They learn to represent words as vectors in a multidimensional space, wherein words that are used in more similar linguistic contexts—that is, are more semantically related—are located closer together.

However, whereas inter-word proximity captures only overall relatedness, human judgements are highly context dependent. For example, dolphins and alligators are similar in size but differ in danger.

Here, we use a domain-general method to extract context-dependent relationships from word embeddings [GloVe]: ‘semantic projection’ of word-vectors onto lines that represent features such as size (the line connecting the words ‘small’ and ‘big’) or danger (‘safe’ to ‘dangerous’), analogous to ‘mental scales’. This method recovers human judgements across various object categories and properties.

Thus, the geometry of word embeddings explicitly represents a wealth of context-dependent world knowledge.

See Also:

• Geometry of abstract learned knowledge in the hippocampus

• The neural architecture of language: Integrative reverse-engineering converges on a model for predictive processing

• A mathematical theory of semantic development in deep neural networks

• Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color

• Implicit Representations of Meaning in Neural Language Models

• Discovering objects and their relations from entangled scene representations

• Shared computational principles for language processing in humans and deep language models

• Deep contextualized word representations