“The Structure of the Token Space for Large Language Models”, 2024-10-11 (; similar):
[cf. glitch tokens] Large language models encode the correlational structure present in natural language by fitting segments of utterances (tokens) into a high-dimensional ambient latent space upon which the models then operate. We assert that in order to develop a foundational, first-principles understanding of the behavior and limitations of large language models, it is crucial to understand the topological and geometric structure of this token subspace.
In this article, we present estimators for the dimension and Ricci scalar curvature of the token subspace, and apply it to 3 open-source large language models of moderate size: GPT-2, LLEMMA-7B, and Mistral-7B. In all 3 models, using these measurements, we find that the token subspace is not a manifold, but is instead a stratified manifold, where on each of the individual strata, the Ricci curvature is negative.
We additionally find that the dimension and curvature correlate with the generative fluency of the models, which suggests that these findings have implications for model behavior.
…1.3 Implications: If the token subspace is not a manifold, this has important implications because the behavior of the transformer blocks, which are piecewise smooth (hence continuous) transformations of the latent space18, must therefore preserve the dimensions we observe. As a result, queries that cross stratification boundaries will yield responses that exhibit dramatic changes in behavior.
This instability will likely preclude strong guarantees about the model’s generative performance without intimate knowledge of how the token subspace is embedded within the ambient latent space.