[cf. et al 2020; talk] Language models (LMs) are powerful tools for natural language processing, but they often struggle to produce coherent and fluent text when they are small. Models with around 0.125b parameters such as GPT-Neo (small) or GPT-2 (small) can rarely generate coherent and consistent English text beyond a few words even after extensive training. This raises the question of whether the emergence of the ability to produce coherent English text only occurs at larger scales (with hundreds of millions of parameters or more) and complex architectures (with many layers of global attention).
In this work, we introduce TinyStories, a synthetic dataset of short stories that only contain words that a typical 3 to 4-year-olds usually understand, generated by GPT-3.5 and GPT-4. We show that TinyStories can be used to train and evaluate LMs that are much smaller than the state-of-the-art models (below 0.01b total parameters), or have much simpler architectures (with only 1 Transformer block), yet still produce fluent and consistent stories with several paragraphs that are diverse and have almost perfect grammar, and demonstrate reasoning capabilities.
We also introduce a new paradigm for the evaluation of language models: We suggest a framework which uses GPT-4 to grade the content generated by these models as if those were stories written by students and graded by a (human) teacher. This new paradigm overcomes the flaws of standard benchmarks which often requires the model’s output to be very structured, and moreover provides a multidimensional score for the model, providing scores for different capabilities such as grammar, creativity and consistency.
We hope that TinyStories can facilitate the development, analysis and research of LMs, especially for low-resource or specialized domains, and shed light on the emergence of language capabilities in LMs.
…However, it is currently not clear whether the inability of SLMs to produce coherent text is a result of the intrinsic complexity of natural language, or of the excessive breadth and diversity of the corpora used for training. When we train a model on Wikipedia, for example, we are not only teaching it how to speak English, but also how to encode and retrieve an immense amount of facts and concepts from various domains and disciplines. Is it possible that SLMs are overwhelmed by the amount and variety of information they have to process and store, and that this hinders their ability to learn the core mechanisms and principles of language?
This raises the question of whether we can design a dataset that preserves the essential elements of natural language, such as grammar, vocabulary, facts, and reasoning, but that is much smaller and more refined in terms of its breadth and diversity. Such a dataset would allow us to isolate and examine the minimal requirements for a language model to generate coherent and fluent text, and to evaluate its performance and capabilities more precisely and fairly. Moreover, such a dataset would facilitate the development and analysis of SLMs, especially for low-resource or specialized domains, where large and diverse corpora are not readily available or desirable.
…While each story consists of 2–3 paragraphs that follows a simple plot and a consistent theme, the entirety of the dataset aims to span the vocabulary and the factual knowledge base of a 3–4 year old child.
…Based on this dataset, our paper makes several main contributions:
-
Our main contribution is that we show TinyStories can be used to train and evaluate SLMs that are much smaller than the state-of-the-art models (below 0.01b parameters with an embedding dimension of 256), or have much simpler architectures (with only one Transformer block), yet still produce a diverse set of fluent and consistent stories that are comparable or superior to those generated by larger and more complex models. Moreover, despite of the small size of the models, we still observe an emergence of reasoning capabilities, knowledge of general facts and ability to follow certain instructions.
-
We show that although the training of SLMs on TinyStories can typically be done in less than a day on a single NVIDIA V100 GPU, it still exhibits many behaviors similar to the ones observed in LLMs, such as scaling laws, trade-offs between width and depth, etc. Even with limited computational resources, we are able to conduct extensive experiments to study the effects of different hyperparameters, architectures and training methods on the performance and quality of the models…To this end, we rely on the latest text generation models by OpenAI (GPT-3.5 and GPT-4) which are able to produce large amounts of synthetic content according to instructions. In particular, we instruct the models to produce content that only uses vocabulary that a typical 3-year-old child would understand. We restrict the content to have the format of short stories in English. The main challenge in using large language models for producing training data is generating a dataset that is sufficiently diverse: prompting those models to produce stories, even if the temperature of generation is set to a high value, will still produce a very repetitive dataset, whose diversity is very far from what is required for training a language model that has a comparable “understanding” of language to that of children.
In order to address the problem of diversity, we collected a vocabulary consisting of about 1500 basic words, which try to mimic the vocabulary of a typical 3–4 year-old child, separated into nouns, verbs, and adjectives. In each generation, 3 words are chosen randomly (one verb, one noun, and one adjective). The model is instructed to generate a story that somehow combines these random words into the story. As we argue below, this greatly increases the diversity of the dataset, forcing the stories to span the entire vocabulary a child is familiar with, and to include a rich set of ways to combine different concepts. In addition, we constructed a list of possible features a story could have (such as a dialogue, a plot twist, a bad ending or a moral value). For each story we generated a random subset of those features and prompted the model with the extra requirement for the story to have these features.
[This makes it a kind of “active learning”: the datapoints are different by construction, reducing the inherent redundancy of i.i.d. data, because it more evenly samples the domain and enables dense interpolation.]
Here is a sample generation, including the prompt and the story generated by GPT-3.5:
Write a short story (3–5 paragraphs) which only uses very simple words that a 3 year old child would likely understand. The story should use the verb “decorate”, the noun “thunder” and the adjective “ancient”. The story should have the following features: the story should contain at least one dialogue, the story has a bad ending. Remember to only use simple words!
Once upon a time, in an ancient house, there lived a girl named Lily. She loved to decorate her room with pretty things. One day, she found a big box in the attic. She opened it and saw many shiny decorations. Lily was very happy and decided to use them in her room. As Lily was decorating her room, the sky outside became dark. There was a loud thunder sound, and Lily got scared. She ran to her mom and said, “Mommy, the thunder is so loud!” Her mom hugged her and said, “Don’t worry, it will pass soon.” But the thunder did not stop. It got louder and louder, and the ancient house started to shake. Suddenly, the roof fell down on the pretty decorations. Lily was sad because her room was not pretty anymore. The end.
…Model size versus the training FLOPs: For a fixed amount of training flops, there is a trade-off between the size of the model and the number of training steps (the total number of flops is the product of both). Previous works16, 11 have shown that there is a polynomial scaling law between model size and learning budget for LLMs, ie. the optimal model size for a given amount of flops is proportional to the flops raised to some power α > 1. However, these works used different ranges of model sizes (from a few million to tens of billions of parameters) and found different values of α (~0.7 & 0.5, respectively). A natural question is whether this scaling law is universal or depends on the dataset. Our dataset allows us to conduct a similar experiment but with much smaller models and flops. Surprisingly, we find evidence for a polynomial scaling law as well, which suggests that there might be a universal phenomenon here.
We train models of various sizes and architectures on TinyStories. For each amount of flops, we select the model and the number of training steps that achieve the lowest validation loss among the possible combinations. We vary the number of layers from 2, 4, 8, 12 and the hidden dimension from 64, 128, 256, 512, 768, 1024, 2048. The result is shown in Figure 6. Although the number of points may be a bit small for the data to be very conclusive, the plot points to a polynomial dependence.
…While all the evaluation scores are consistently increasing with the decrease of evaluation loss, a more careful scrutiny of the results reveals the following:
- Figure 3 suggests that shallower models perform better in terms of grammar compared to content consistency, meaning that model depth is more important for keeping consistent with the content than for generating syntactically correct language (we provide additional evidence for this in the next section).
-
In the same figure, we observe that the score for grammar plateaus at an earlier stage than that other two scores. Furthermore, in Table 4, we also see that while grammar can be mastered by relatively small models, consistency and creativity only emerge at a larger size.
-
Table 4 further suggests that the ability to generate a completion that is consistent with the beginning of the story emerges when the hidden size of the model increases 64 → 128.
-
We also see that the largest model that we have trained on TinyStories (with roughly 0.08b parameters) reaches almost perfect scores in terms of grammar and consistency. However, it falls short of GPT-4’s abilities in terms of creativity quite substantially, suggesting that creativity continues to improve more substantially with the sizes of the model and dataset, compared to grammar and consistency.
-
The right-hand columns of Table 4 suggests that the models that have only 1 layer seem to struggle quite substantially with following instructions (which likely heavily relies on global attention), and 2 layers seem to be sufficient for a certain extent of instruction-following. Comparing the
InstructandPlotscores we also see that the quality of instruction-following depends more heavily on the number of layers, in comparison with the coherence of the plot for which the hidden dimension is more important.
…Throughout the section, we work with several architectures of models whose size ranges between roughly 1M and 35M parameters, and whose number of layers range 1–8 layers. All the models can be trained on a single V100 GPU within <30 hours.
…Our findings suggest that the attention heads exhibit diverse and meaningful functions, such as attending to the previous word, the subject of the sentence, the end of the sentence, or the main topic of the story. We also observe that some attention heads specialize in generating certain types of words, such as nouns, verbs, or punctuation. These results suggest that the attention heads learn to perform different linguistic tasks and capture different aspects of the stories…We find that some neurons are activated on words that have a specific role in the sentence (such as the subject or the action), or in the story (such as the introduction of the protagonist). These findings suggest that the neurons in the MLP learn to encode different semantic and stylistic information and influence the generation process.