We study empirical scaling laws for transfer learning between distributions in an unsupervised, fine-tuning setting. When we train increasingly large neural networks from-scratch on a fixed-size dataset, they eventually become data-limited and stop improving in performance (cross-entropy loss). When we do the same for models pre-trained on a large language dataset, the slope in performance gains is merely reduced rather than going to zero.
We calculate the effective data “transferred” from pre-training by determining how much data a transformer of the same size would have required to achieve the same loss when training from scratch. In other words, we focus on units of data while holding everything else fixed. We find that the effective data transferred is described well in the low data regime by a power-law of parameter count and fine-tuning dataset size.
We believe the exponents in these power-laws correspond to measures of the generality of a model and proximity of distributions (in a directed rather than symmetric sense). We find that pre-training effectively multiplies the fine-tuning dataset size. Transfer, like overall performance, scales predictably in terms of parameters, data, and compute.
…The effective data transferred is well-described by a power-law in the low-data regime: We use DT to represent the effective data transferred, ie. the amount of additional python data that a model of the same size trained on only python would have needed to achieve the same loss on python as a model pre-trained on language. Our notation is indicated visually in Figure 1. The scaling law for transfer in equation 1.1 is at the core of many key insights and predictions in this work. We find the simplicity of this result very intriguing:
DT = effective data transferred = k(DF)α(N)β
where N is the number of non-embedding model parameters, and DF is the size of the fine-tuning data distribution.
