“On Warm-Starting Neural Network Training”, 2019-10-18 ():
In many real-world deployments of machine learning systems, data arrive piecemeal. These learning scenarios may be passive, where data arrive incrementally due to structural properties of the problem (eg. daily financial data) or active, where samples are selected according to a measure of their quality (eg. experimental design). In both of these cases, we are building a sequence of models that incorporate an increasing amount of data. We would like each of these models in the sequence to be performant and take advantage of all the data that are available to that point.
Conventional intuition suggests that when solving a sequence of related optimization problems of this form, it should be possible to initialize using the solution of the previous iterate—to “warm start” the optimization rather than initialize from scratch—and see reductions in wall-clock time. However, in practice this warm-starting seems to yield poorer generalization performance than models that have fresh random initializations, even though the final training losses are similar. While it appears that some hyperparameter settings allow a practitioner to close this generalization gap, they seem to only do so in regimes that damage the wall-clock gains of the warm start.
Nevertheless, it is highly desirable to be able to warm-start neural network training, as it would dramatically reduce the resource usage associated with the construction of performant deep learning systems.
In this work, we take a closer look at this empirical phenomenon and try to understand when and how it occurs.
We also provide a surprisingly simple trick that overcomes this pathology in several important situations, and present experiments that elucidate some of its properties.
…The warm-start phenomenon has implications for other situations as well. In active learning, for example, unlabeled samples are abundant but labels are expensive: the goal is to identify maximally-informative data to have labeled by an oracle and integrated into the training set. It would be time efficient to simply warm-start optimization each time new samples are appended to the training set, but such an approach seems to damage generalization in deep neural networks. Although this phenomenon has not received much direct attention from the research community, it seems to be common practice in deep active learning to retrain from scratch after every query step;6, 7 popular deep active learning repositories on Github randomly reinitialize models after every selection.[8, 9]
View PDF: