Convolutional networks trained on large supervised dataset produce visual features which form the basis for the state-of-the-art in many computer-vision problems. Further improvements of these visual features will likely require even larger manually labeled data sets, which severely limits the pace at which progress can be made. In this paper, we explore the potential of leveraging massive, weakly-labeled image collections for learning good visual features. We train convolutional networks on a dataset of 100 million Flickr photos & captions, and show that these networks produce features that perform well in a range of vision problems. We also show that the networks appropriately capture word similarity, and learn correspondences between different languages.
…AlexNet takes up to 2 weeks to train on a setup with 4 GPUs, while training a GoogLeNet takes up to 3 weeks [ie. just <84 GPU-days].
…To investigate the performance of our models as a function of the amount of training data, we also performed experiments in which we varied the Flickr training set size. The left-hand side of Figure 2 presents the resulting learning curves for the AlexNet architecture with K = 1,000. The figure shows that there is a clear benefit of training on larger datasets: the word prediction performance of the networks increases substantially when the training set is increased beyond 1 million images (which is roughly the size of Imagenet); for our networks, it only levels out after ~50 million images.
[ et al 2015 didn’t scale up model capacity accordingly, probably because they were so they were underfitting, and so don’t get the log-linear curve we know goes out to billions, but just an asymptote n ~ 0.05b.]