“Surprises in High-Dimensional Ridgeless Least Squares Interpolation”, Trevor Hastie, Andrea Montanari, Saharon Rosset, Ryan J. Tibshirani2019-03-19 (neural net, AI scaling; similar)⁠:

Interpolators—estimators that achieve zero training error—have attracted growing attention in machine learning, mainly because state-of-the art neural networks appear to be models of this type.

In this paper, we study minimum 𝓁₂ norm (“ridgeless”) interpolation in high-dimensional least squares regression. We consider two different models for the feature distribution: a linear model, where the feature vectors x_i ∈ ℝ^p are obtained by applying a linear transform to a vector of i.i.d. entries, x_i = ∑^1⁄2z_i (with z_i ∈ ℝ^p); and a nonlinear model, where the feature vectors are obtained by passing the input through a random one-layer neural network, x_i = ϕ(Wz_i) (with z_i ∈ ℝ^d, W ∈ ℝ^p×d a matrix of i.i.d. entries, and ϕ an activation function acting component-wise on Wz_i).

We recover—in a precise quantitative way—several phenomena that have been observed in large-scale neural networks and kernel machines, including the “double descent” behavior of the prediction risk, and the potential benefits of overparametrization.

Similar Links: