“Power Law Trends in Speedrunning and Machine Learning”, 2023-04-19 ():
We find that improvements in speedrunning world records follow a power law pattern. Using this observation, we answer an outstanding question from previous work: How do we improve on the baseline of predicting no improvement when forecasting speedrunning world records out to some time horizon, such as one month?
Using a random effects model, we improve on this baseline for relative mean square error made on predicting out-of-sample world record improvements as the comparison metric at a p < 10−5 level. The same set-up improves even on the ex-post best exponential moving average forecasts at a p = 0.15 level while having access to substantially fewer data points.
We demonstrate the effectiveness of this approach by applying it to Machine Learning benchmarks and achieving forecasts that exceed a baseline.
Finally, we interpret the resulting model to suggest that (1) ML benchmarks are far from saturation and (2) sudden large improvements in Machine Learning are unlikely but cannot be ruled out.