“Chinchilla Scaling: A Replication Attempt”, Tamay Besiroglu, Ege Erdil, Matthew Barnett, Josh You2024-04-15 (Transformer, AI scaling)⁠:

Hoffmann et al 2022 propose 3 methods for estimating a compute-optimal scaling law. We attempt to replicate their third estimation procedure, which involves fitting a parametric loss function to a reconstruction of data from their plots.

We find that the reported estimates are inconsistent with their first two estimation methods, fail at fitting the extracted data, and report implausibly narrow confidence intervals–intervals this narrow would require over 600,000 experiments, while they likely only ran fewer than 500.

In contrast, our re-derivation of the scaling law using the third approach yields results that are compatible with the findings from the first two estimation procedures described by Hoffmann et al 2022.

[Apparently the original Chinchilla anomalies turn out to be due to a bug in the early-stopping DM code, and when fixed, does yield the revised Besiroglu et al 2024 numbers.]