The Larger World ArchiveClaims · Sources · Evidence
Internet, media and modern lifeLive debateMachine learning · 2020 – present

Scaling Laws

Double the parameters and the loss drops by a predictable amount, across seven orders of magnitude. Then the awkward part: loss is not capability, and the smooth curves stop being smooth.

A graphics card from 1998. The scaling-law papers of 2020 were about models trained on har
A graphics card from 1998. The scaling-law papers of 2020 were about models trained on hardware many orders of magnitude larger than this. Image: Timothy Borkowski · CC BY-SA 3.0 — Wikimedia Commons.

What the laws say

Work from OpenAI in 2020 showed that language-model loss follows a power law in parameter count, dataset size and compute, with smooth curves across an enormous range. That made it possible to predict the performance of a large model from small, cheap experiments.

The Chinchilla result in 2022 corrected the recipe: for a fixed compute budget, models were badly undertrained. Given the same compute, a smaller model on more data beats a larger model on less. Most of the industry's subsequent training runs follow that arithmetic.

More recently the same logic has been applied to inference: spending more compute at generation time — sampling many candidates, or letting a model think longer — improves results on reasoning benchmarks along a similar curve.

Where it gets murky

Loss is not capability. The laws predict cross-entropy on held-out text, not whether a model can do arithmetic, follow instructions or avoid fabricating citations.

Emergence is contested. Claims that abilities appear suddenly at certain scales were challenged in 2023 by work showing that with better metrics many 'emergent' jumps become smooth curves. Some discontinuities are real; the methodological dispute is not settled.

The curves have limits. Data quality, data exhaustion, and the returns from reinforcement learning and tool use are not captured by the original three-variable fits.

What this means in practice

Reliable: within a model family and a data distribution, more compute predictably lowers loss. That is engineering gold — it tells you what a bigger run will cost and roughly what it will get.

Unreliable: extrapolating from loss to usefulness, and extrapolating past the regime where the fits were made.

Often misreported: 'scaling will solve it' is a bet, not a law. The laws are empirical regularities with measured exponents, not a mechanism.

The archive labels this live debate because the regularity is documented and the interpretation — how far it takes you — is genuinely disputed.

Why it is in the archive

Because scaling is the central empirical fact of the current AI era, and because it is routinely cited as if it were a theory of intelligence.

Related: what models learn, AI risk.

Sources for this page

  • Kaplan et al. (2020), scaling laws for neural language models, arXiv:2001.08361arxiv.org
  • Hoffmann et al. (2022), Chinchilla, arXiv:2203.15556arxiv.org
  • Schaeffer et al. (2023), are emergent abilities a mirage?, arXiv:2304.15004arxiv.org
  • Wikipedia, “Neural scaling law” — findings and criticismen.wikipedia.org