The Scaling Laws of LLMs: Why Bigger Often Means Better

Key Takeaways

The Discovery That Changed Everything

In 2020, OpenAI published 'Scaling Laws for Neural Language Models,' demonstrating that LLM performance improves smoothly and predictably with model size, dataset size, and compute. The relationship is a power law: doubling compute improves performance by a consistent multiplicative factor. This meant AI progress could be planned and projected like an engineering problem, not hoped for like a research breakthrough.

The Chinchilla scaling laws (DeepMind, 2022) refined this: the original OpenAI findings trained large models on insufficient data. Chinchilla showed that optimal training requires approximately 20 tokens per parameter, meaning a 70B model needs 1.4 trillion tokens of training data. This insight shifted the field toward training smaller models on more data.

The Three Scaling Axes

Performance scales smoothly with three quantities: N (number of parameters), D (dataset size in tokens), and C (compute, measured in FLOPs). For a fixed compute budget, there's an optimal allocation between N and D, the Chinchilla optimal frontier. Frontier labs use these scaling laws to decide how large to make their models and how long to train them before deploying.

Importantly, the scaling laws have no bottom floor visible yet for the capabilities that matter, more compute consistently produces better models. This is why the major labs are in a race to build larger and larger compute clusters, and why AI capability improvements have been so rapid and sustained.

Emergent Capabilities at Scale

The scaling laws predict smooth continuous improvements in perplexity. But certain capabilities appear discontinuously, they're absent in smaller models and suddenly present in larger ones. These emergent capabilities include Few-Shot learning, Chain of Thought reasoning, arithmetic, translation, and code generation. They appeared without explicit training on these tasks. A Foundation Model large enough to exhibit these emergent capabilities becomes far more useful as an Instruct Model after fine-tuning and alignment.

The emergence of capabilities at scale is a key reason why larger models are not just 'slightly better' at existing tasks, they can do things qualitatively different from what smaller models can do. This unpredictability (which capabilities will emerge and when) makes AI development partially like engineering and partially like discovery.

Test-Time Compute Scaling

The latest scaling frontier is test-time compute: rather than only scaling training, also scale how much compute is used at inference. Reasoning models (o3, DeepSeek R1) allocate more tokens to 'thinking' for hard problems, extracting more capability from the same model weights. The scaling law analog: performance continues to improve as you give the model more tokens to think.

This opens a new dimension of AI improvement that doesn't require building ever-larger training runs. A smaller model with extended reasoning may outperform a larger model without it on sufficiently hard tasks. Adaptive compute allocation, spending more thinking tokens on hard problems than easy ones, is the leading edge of current scaling research.