Parameters
The numerical weights inside a neural network that are learned during training — the 'knowledge' of the model, measured in billions for modern LLMs.
Parameters are the tunable numerical weights in a neural network. During training, these values are adjusted to minimize prediction error on the training data. A model's parameter count is a primary indicator of its capacity to learn and store information. GPT-3 had 175 billion parameters; modern frontier models are estimated to have hundreds of billions to over a trillion.
More parameters generally means better performance, but with diminishing returns and dramatically higher compute costs. The scaling laws of LLMs show that performance improves predictably with more parameters, more data, and more compute — but the most important variable shifts depending on what resources are available.
Parameter count is often conflated with model capability, but the relationship is complex. A 70B parameter model like Llama 3.3 can outperform older 175B models because of improvements in training data quality and techniques. MoE models further complicate the comparison — a 47B active parameter MoE model may behave like a much larger dense model in practice.
Read next
How LLMs Work: A Technical Overview
A clear technical explanation of how large language models actually process text, generate responses, and represent knowledge — from tokenization to sampling.
The Scaling Laws of LLMs: Why Bigger Often Means Better
The mathematical relationship between model size, training data, compute, and capability — and what the scaling laws predict about the future of AI.
Model Quantization: Running LLMs on Less Hardware
How quantization reduces model size and inference cost by using lower-precision numbers — making 70B parameter models run on a single GPU and enabling on-device AI.