Mixture of Experts
A neural network architecture where only a subset of model parameters are activated for each input token, enabling very large models that are surprisingly efficient to run.
Mixture of Experts (MoE) is an architecture that replaces the dense feed-forward layers in a transformer with a collection of 'expert' networks and a routing mechanism. For each token, the router selects a small number of experts (typically 2–8 out of dozens) to process that token. This means a model with 100 billion total parameters might only activate 10 billion for any given input.
MoE models offer a favorable trade-off: they can achieve the capability of a large dense model at the inference cost of a much smaller one. GPT-4 is widely believed to use MoE architecture. Mixtral 8x7B demonstrated that a 47B parameter MoE model could outperform a 70B dense model. Meta's Llama 4 Maverick uses 128 experts.
The main challenges of MoE are training stability, load balancing across experts, and memory footprint. All expert weights must be loaded into memory even though only a fraction are used per token. Despite these challenges, MoE has become the dominant architecture for frontier models that need to balance raw capability with inference efficiency.
Read next
Mixture of Experts: How LLMs Scale Efficiently
The architecture behind GPT-4, Llama 4, and Mistral — where only a subset of model parameters are active per token, enabling huge capacity at manageable inference cost.
The Transformer Architecture Explained
A deep dive into the transformer architecture — the neural network design that powers virtually every major LLM, from its attention mechanism to positional encodings.
Model Quantization: Running LLMs on Less Hardware
How quantization reduces model size and inference cost by using lower-precision numbers — making 70B parameter models run on a single GPU and enabling on-device AI.