Quantization
A compression technique that reduces model size and inference cost by storing weights in lower-precision numerical formats, trading a small amount of accuracy for much lower memory and compute requirements.
Quantization reduces the numerical precision of model weights from 32-bit or 16-bit floating point to lower-precision formats like 8-bit integers (INT8) or 4-bit integers (INT4). A model quantized to 4-bit uses roughly 4x less memory than its 16-bit equivalent, enabling larger models to run on consumer hardware.
Modern quantization methods like GPTQ, AWQ, and GGUF have become sophisticated enough that 4-bit quantized models typically retain over 95% of the original model's performance on benchmarks. The most accessible format is GGUF (used by llama.cpp), which allows running quantized versions of Llama, Mistral, and other open models on a laptop CPU.
Quantization is one of the most important techniques for democratizing access to powerful AI models. It is what allows a 70B parameter model — which would normally require 8 professional-grade GPUs — to run on a single consumer GPU with 24GB of VRAM, or even on a high-end laptop using the CPU.
Read next
Model Quantization: Running LLMs on Less Hardware
How quantization reduces model size and inference cost by using lower-precision numbers — making 70B parameter models run on a single GPU and enabling on-device AI.
Open-Weight vs Open-Source Models: What's the Difference?
Why 'open-source AI' is often a misleading term — and what it actually means when a model is open-weight, what's included, what's not, and why it matters for developers.
Mixture of Experts: How LLMs Scale Efficiently
The architecture behind GPT-4, Llama 4, and Mistral — where only a subset of model parameters are active per token, enabling huge capacity at manageable inference cost.