In-Context Learning
The ability of an LLM to learn from examples provided directly in the prompt, without any weight updates or training, just from seeing patterns in context.
In-context learning (ICL) is one of the most striking emergent properties of large language models. Rather than requiring task-specific training, LLMs can adapt their behavior based on examples provided in the prompt itself. Show a model three examples of translating English to French, and it will correctly translate the fourth without any fine-tuning.
ICL comes in several varieties: zero-shot (no examples, just instructions), one-shot (one example), and few-shot (several examples). GPT-3's strong few-shot learning ability was a major reason for its impact in 2020, it showed that a single model could flexibly perform many tasks without modification.
The mechanism behind ICL is still debated among researchers. Some evidence suggests models perform a form of implicit gradient descent in their forward pass. Others argue they pattern-match to similar sequences seen during training. Practically, the main limitation is context window size: you can only fit so many examples in a single prompt.
Read next
Prompt Engineering: The Complete Guide
Master the art and science of writing effective prompts, from basic techniques to advanced methods like chain-of-thought, few-shot learning, and structured output generation.
What Is a Large Language Model?
A clear, jargon-free introduction to large language models, what they are, how they work at a high level, and why they're transforming software and society.
The Scaling Laws of LLMs: Why Bigger Often Means Better
The mathematical relationship between model size, training data, compute, and capability, and what the scaling laws predict about the future of AI.