How LLMs Work: A Technical Overview

Key Takeaways

The LLM Processing Pipeline

When you send a message to an LLM, a series of transformations occurs before text appears in response. First, your text is broken into tokens, subword units that the model processes. These tokens are converted to numerical vectors (embeddings) that enter a stack of transformer layers. Each layer refines the representations through attention and feed-forward operations. Finally, the output layer produces a probability distribution over the vocabulary, and a token is sampled. This process repeats for every token in the response.

The full pipeline: Text → Tokenizer → Token embeddings → Transformer layers (×N) → Output logits → Sampling → Token → Detokenizer → Text. Each step happens in milliseconds on modern hardware, which is why you see responses streaming at 50-100 tokens per second.

Tokens and Embeddings

Tokens are the atoms of LLM processing. A tokenizer maps text to integer IDs: 'Hello world' might become [15496, 995]. The model never sees raw text, only these integers. An embedding layer maps each integer to a high-dimensional vector (typically 4,096-8,192 dimensions for large models). These vectors encode the initial meaning of each token before context is applied.

The embedding dimension is a fundamental architectural choice. Higher-dimensional embeddings can encode richer information but require more memory and compute. The embeddings are learned during training, by the end of training, semantically similar tokens have similar embedding vectors, even if their surface forms are completely different.

What Happens Inside a Transformer Layer

A transformer layer applies two operations in sequence. First, multi-head self-attention: each token's representation is updated based on weighted information from all other tokens in the context. The attention weights determine how much each token 'attends to' each other token, creating context-aware representations. Second, a feed-forward network: a two-layer MLP applied independently to each position, adding non-linearity and enabling the model to transform representations further.

Residual connections wrap both operations, meaning each layer's output is added to its input rather than replacing it. This is crucial for training deep networks, gradients flow back through the residual path, enabling effective learning across dozens or hundreds of layers. Layer normalization stabilizes training by normalizing activations before each operation.

Where 'Knowledge' Lives

An LLM's 'knowledge' is not stored in a database or lookup table. It is encoded in billions of floating-point numbers, the model's weights, distributed across embedding matrices, attention projections, and feed-forward networks. This distributed, implicit storage is what makes LLMs both powerful (they compress patterns from vast data) and unreliable (they can confabulate with fluent confidence).

The feed-forward layers in particular appear to act as 'memory' banks. Research has shown that factual associations are often stored in specific FFN neurons and can be surgically modified, a technique called model editing. The model doesn't 'know' that Paris is the capital of France the way a database does; it has learned that these tokens tend to co-occur in ways consistent with that fact.

How Text Is Generated

Generation is autoregressive: the model produces one token at a time, with each new token appended to the context before generating the next. This creates a left-to-right dependency chain. The output logits (unnormalized scores for each vocabulary token) are transformed by temperature and sampling parameters (top-p, top-k) into a probability distribution, and a token is drawn from that distribution.

This process has implications for LLM behavior: the model cannot revise earlier tokens, so errors early in a response can propagate and compound. It also means the model 'commits' to a reasoning path as it generates, which is why techniques like Chain of Thought prompting and multi-step reasoning (which allocate explicit tokens to thinking) improve accuracy on hard tasks. The Foundation Model underlying a chat assistant is also what powers Agents, autonomous systems that call tools and reason across multiple steps.