Token
The basic unit of text that an LLM processes — roughly corresponding to a word or word-piece. Models read input and produce output in tokens, which is also how API usage is measured and billed.
Tokens are the fundamental units of text that language models process. A tokenizer splits input text into a sequence of tokens before feeding it to the model. Common English words are typically single tokens. Longer or less common words may be split into multiple subword tokens. A rough rule of thumb: 1 token ≈ 0.75 English words, or about 4 characters.
Tokenization is not purely word-based. GPT-style models use Byte Pair Encoding (BPE) tokenization, which splits text at the subword level based on frequency patterns in training data. The word 'tokenization' might be a single token; 'antidisestablishmentarianism' would be split into several. Numbers, code, and non-English text often tokenize less efficiently than English prose.
Tokens are also the unit of API pricing. When an LLM API charges $3 per million input tokens, every character in your prompt, documents, and conversation history contributes to that count. Output tokens (the generated response) are typically priced separately and higher. Understanding tokenization helps optimize prompts for cost and context window usage.
Read next
Tokens and Tokenization: The Building Blocks of LLMs
Everything you need to know about tokens — how LLMs split text into pieces, why tokenization matters for cost and performance, and how different languages tokenize.
Context Windows Explained: The Working Memory of LLMs
What context windows are, why they matter for building AI applications, how they've grown from 4K to 10M tokens, and how to manage them effectively.
Cost Per Million Tokens: The AI Economics Guide
How LLM API pricing works, why output tokens cost more than input, how to calculate actual task costs, and how prices have changed over time.