The system prompt is the most powerful lever you have over LLM behaviour. This guide covers the key components of an effective system prompt, common mistakes, and battle-tested patterns for production use.
The system prompt sets the model's role, constraints, and context for the entire conversation. A well-crafted system prompt can take a general-purpose model and make it behave like a highly specialised expert that gives consistent, formatted, on-brand responses.
Most inconsistency in LLM applications comes from underspecified system prompts. If you have not told the model what tone to use, what to do when it doesn't know something, what format to respond in, and what topics to avoid — it will make its own choices, which may vary between calls.
1. Role: Who is the model? 'You are a senior software engineer at a fintech startup specialising in Python and TypeScript.' Be specific — 'helpful assistant' is too generic.
2. Task context: What is the model's purpose in this application? 'You help developers debug their code, explain error messages, and suggest improvements.'
3. Response format: How should responses be structured? 'Always use markdown. For code, include the language tag. For explanations, use a numbered list of steps.'
4. Constraints: What should the model not do? 'Do not provide financial or legal advice. If asked, redirect the user to a qualified professional.' Be explicit about off-topic boundaries.
5. Fallback behaviour: What should happen when the model doesn't know? 'If you are not certain of an answer, say so clearly rather than guessing. Uncertainty is better than confident misinformation.'
Being too vague: 'You are a helpful assistant' gives the model no actionable guidance. Every instruction should be specific enough that a human new to the task could follow it.
Contradictory instructions: 'Be concise but thorough' gives the model conflicting directives. Choose one: 'Responses should be under 200 words unless the user explicitly asks for a detailed explanation.'
Missing edge cases: Think about the 10% of inputs that are off-topic, ambiguous, or attempts to abuse the system. Your system prompt should specify how to handle each case.
If you need consistent JSON or structured output, define the exact schema in the system prompt with an example: 'Always respond with a JSON object matching this schema: {"answer": string, "confidence": 0-1, "sources": string[]}. Never include any text outside the JSON object.'
For even better reliability, use the `response_format: {type: 'json_schema', json_schema: {...}}` parameter available in newer OpenAI and Anthropic APIs. This enforces schema compliance at the model output layer.
Write 10–15 representative test prompts covering normal cases, edge cases, and adversarial inputs. Run them against your system prompt and check for inconsistencies. A good system prompt should produce correct behaviour for 90%+ of your test cases before you ship.
Version your system prompts like code. Small changes can have unexpected effects — keeping a changelog lets you roll back if a seemingly minor edit causes regressions.
Chain-of-thought (CoT) prompting dramatically improves LLM performance on multi-step reasoning tasks. This guide explains when and how to use it, from simple 'think step by step' to structured CoT templates.
Read guideFew-shot prompting uses examples to teach LLMs new formats and behaviours without fine-tuning. This guide explains how to construct effective examples and when few-shot beats zero-shot.
Read guide