A prompt library treats prompts like code — versioned, tested, and reusable. This guide covers structuring a prompt library, versioning strategies, and tooling for testing prompt quality.
As you build LLM applications, prompts proliferate across scripts, notebooks, and application code. Without organisation, you end up with duplicate prompts, no way to track what changed when something breaks, and no mechanism to share good prompts across the team.
A prompt library treats prompts like code: each prompt has a name, a version, tests, and a changelog. When a model update changes prompt behaviour, you catch it in your test suite rather than in production.
The simplest approach is a YAML file per prompt: slug, version, content (the prompt text), model, parameters (temperature, max tokens), and test_cases (input/expected output pairs). Store them in a `prompts/` directory in your repository.
For teams, a dedicated prompt management platform (LangSmith, Promptfoo, PromptLayer) adds a UI for editing, A/B testing, and tracking performance over time. These are worth the setup cost once you have more than 20 prompts in production.
Use semantic versioning for prompts: 1.0.0 for the initial version, 1.0.1 for minor fixes, 1.1.0 for changes that alter the output format, 2.0.0 for complete rewrites. Reference prompts by version in your application code: `get_prompt('summarise-article', version='1.2.0')`.
Write test cases for each prompt: a set of input-output pairs that define correct behaviour. Run them automatically in CI using an LLM-as-judge pattern — have a second model evaluate whether the output matches the expected criteria. Set a threshold (e.g. 90% pass rate) as a quality gate.
A good prompt YAML includes: name, description (what it does and when to use it), model (which model it was tested with), parameters, system_prompt, user_prompt_template (with placeholders like `{{document}}`), and examples with expected outputs.
Keep prompt logic in templates, not in Python string interpolation. Templates are readable, diffable, and editable by non-engineers. Use Jinja2 or Mustache syntax — both are well-supported and produce clean diffs in version control.
The system prompt is the most powerful lever you have over LLM behaviour. This guide covers the key components of an effective system prompt, common mistakes, and battle-tested patterns for production use.
Read guideChain-of-thought (CoT) prompting dramatically improves LLM performance on multi-step reasoning tasks. This guide explains when and how to use it, from simple 'think step by step' to structured CoT templates.
Read guide