Getting LLMs to output reliable JSON, YAML, or custom formats is essential for production applications. This guide covers JSON mode, JSON schema enforcement, grammar-based sampling, and fallback strategies.
LLMs produce text. Parsing that text into a data structure requires either trust (the model produces valid JSON) or defensive code (parse failures handled gracefully). Neither is a great foundation for production systems. Structured output features enforce the format at the generation level, making parsing failures essentially impossible.
Three approaches exist: (1) prompt-only — ask the model for JSON and hope; (2) JSON mode — guarantee valid JSON but not a specific schema; (3) JSON Schema enforcement — guarantee output matches your exact schema. Use approach 3 whenever possible.
OpenAI: pass `response_format={"type": "json_schema", "json_schema": {"name": "result", "strict": true, "schema": {...}}}`. With `strict: true`, the model is constrained to produce output that exactly matches your schema — no additional fields, no missing required fields. This is guaranteed at the grammar level, not just prompted.
Anthropic: use the `tools` parameter with a single tool whose `input_schema` matches your desired output structure. Force the model to call the tool with `tool_choice={"type": "tool", "name": "your_tool"}`. The tool input is always valid JSON matching your schema.
Instructor is a Python library that patches OpenAI and Anthropic clients to return Pydantic models instead of raw JSON: `pip install instructor`. Then: `import instructor; from pydantic import BaseModel; client = instructor.from_openai(OpenAI()); class User(BaseModel): name: str; age: int; user = client.chat.completions.create(model='gpt-4o-mini', messages=[...], response_model=User)`. `user.name` and `user.age` are populated and type-safe.
Instructor handles retries automatically. If the model produces malformed output, it feeds the validation error back to the model and asks it to fix the response — up to a configurable maximum number of retries. This achieves near-100% reliability even with models that don't natively support structured output.
For local models served by llama.cpp or Ollama, use grammar-based constrained generation. This works at the sampling level — invalid tokens are assigned zero probability, so the model literally cannot produce invalid output.
Ollama supports JSON format natively: add `format: 'json'` to your API request. For a specific schema, use the `format` field with a JSON Schema object: `{"format": {"type": "object", "properties": {"name": {"type": "string"}}, "required": ["name"]}}`. This guarantees schema-valid output regardless of model size.
The system prompt is the most powerful lever you have over LLM behaviour. This guide covers the key components of an effective system prompt, common mistakes, and battle-tested patterns for production use.
Read guideChain-of-thought (CoT) prompting dramatically improves LLM performance on multi-step reasoning tasks. This guide explains when and how to use it, from simple 'think step by step' to structured CoT templates.
Read guide