LangSmith is LangChain's observability platform for logging, tracing, and evaluating LLM applications. This guide covers setup, automatic tracing, custom traces, and using the dashboard to debug production issues.
LLM applications are opaque by default. When a user reports a bad answer, you need to see the exact prompt that was sent, the model's response, which tools were called, what they returned, and how long each step took. Without tracing, debugging requires reproducing the issue from scratch — often impossible with production data.
LangSmith captures every LLM call and tool execution automatically when you use LangChain or LangGraph. For non-LangChain applications, the SDK provides a simple decorator and context manager API for manual instrumentation.
Create a free account at smith.langchain.com. Create a project and copy your API key. Set environment variables: `export LANGCHAIN_TRACING_V2=true; export LANGCHAIN_API_KEY=ls__...; export LANGCHAIN_PROJECT=my-project`. That's it — any LangChain code now automatically logs to LangSmith.
Free tier limits: 5000 traces per month. This is generous for development but will be exceeded in production. The Plus plan ($39/month) includes 10M traces. For self-hosted deployments, LangSmith is open-source and can be run on your own infrastructure.
Each trace shows the full execution tree: the top-level chain, sub-chains, individual LLM calls with the exact prompt and response, tool calls with inputs and outputs, and latency at each step. Click any node to expand it and see the full content.
Use the filter UI to find specific traces: filter by date range, model name, latency, token count, or custom tags. For multi-user applications, tag traces with the user ID: `from langsmith import traceable; @traceable(metadata={'user_id': user.id}) def my_function(): ...`. Then filter by user in the dashboard.
Use the `@traceable` decorator for any function you want to trace: `from langsmith import traceable; @traceable(name='classify_document', tags=['production']) def classify(text: str) -> str: response = openai.chat.completions.create(...); return response.choices[0].message.content`. The decorator automatically captures inputs, outputs, and timing.
For manual tracing with more control: `from langsmith import Client; client = Client(); run = client.create_run(name='my_run', inputs={'prompt': prompt}); # ... do work ...; client.update_run(run.id, outputs={'result': result}, end_time=datetime.now())`.
Create a dataset of test prompts and expected outputs in the LangSmith UI. Run your application against the dataset with an evaluator: `from langsmith.evaluation import evaluate; results = evaluate(my_app, data='my-dataset', evaluators=[exact_match, llm_judge])`. The dashboard shows pass rates and diffs between runs.
Use datasets to catch regressions when you update prompts or switch models. Run the evaluation before and after the change and compare the scores. This turns LLM evaluation from an ad-hoc process into a repeatable CI check.
Promptfoo is an open-source CLI for testing, evaluating, and comparing LLM prompts. This guide covers writing test cases in YAML, running evaluations, comparing models, and catching prompt regressions in CI.
Read guideLLM API costs can grow unexpectedly as usage scales. This guide covers cost attribution, per-request tracking, anomaly detection, and practical techniques that reduce costs by 50–80% without sacrificing quality.
Read guideOpenTelemetry (OTel) is the vendor-neutral standard for distributed tracing. The GenAI semantic conventions extend it to LLM calls. This guide covers setting up OTel tracing for LLM applications, exporting to Jaeger or Grafana, and the GenAI conventions.
Read guide