Observability·6 min read

How to instrument LLM apps with OpenTelemetry

OpenTelemetry (OTel) is the vendor-neutral standard for distributed tracing. The GenAI semantic conventions extend it to LLM calls. This guide covers setting up OTel tracing for LLM applications, exporting to Jaeger or Grafana, and the GenAI conventions.

Why use OpenTelemetry for LLMs

Proprietary observability tools (LangSmith, Helicone, Braintrust) are convenient but create vendor lock-in. OpenTelemetry is an open standard supported by all major observability backends — Jaeger, Grafana Tempo, Honeycomb, DataDog, New Relic. Instrument once, export anywhere.

The OpenTelemetry GenAI Semantic Conventions (released in 2024) define standard attribute names for LLM spans: `gen_ai.system`, `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`. Using these conventions makes your traces compatible with any GenAI-aware observability tool.

Set up the OTel SDK

Install: `pip install opentelemetry-sdk opentelemetry-exporter-otlp opentelemetry-instrumentation-openai`. Configure the tracer provider and set up the OTLP exporter: `from opentelemetry import trace; from opentelemetry.sdk.trace import TracerProvider; from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter; provider = TracerProvider(); provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint='http://localhost:4318/v1/traces'))); trace.set_tracer_provider(provider)`.

For automatic OpenAI instrumentation: `from opentelemetry.instrumentation.openai import OpenAIInstrumentor; OpenAIInstrumentor().instrument()`. Every OpenAI API call now automatically creates a span with the request model, token usage, and response.

Create custom spans

Wrap logical operations in spans: `tracer = trace.get_tracer('my-app'); with tracer.start_as_current_span('retrieve-context') as span: results = vector_store.search(query); span.set_attribute('retrieved_chunks', len(results)); span.set_attribute('query_length', len(query))`.

Add LLM-specific attributes to your spans: `span.set_attribute('gen_ai.request.model', model); span.set_attribute('gen_ai.usage.input_tokens', usage.input); span.set_attribute('gen_ai.usage.output_tokens', usage.output); span.set_attribute('gen_ai.usage.cost_usd', cost)`. These show up in dashboards as queryable dimensions.

Export and visualise

Run Jaeger locally for development: `docker run -p 16686:16686 -p 4318:4318 jaegertracing/all-in-one`. Open `http://localhost:16686` to browse traces. For production, use a managed OTLP endpoint (Grafana Cloud, Honeycomb, DataDog OTLP endpoint) — just change the exporter URL.

Build Grafana dashboards over your OTel data. Key metrics to visualise: P50/P90/P99 latency by model, token usage over time, error rate by model and endpoint, cost per feature/user. Grafana's OTel integration generates these dashboards automatically from your traces.

More Observability guides

Observability·6 min read

How to trace and debug LLM calls with LangSmith

LangSmith is LangChain's observability platform for logging, tracing, and evaluating LLM applications. This guide covers setup, automatic tracing, custom traces, and using the dashboard to debug production issues.

Read guide

Observability·6 min read

How to test LLM prompts automatically with Promptfoo

Promptfoo is an open-source CLI for testing, evaluating, and comparing LLM prompts. This guide covers writing test cases in YAML, running evaluations, comparing models, and catching prompt regressions in CI.

Read guide

Observability·5 min read

How to monitor and reduce LLM API costs in production

LLM API costs can grow unexpectedly as usage scales. This guide covers cost attribution, per-request tracking, anomaly detection, and practical techniques that reduce costs by 50–80% without sacrificing quality.

Read guide

Browse all guides