LLM observability is the practice of collecting telemetry from applications built on large language models (LLMs) to understand how they behave in production: what the model was asked, what it returned, how much it cost, how it reasoned across multiple steps, and whether the output was actually correct. It extends conventional observability (logs, metrics, and traces) with signals that only exist once a model is in the loop, like token usage, tool-call decisions, and quality scores that stand in for the error codes a language model never emits.
The practice grew out of two older disciplines that each covered part of the problem and neither covered the whole. Traditional observability had spent years standardizing how distributed systems report logs, metrics, and traces, largely around OpenTelemetry. MLOps had built tooling for monitoring machine learning models in production: data drift, feature quality, prediction accuracy. When applications built on models like GPT and Claude started reaching production in 2023, neither fit cleanly. Accuracy metrics didn't capture whether a generated answer was good, and infrastructure dashboards stayed green while outputs went wrong.
So a new layer formed on top. Early LLM applications were mostly single API calls, and the first dedicated tools (LangSmith, Arize Phoenix, Traceloop's OpenLLMetry) focused on recording each prompt and completion with its latency, token count, and cost. As applications grew from single calls into Retrieval-Augmented Generation (RAG) pipelines and then into multi-step agents that plan and invoke tools, observability had to grow with them, shifting from logging one request to tracing an entire reasoning chain. The umbrella discipline picked up names along the way (LLMOps, and AgentOps for the agent-specific parts), but the observability piece kept a constant goal: make the non-deterministic behavior of a model visible enough to debug, evaluate, and control.
The most consequential recent shift is standardization. In April 2024, OpenTelemetry formed a Generative AI Special Interest Group to define a shared vocabulary for LLM telemetry, so a model call looks the same whether it came from a Python service, a Java app, or a hand-rolled wrapper around an HTTP endpoint. That work, covered in detail below, is what's moving LLM observability from a patchwork of proprietary schemas toward something portable across tools.
Why traditional monitoring falls short
Standard observability still matters for LLM applications. Latency, throughput, and error rates are as relevant as ever. They're just no longer sufficient on their own, and the reasons are worth being specific about.
The first is non-determinism. With a conventional service, a slow p99 usually points to a slow database query or resource contention. You find the slow span and fix it. With an LLM, the same elevated latency could mean the model is working through a long reasoning chain, a retrieval step is pulling too many documents, or a tool call is timing out three hops into an agent workflow. Same symptom, unrelated causes, different fixes. And because outputs vary between identical calls, you can't reproduce a bad response without capturing the exact input, model, and parameters from the moment it happened.
The second is that correctness lives outside the status code. A response that invents a product feature, cites a policy that doesn't exist, or answers a question the user never asked returns HTTP 200 every single time. Traditional tooling was built to catch crashes and slowdowns, not wrong-but-well-formed text. Catching that requires a second layer of observability aimed at what the outputs say, not how quickly they arrive.
The third is cost. LLM spend is driven by token counts, not request counts, so an application can be fast, error-free, and quietly expensive all at once. Without telemetry on token usage, that cost is invisible until the provider bill arrives.
The layers of LLM observability
A complete picture spans three kinds of signals, and most teams underinvest in the last two.
Operational signals are the ones your existing stack already handles: latency per request, throughput, provider error rates, and token usage. Tokens deserve special attention because they map straight to cost. An application that works correctly but sends four times more input tokens than it needs is burning money with nothing in your dashboards to flag it.
Semantic signals measure what the model actually produced. This is where correctness, relevance, faithfulness to retrieved context, and safety get quantified. These can't come from infrastructure alone. They require evaluation, which is involved enough that it deserves its own section below.
Agentic signals trace multi-step reasoning. Modern AI products rarely make a single model call. They plan, retrieve documents, call tools, feed results back to the model, and loop. When something breaks in that chain, you need the full execution tree: which tool ran with which arguments, which reasoning branch went sideways, where tokens are piling up across turns. Without it, debugging an agent is guesswork, and the failure modes unique to agents (stuck tool loops, runaway token costs, lost context between steps) stay invisible.
How output quality actually gets scored
Semantic quality is the hard part, because "is this a good answer?" has no single number behind it. In practice, teams break it into distinct dimensions and measure each separately. Correctness is factual accuracy against a known answer. Faithfulness is whether a Retrieval-Augmented Generation (RAG) response stays grounded in the documents it retrieved instead of inventing details. Relevance is whether the answer addressed the actual question. Safety covers toxicity, bias, and leaking sensitive data. A system can score well on one and badly on another, which is exactly why a single "quality" metric hides the failures that cause incidents.
Three methods do the measuring. Reference-based checks compare output to a known-correct answer and work when ground truth exists, like question-answering or classification. Human review sets the gold standard but doesn't scale to production traffic. And increasingly the default for open-ended quality is LLM-as-a-judge: using a separate model to score outputs against a rubric you define. It's flexible, cheap relative to human review, and reportedly agrees with human raters at roughly the rate two humans agree with each other, though that number depends heavily on the task and the prompt.
LLM-as-a-judge is not plug-and-play, and the failure modes are worth knowing before you trust the scores. Judges carry position bias (favoring whichever answer comes first), verbosity bias (rewarding longer responses), and self-preference bias (scoring their own model family higher). The practical pattern that's emerged: run a cheap, fast judge on production traffic for span-attached scoring, calibrate it against a stronger frontier model and a set of human-labeled examples, and route the lowest-scoring outputs to human review. One rule that keeps coming up is to never use the same model family as both generator and judge, since it will grade itself too kindly.
There's also a timing distinction. Offline evaluation runs pre-deployment against curated datasets with known answers, and it's how you catch regressions before shipping a new prompt or model. Online evaluation scores live production traffic, usually with referenceless judges since real requests don't arrive with labeled answers. You want both: offline to gate changes, online to catch the drift that only shows up under real usage.
Security and privacy signals
LLM applications open attack surfaces that traditional apps don't have, and observability is where you catch them. Prompt injection, where a user crafts input designed to override the model's instructions and make it misbehave, is the headline risk for anything user-facing. Leakage of personally identifiable information (PII) is another: a model can repeat sensitive data back in a completion, and without monitoring you won't know until it's a problem. Both belong in your telemetry as signals you can alert on.
Privacy also shapes how you instrument in the first place. Prompts and completions are the richest debugging data you have and the most dangerous to store, because they routinely contain personal data, credentials, and proprietary content. The right default is to capture the structure of every LLM interaction (timings, token counts, model, status, the span graph) while keeping the actual message content opt-in, redacted, and governed by retention rules that match your data policy. As it happens, this is exactly how the OpenTelemetry conventions are designed to work, which is a good segue.
How OpenTelemetry standardizes LLM telemetry
For a while, every LLM observability tool invented its own schema. One library recorded model, another model_name, a third stuffed the whole request into a single blob. Six months in, nobody could write a dashboard spanning two services, and switching backends meant rewriting every query. The OpenTelemetry GenAI Special Interest Group, formed in April 2024, exists to end that fragmentation with a shared gen_ai.* vocabulary that means the same thing regardless of which library, language, or provider emitted it.
When your application calls a provider, the instrumentation emits a span with attributes like gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and gen_ai.response.finish_reasons. Agent workflows nest into a span tree: an invoke_agent span at the top, a chat span for each model call underneath, and an execute_tool span for each tool invocation, with gen_ai.operation.name distinguishing them and gen_ai.conversation.id tying a multi-turn session together as one unit. Your LLM traces then live in the same distributed trace as the rest of your system, correlated by trace context, routed through the same OpenTelemetry Collector, queried with the same tooling you already run.
Message content follows the privacy model described above. Prompts and completions aren't recorded by default; you opt in with a flag, and the spec recommends emitting them as structured events correlated back to the span rather than cramming them into span attributes, which are indexed, size-limited, and a bad place to leak PII. That design lets you ship structural telemetry everywhere and turn on payload capture only where it's safe.
A caveat on maturity, because it's genuinely load-bearing. As of mid-2026, none of the gen_ai.* spans, metrics, or events are marked stable. They're all still in development status. In June 2026 the conventions were extracted into their own repository, semantic-conventions-genai, which also absorbed the Model Context Protocol (MCP) conventions so tool calls share the same trace vocabulary as the agents that issue them. The core chat and token attributes are solid enough to build production dashboards on, and major platforms already ingest them natively, but the agent and tool-orchestration pieces are still settling. The sensible move is to instrument against the conventions now, pin the version you run, isolate the attribute strings behind a thin mapping layer, and expect some churn in the exact names over the next few releases.
Common pitfalls
The failure mode that catches the most teams is the fully green operational dashboard sitting on top of bad output. If you only watch latency and error rates, you have no signal at all until users complain. Add at least a basic quality signal early, even something as crude as thumbs-up and thumbs-down feedback gives you something to alert on and a dataset to grow from later.
Token creep in long conversations is a quiet cost trap. When a chatbot carries prior turns in its context, input tokens grow with every message. A session that costs 1,200 tokens at turn one can hit 4,000 by turn four, and that curve is invisible unless you're instrumenting gen_ai.usage.input_tokens across the whole session. Worth checking before it shows up on the bill.
In multi-step agent systems, trace context breaks across async boundaries more often than you'd expect. A tool call that spawns a goroutine or async task without propagating the trace context shows up as an orphaned trace instead of a child span, so one coherent agent execution fractures into five disconnected requests in your backend. This is far easier to get right at the start than to diagnose after the fact, so verify propagation end to end before production.
The last one is treating the OpenTelemetry schema as frozen. The conventions have gone through several revisions and a repository split, and many instrumentation libraries still emit older attribute names alongside the new ones. Querying across mixed-generation data without normalizing it produces confusing gaps: missing token counts, filters that match nothing. Check what your instrumentation actually emits before you trust the data, and use the standard dual-emission opt-in when you migrate.
How to get started
Start by deciding what you actually need to watch and why. Cost control, latency, output quality, and compliance pull instrumentation in different directions, and tying your signals to a concrete outcome keeps you from drowning in metrics that don't drive a decision. From there, instrument with OpenTelemetry so your telemetry stays portable, capture structural signals everywhere, and turn on content capture only where privacy allows. Add quality evaluation early rather than retrofitting it under pressure once something has already gone wrong in production. Then close the loop: gate changes with offline evaluation, monitor live traffic with online scoring, and feed your worst production traces back into your test datasets so today's incident becomes tomorrow's regression test.
Final thoughts
Adding an LLM to your stack doesn't mean rebuilding observability from scratch. You still need traces, metrics, and logs. What changes is scope: you now have to cover output quality, agent execution paths, and a new class of security risks that conventional tooling never had to think about.
OpenTelemetry's GenAI conventions are the right foundation to build on. Instrument against them and your telemetry stays portable across backends, so you're not locked into one vendor's schema and you're not maintaining a separate pipeline just for AI signals.
Dash0 is an OpenTelemetry-native observability platform that treats GenAI signals as first-class telemetry, so LLM traces, token metrics, and agent spans flow through the same pipeline as everything else you run. The guide to observing vLLM with OpenTelemetry and Dash0 and the guide to agentic observability in Dash0 show what instrumented AI applications look like in production.
Start a free trial to see your LLM traces, token usage, and agent spans in one place. No credit card required.