Last updated:
How OpenTelemetry Tracing Works (with Examples)
Modern distributed systems fail in ways that are difficult to diagnose from logs alone. When a single request crosses a dozen services before returning a response, each service can tell you what happened locally, but not how those events fit together across the full request path.
Distributed tracing was created to solve this problem. For years, adopting it often meant installing a proprietary APM agent and instrumenting applications with vendor-specific SDKs. The tooling could be excellent, but the tradeoff was tighter coupling between your instrumentation, telemetry format, and observability backend.
OpenTelemetry tracing changes that model by providing a standard, vendor-neutral data model to instrument applications, represent spans, and propagate trace context across service boundaries.
Because the APIs, data model, and protocols are open standards, you can instrument applications independently of the backend that ultimately stores and analyzes the telemetry.
In this guide, you'll learn how OpenTelemetry represents spans, connects them across service boundaries through context propagation, and uses its API, SDK, and instrumentation libraries to collect tracing data. We'll also cover how the OpenTelemetry Collector fits into production tracing pipelines.
How does OpenTelemetry tracing work?
OpenTelemetry tracing works by recording operations as spans, propagating trace context across service boundaries, and exporting spans to a tracing backend.
A trace is the execution path of a request through a distributed system. It's made up of spans, each describing an individual operation such as an HTTP request, database query, cache lookup, or business logic. Every span records information about the operation, including its duration, attributes, and status.
Spans belonging to the same trace share a trace ID, while parent-child relationships describe how the operations are connected. Together, they form a tree that shows how work flows through the system.
Here's how OpenTelemetry produces this view when a request travels through a distributed application:
- Instrumentation creates spans: Automatic or manual instrumentation records operations such as incoming HTTP requests, database queries, and downstream service calls.
- Context propagation connects services: Trace context travels between
services, typically through
W3C
traceparentheaders, allowing downstream spans to join the same trace. - The SDK exports spans: The SDK processes and exports spans over the OpenTelemetry Protocol (OTLP), either to an OpenTelemetry Collector or directly to a compatible backend.
- The Collector processes telemetry: When deployed, the Collector can batch, filter, sample, and route spans to their destination.
- The backend assembles the trace: A tracing backend such as Dash0 uses trace IDs, parent-span relationships, and timestamps to reconstruct and visualize the request's execution.
The result is a view of the request's execution path that goes beyond showing that a request was slow or failed. You can identify which service introduced latency, pinpoint expensive database operations, and trace failures across service boundaries to understand where they originated.
Understanding the OpenTelemetry span data model
The OpenTelemetry tracing specification defines spans as the fundamental units of tracing. Each span represents a named operation with a start and end time, along with metadata that describes what happened during its execution.
The most important span fields are:
TraceId: A 16-byte identifier shared by every span in the same trace.SpanId: An 8-byte identifier for each individual span.ParentSpanId: References the parent span, allowing backends to reconstruct the trace hierarchy. Root spans (the first one in a trace) have no parent.Name: A human-readable description of the operation, such asGET /api/ordersorSELECT orders.Kind: Describes the operation's role. It could beSERVERfor incoming requests,CLIENTfor outgoing calls,PRODUCERandCONSUMERfor messaging operations, andINTERNALfor in-process work.StartTimeandEndTime: Records when the operation began and ended, allowing its duration to be calculated.Attributes: Key-value pairs that provide context about the operation being tracked, such ashttp.response.status_codeordb.operation.name. OpenTelemetry also provides semantic conventions to standardize commonly used attribute names.Events: Timestamped annotations that record notable moments during an operation, such as retries or exceptions.Status: Indicates whether the operation succeeded (OK), failed (ERROR), or has not been explicitly set (UNSET).Links: Used to associate spans that are causally related without having a direct parent-child relationship, such as messages processed together in a batch.
Trace and span IDs establish relationships between operations, timestamps reveal where time was spent, and attributes, events, and status provide the context needed to investigate the observed behavior. Together, they allow an observability backend to turn independently recorded spans into a navigable trace showing how a request moved through the system.
Understanding individual span fields is only part of the picture. It's equally important to understand how spans combine to form traces and why the distinction matters when debugging distributed systems.
For a deeper explanation, see our guides to traces vs spans in OpenTelemetry, which explains how backends reconstruct traces from independently exported spans, and OpenTelemetry spans, which covers span anatomy and instrumentation best practices.
How trace context propagation works in OpenTelemetry
For spans to form a coherent trace, OpenTelemetry must preserve trace identity and parent-child relationships as execution moves between operations. This is handled through context propagation, which works both within a process and across service boundaries.
Within a process, OpenTelemetry tracks the currently active span so that new
spans can inherit its context. For example, the Node.js SDK uses
AsyncLocalStorage to preserve context across asynchronous operations, allowing
a database span created inside a request handler to become a child of the active
HTTP server span.
Across service boundaries, trace context travels with the request, typically
through the W3C traceparent HTTP header:
The header carries the trace ID, the calling span's ID, and trace flags. When the receiving service extracts this context, it can create spans that belong to the same trace and preserve the relationship to the caller.
OpenTelemetry uses propagators to inject context into outgoing requests and extract it from incoming ones. HTTP and gRPC instrumentation libraries usually handle this automatically, but custom transports may require manual propagation.
When context propagation breaks, so does the trace. Downstream services start new, disconnected traces, fragmenting what should be a single request timeline into isolated operations. Without the relationships connecting them, you lose the ability to follow a request across services and pinpoint where failures originate.
For a deeper explanation of the traceparent and tracestate headers,
propagation formats, and interoperability, see our guide to
W3C Trace Context.
Understanding the OpenTelemetry tracing API and SDK
OpenTelemetry separates tracing into two layers. The Trace API defines how applications create and interact with spans, while the Trace SDK implements the API and handles recording, processing, and exporting telemetry.
At the API level, the following components are exposed:
- The
TracerProvideris the entry point for obtainingTracerinstances. - A
TracercreatesSpanobjects. Each tracer is associated with an instrumentation scope, typically identified by a name and optional version. - A
Spanrepresents an operation and provides methods for adding attributes, setting its status, and ending it.
The API doesn't record or export telemetry on its own, allowing libraries to include OpenTelemetry instrumentation without requiring applications to enable tracing. Once an SDK is configured, those instrumentation calls can produce actual telemetry.
At a high level, the SDK is responsible for:
- Applying sampling decisions when spans are created.
- Recording span data throughout an operation.
- Processing completed spans, typically batching them for efficient export.
- Exporting spans to a Collector or another configured destination.
The SDK can be configured through code or environment variables, depending on the language and distribution.
Tracing your application with OpenTelemetry
For end-to-end visibility into a request path, you need to instrument the important operations across participating services and ensure trace context propagates correctly between them.
OpenTelemetry provides two complementary approaches to instrumenting these services: zero-code instrumentation for common operations and manual instrumentation for application-specific logic.
Together, they let you capture both the underlying technical operations and the business logic that matters to your application.
Zero-code instrumentation
OpenTelemetry provides zero-code instrumentation for popular frameworks and libraries, allowing you to capture telemetry without changing your application code. Depending on the language and supported libraries, this can automatically
capture:
- Incoming and outgoing HTTP requests.
- Database queries and cache operations.
- gRPC calls and messaging operations.
For example, the @opentelemetry/auto-instrumentations-node package for Node.js bundles instrumentation for popular libraries like Express, PostgreSQL, MySQL, Redis, Pino and many others.
You can enable it in a CommonJS application by installing the required packages and configuring a few environment variables:
# install required packages
npm install @opentelemetry/api @opentelemetry/auto-instrumentations-node
# configure export to a backend or collector
export OTEL_SERVICE_NAME=checkout-service
export OTEL_TRACES_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4318/v1/traces
# start the application with OpenTelemetry enabled
node --require @opentelemetry/auto-instrumentations-node/register app.jsThe endpoint assumes an OTLP receiver is running locally. Instrumentation must load before your application dependencies, and ESM applications may require additional loader configuration.
Once enabled, supported libraries automatically create spans and propagate context. An incoming HTTP request, for example, can produce a server span with child spans for database queries and downstream HTTP calls, giving you a connected view of the request without manually instrumenting those operations.
Zero-code instrumentation provides a useful starting point, but its coverage depends on the language, framework, and libraries you use. For more details, see our guide to OpenTelemetry auto-instrumentation.
Manual instrumentation
Manual instrumentation complements automatic instrumentation by revealing what your application is doing at the business level, while automatic instrumentation captures the underlying framework and dependency operations.
For example, suppose you want to measure how long payment authorization takes during checkout:
import { trace, SpanStatusCode } from "@opentelemetry/api";
const tracer = trace.getTracer("checkout-service");
async function authorizePayment(orderId, amount) {
return tracer.startActiveSpan("payment.authorize", async (span) => {
span.setAttributes({
"payment.amount_cents": amount,
"payment.provider": "stripe",
});
try {
return await paymentClient.authorize(orderId, amount);
} catch (err) {
span.setStatus({
code: SpanStatusCode.ERROR,
message: err.message,
});
throw err;
} finally {
span.end();
}
});
}Here, paymentClient.authorize() represents your application's payment
integration.
The startActiveSpan() method creates a span and makes it active for the
duration of the callback. If the payment operation runs inside an instrumented
HTTP request, the new span becomes a child of that request's server span. Any
instrumented outbound calls made during payment authorization can then appear
beneath it.
The setAttributes() call attaches business context to the span, while
setStatus() marks the operation as failed when an exception occurs. Exception
details should be captured through structured logs
correlated with the active trace context,
following OpenTelemetry's
latest recommendations.
In an OpenTelemetry backend such as Dash0, these spans appear together in the request timeline, allowing you to distinguish time spent handling the checkout request from time spent authorizing the payment.
The role of the OpenTelemetry Collector in tracing
Instrumented applications can export spans directly to a tracing backend, but most production deployments use the OpenTelemetry Collector as an intermediary.
The Collector receives all kinds of telemetry (logs, spans, metrics), processes it through configurable pipelines, and forwards it to one or more destinations. This decouples applications from backend-specific ingestion endpoints and provides a central place to manage how tracing data is handled.
A Collector pipeline consists of three main components:
- Receivers accept incoming spans, typically over OTLP/gRPC or OTLP/HTTP.
- Processors modify, filter, sample, or batch spans before export. For
example, the
filter processor
can remove low-value health-check spans, while the
tail_samplingprocessor can retain traces based on errors or latency. - Exporters send the processed spans to an observability backend, another Collector instance, or multiple destinations.
One particularly useful capability is tail-based sampling. Unlike head-based
sampling, which makes decisions when spans are created, the Collector's
tail_sampling processor
waits until more of the trace has arrived before deciding whether to retain it.
This lets you prioritize traces containing errors, slow requests, or operations
matching specific attributes while discarding less interesting traffic.
The Collector can also derive metrics directly from spans. The
spanmetrics connector
generates request rate, error rate, and duration (RED) metrics from tracing
data, without requiring separate metric instrumentation. It can also attach
exemplars that link metric measurements back to individual traces.
Similarly, the
servicegraph connector
correlates client and server spans to generate service dependency metrics,
helping observability backends visualize which services communicate and how
those interactions perform.
For most production deployments, the Collector is worth introducing. It extends tracing beyond basic span collection and gives you a central place to manage telemetry processing and routing without modifying application code.
Best practices for OpenTelemetry tracing
Getting traces flowing is only the beginning. Keeping them useful in production requires deliberate decisions about instrumentation, metadata, and sampling.
1. Ensure trace completeness
A trace is only as useful as the spans it contains. For end-to-end visibility, you must instrument every service along critical request paths and ensure trace context propagates correctly across service boundaries.
Missing instrumentation creates blind spots, while broken context propagation fragments a single request into disconnected traces. Both make it harder to pinpoint where latency or failures originate, which leaves you without the evidence you need to find the root cause during an incident.
2. Configure resource attributes consistently
Resource attributes identify the service or environment that produced a span. Without them, even well-instrumented applications can be difficult to distinguish in a distributed system.
At minimum, explicitly configure service.name. Adding service.version and
deployment.environment.name makes it easier to correlate failures with
deployments and distinguish production traffic from staging or development.
You can configure these attributes through OpenTelemetry environment variables:
export OTEL_SERVICE_NAME=checkout-service
export OTEL_RESOURCE_ATTRIBUTES="service.version=2.4.1,deployment.environment.name=production"In Kubernetes environments, the Collector's k8sattributes processor can enrich spans with pod, namespace, and node metadata, providing additional context without modifying application code.
3. Follow semantic conventions
OpenTelemetry defines
semantic conventions
for describing common operations using standardized attributes such as
http.request.method, http.response.status_code, and db.query.text.
These conventions allow OpenTelemetry backends to interpret telemetry consistently, support pre-built dashboards, and correlate data across signals. Most auto-instrumentation libraries already implement relevant conventions, though support varies by library and version.
When instrumenting applications manually, prefer established semantic conventions over custom attribute names wherever applicable.
4. Capture useful diagnostic context
A trace becomes much more valuable when its spans contain enough context to explain why a request behaved differently from others.
Consider recording attributes such as deployment versions, feature flags, experiment variants, regions, customer tiers, and tenant identifiers where appropriate. These let you narrow an investigation to the exact requests affected by a particular change or condition.
Don't shy away from high-cardinality attributes when they provide genuine diagnostic value, but be mindful of how your backend handles such data. And definitely redact sensitive data before exporting spans from the Collector.
5. Sample traces strategically
High-traffic applications can generate enormous volumes of spans, making it expensive to retain every trace. Sampling controls these costs by reducing the amount of telemetry stored.
Rather than treating every request equally, use tail-based sampling to prioritize failures, slow requests, and other diagnostically valuable traces while retaining only a fraction of routine traffic.
The tradeoff is additional memory and processing overhead in the Collector, along with the need to route related spans to the same sampling instance. For many production systems, that complexity is worthwhile because it reduces tracing costs without sacrificing as much diagnostic value.
Final thoughts
Distributed tracing is ultimately about understanding how a system behaves as a whole, rather than as a collection of isolated services. A slow request is rarely the entire story. What matters is the chain of operations that led to it, the dependencies involved, and the context needed to understand why something went wrong.
OpenTelemetry provides the foundation for that understanding, but instrumentation alone isn't enough. A trace is only as useful as the spans it contains, the context connecting them, and the metadata that gives them meaning. Getting telemetry to flow is the starting point; making it useful for investigating real production failures is the harder and more important work.
Start with zero-code instrumentation to establish broad coverage, then add manual spans around the business operations that matter most. As your tracing matures, use the Collector to control telemetry volume, enrich spans, and retain the traces most valuable for troubleshooting.
Ultimately, the value of tracing isn't measured by how many spans you collect, but by how quickly you can turn them into answers when something goes wrong. And that's where your choice of observability backend matters.
Dash0 is built natively on OpenTelemetry's data model, giving you a unified way to investigate traces alongside logs and metrics, with the rich attributes and relationships needed to understand complex failures.
Start a free trial with Dash0 to see what OpenTelemetry tracing can reveal about your applications.






