OpenTelemetry Traces vs Metrics vs Logs: Practical Guide

When a checkout endpoint slows down, a dashboard can show the latency increase without explaining which dependency consumed the time. A trace can expose the request path, but may not tell whether the failure is widespread. A log can preserve the database exception, yet without context it may be impossible to associate that message with the failing request. These are different forms of evidence, not competing products.

OpenTelemetry provides APIs, SDKs, protocol definitions and collection components for generating and transporting traces, metrics and logs. It does not itself supply all of the long-term storage, alerting and visualization features of an observability platform. A useful instrumentation plan begins with the operational question and then chooses the signal, attributes and retention policy that can answer it.

What each telemetry signal is for

A metric series can reveal that the latency of the checkout route is increasing. A trace may show time spent in an inventory call. A correlated log may contain the precise retry or timeout exception. That progression is a diagnostic pattern, not an automatic root-cause guarantee.

The classic Google SRE monitoring guidance frames user-facing service monitoring around latency, traffic, errors and saturation. Translate those signals into instrumented measurements before building visualizations. Choose counters for request and failure totals, histograms for durations and gauges for current resource pressure. Put alerts on symptoms users experience, then use traces and logs for investigation.

Choose the signal by the decision it supports

Begin with a concrete decision rather than an instrumentation inventory. For a checkout service, a product owner might need to know whether customers can complete an order; an on-call engineer needs to know whether errors are isolated to one operation; a developer needs to know which database query or downstream service failed. One signal rarely answers all three questions economically.

Metrics are suitable for monitoring health continuously because they aggregate repeated observations. They are less suitable for explaining a single unusual transaction. Traces preserve request-specific structure, but can be absent when sampling or propagation fails. Logs can carry fine-grained error context, but searching a large body of unconstrained messages may be slower and more expensive than querying a bounded metric.

An observability design should explicitly state its expected investigations: for example, "distinguish a pricing-service timeout from a database lock" or "identify whether a release changed authorization latency." Then define the event, counter, histogram or span required to discriminate the alternatives. This prevents a common failure mode: every service emits plentiful telemetry, yet the operational question has no corresponding dimension or event.

Establish consistent identity before collecting everything

An instrumentation library needs to describe which process emitted the telemetry. OpenTelemetry uses resource attributes such as service.name, service.version and deployment.environment.name. The resource documentation recommends setting the service name explicitly so an application does not show up as an unknown service.

A practical starting point for an instrumented service is:

export OTEL_SERVICE_NAME=checkout-api
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production,service.version=release-42"
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318

These settings are illustrative. Use the endpoint of an accessible Collector or backend and protect traffic across trust boundaries. The OTLP configuration reference distinguishes the base endpoint from per-signal endpoints; with OTLP/HTTP the base endpoint leads exporters to the signal paths, while a per-signal endpoint is used as given.

Resources help answer whether a problem occurs only in one release or environment. Keep their names consistent across applications. An unbounded user identifier is a poor substitute for an application identity.

Resource governance and semantic conventions

A resource is the identity of the emitting component, whereas a span attribute is context about a particular operation. Put stable service identity on resources and request-specific details on spans. A version tag helps compare releases; a request ID belongs in a log or span when privacy policies allow it, not in a metric label.

Semantic conventions standardize names such as HTTP route and network attributes, but the OpenTelemetry semantic convention documentation includes both stable and developing conventions. Do not assume an attribute name copied from an older tutorial remains canonical in every SDK. For Python, the instrumentation documentation documents a versioned opt-in mechanism for migrating certain HTTP and database conventions. Review dashboard queries and alerts when upgrading auto-instrumentation; dual emission during migration can produce duplicate-looking series and extra cost.

Treat resource metadata as configuration owned by the service deployment. A CI deployment can supply the version; platform defaults can supply the environment; the service should still verify what reaches its backend. The configuration shown above describes a single application, not a requirement to expose the Collector listener publicly.

Add distributed traces at meaningful boundaries

A trace is made of spans representing operations. A web framework can automatically create incoming HTTP spans; instrumented clients may add database and outbound HTTP spans. Manual spans become useful at domain boundaries the framework cannot infer: payment authorization, inventory reservation, work-queue processing or a particularly expensive calculation.

OpenTelemetry's Python instrumentation guide distinguishes between instrumenting an application with an SDK and instrumenting reusable libraries through the API. The simplified pattern below presumes a tracer provider and exporter have already been initialized:

from opentelemetry import trace

tracer = trace.get_tracer("checkout.business")

def reserve_stock(cart, inventory):
    with tracer.start_as_current_span("inventory.reserve") as span:
        span.set_attribute("inventory.item_count", len(cart.items))
        return inventory.reserve(cart)

The span records a stable operation name and a bounded diagnostic attribute. It does not record a customer's name, token or complete cart. Instrumentation should not turn a payment endpoint into a new source of confidential data.

Across service calls, traces depend on propagation. The W3C Trace Context requirements describe how an active trace context is transferred through headers including traceparent. A queue message can require propagator injection and extraction by the producer and consumer. For fan-out, batch processing and independent messages, span links may describe the relationship more accurately than a single parent.

Make a test request through multiple services and inspect whether its trace contains the expected segments. A trace that appears to end at the message broker warrants checks of context propagation, consumer instrumentation, sampling, asynchronous trace linkage and retention before attributing the gap to application behavior.

Choosing span boundaries and error semantics

Use one span for work whose start and end are meaningful to a reader. Excessively granular spans, such as one per iteration of a tight loop, add processing and storage while obscuring the operation hierarchy. Conversely, one giant request span may hide the relevant dependency boundary. An application may record an error event and span status for a failed operation; the precise API for exceptions and statuses depends on the language SDK and instrumentation library.

A helpful trace investigation starts from an inbound route span, follows an outbound call to the inventory service, and inspects an internal inventory.reserve span. If the second service creates a new unrelated trace, first inspect propagation before concluding that the network request was never made. If a queue batches many jobs, review whether links between traces convey causality more faithfully than strict parent-child nesting.

A successful HTTP response can still represent a failed business workflow, such as an accepted job that later dead-letters. Application-level spans and explicit outcome attributes can capture the distinction, while service health metrics remain independently defined.

Choose metrics for questions that need aggregation

Request duration needs a distribution, not just an average. Histograms group observed values into buckets; they allow useful aggregate latency queries, subject to resolution and backend support. The Prometheus histogram guide explains why averaging precomputed per-instance quantiles is generally invalid.

The important engineering decision is which attributes define a distinct series. A small set such as service, environment, route template, method and outcome is usually useful. Avoid including full URL paths, request IDs, user IDs or unique transaction identifiers in metric labels.

The OpenTelemetry Metrics SDK defines configurable cardinality limiting and overflow behavior. Those limits protect collection resources but do not rescue an ill-designed metric from losing the distinctions an operator needed. Design the label set before accepting the default emissions of every library.

For a checkout path, create an application-specific counter for completed attempts and a histogram for authorization duration when those concepts are not captured by generic HTTP spans. Keep the metric name, unit and permitted dimensions stable. Validate that exporter and backend agree about temporality and histogram mapping.

A small custom metric example

Once a MeterProvider and exporter are configured, the application API can add measurements without making its business logic dependent on a specific monitoring vendor:

from opentelemetry import metrics

meter = metrics.get_meter("checkout.business")
authorization_duration = meter.create_histogram(
    "checkout.authorization.duration",
    unit="s",
    description="Duration of authorization attempts",
)
authorization_attempts = meter.create_counter(
    "checkout.authorization.attempts",
    description="Number of authorization attempts",
)

def record_authorization(seconds, outcome):
    attrs = {"outcome": outcome}  # e.g. "success", "declined", "error"
    authorization_attempts.add(1, attrs)
    authorization_duration.record(seconds, attrs)

This code describes measurement recording, not provider or export initialization. In a real service, restrict outcome to a documented finite set; do not pass arbitrary exception text as an attribute. Choose a unit that is consistent across services. Instrument once at the logical completion of each attempt, including failures, rather than counting only successful operations.

Exemplars and the aggregation boundary

An exemplar can link a sampled metric observation to an active trace, if the exporter and backend preserve the correlation. It does not make the entire histogram an index over all requests. In addition, OpenTelemetry's .NET exemplar guidance notes that attributes removed from aggregated metric dimensions by a view may still be present in exemplar filtered attributes. Treat exemplar configuration as part of the data-protection review, rather than assuming a metric cardinality filter also redacts everything.

Keep logs structured, small and correlated

OpenTelemetry's Logs Data Model includes severity, body, timestamps, resource and optional trace and span identifiers. A log can exist without an active trace. When an instrumented handler emits one, embedding the active trace context allows the log viewer to link it to the request path.

Structured records should describe meaningful events: payment dependency timed out, queue delivery retried or database connection failed. Capture enough context to discriminate failure modes, but not credentials, request bodies or private customer fields. Avoid dumping full exceptions containing query parameters before checking what they reveal.

For legacy JSON logs, OpenTelemetry's trace-context compatibility guidance defines conventional fields including trace_id and span_id. Implementations and collectors differ, so verify the fields arrive in the actual log backend.

Logging maturity and transition strategies

Telemetry compatibility has several independent layers: a stable data model, a language API, an SDK implementation, an exporter and a working backend integration. As of the Python language status documentation, traces and metrics are marked stable while logging is still categorized as development. The OpenTelemetry Logs SDK specification separately reports stable status; those statements refer to different components and should not be conflated.

Teams already using Python's standard logging or another structured-log library do not need to rewrite every message merely to adopt trace correlation. They can connect a supported handler or log transport, then test that the resulting records preserve timestamp, severity, trace identifiers and useful attributes. If the language integration is changing, a maintained JSON logging flow plus Collector ingestion may offer a safer migration path than replacing production logging all at once.

Make the log event name and severity useful without requiring readers to parse a giant message. Preserve an application error category, retry decision, queue name or stable dependency identifier. Do not turn one routine failed dependency retry into a flood of identical log records when a bounded metric conveys its frequency.

Use a Collector as a policy boundary

The Collector can accept OpenTelemetry Protocol (OTLP) data, perform processing and export to downstream systems. In small local experiments, SDK-to-backend export can be sufficient. A gateway becomes valuable when operators want consistent redaction, retries, buffering, routing and backend configuration.

The Collector configuration model splits each signal's pipeline into receivers, processors and exporters. A component definition has no effect until it is referenced by an enabled pipeline. Order matters: a drop or redaction policy after an external export is too late.

For a production design, separate concerns without unnecessarily multiplying infrastructure: applications send data to an OTLP gateway; the gateway has traces, metrics and logs pipelines; each exports to a compatible backend. Protect network listeners and credentials, and monitor the Collector itself. Queue growth and failed exports can produce apparent gaps in telemetry even while application requests succeed.

An illustrative three-pipeline Collector configuration

The following is a development-only example that accepts OTLP over loopback, protects in-memory processing and writes signals to a debug exporter. It is not a production backend or a turnkey Kubernetes deployment. Install a Collector distribution containing these components, connect applications from the appropriate network namespace, and validate the exact configuration against that distribution.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 127.0.0.1:4317
      http:
        endpoint: 127.0.0.1:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
    spike_limit_mib: 128
  batch:
    timeout: 5s

exporters:
  debug:
    verbosity: basic

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug]

The OTel configuration guide documents the otelcol validate --config=... command. Replace the debug exporter with an authenticated and encrypted exporter for real telemetry and expose listeners only where necessary. Check memory-limiter outcomes and exporter failure counters; if a downstream endpoint is unreachable, application success does not imply successful telemetry delivery.

At scale, separate the agent/edge layer from a gateway layer when the topology or ownership justifies it. A first hop can minimize private data before leaving the host; a gateway can enforce central routing. More stages also create more failure domains. The correct topology depends on volume, tenant isolation, network boundaries and who owns the backends.

Sampling and correlation are not free

Trace sampling keeps storage and transfer bounded by discarding some request journeys. Head sampling makes an early decision. Tail sampling waits to inspect much or all of a trace and can prefer errors or slow requests. The OpenTelemetry sampling guidance explains the trade-off: head sampling is simpler, but cannot promise preservation of failures recognized only later.

Tail sampling requires buffering, policy decisions and adequate capacity. A multi-instance Collector deployment may need trace-aware routing so all spans reach the same sampler, as described in the gateway deployment pattern. Without it, apparently random missing spans may reflect a flawed telemetry topology rather than a broken request.

Metrics remain valuable for measuring the full population when traces are sampled. Metric exemplars can associate individual measurements with trace or span identifiers, enabling a jump from a histogram observation to a related trace where SDK, exporter and backend support it. That link is optional and is not a guarantee that every measurement has a retained trace.

Why sampling does not solve every cost problem

Attribution should be explicit: a high-volume metrics pipeline is not automatically cheaper than sampled traces. Metrics with a large product of attribute values can generate substantial time-series volume, and retaining highly detailed logs can dominate ingestion or storage. The answer requires measuring volumes in the chosen destination.

A practical budget allocates high-fidelity telemetry to operations that can justify it. Preserve aggregate error rates even when individual traces are sampled. Use route templates instead of raw paths for metric dimensions. Keep logs needed for security investigations under an explicit access and retention policy. Inspect backend ingest and billing counters before declaring the savings from a change.

Sampling is also not reversible. If an unsampled trace for an unusual failure was discarded, adjusting the retention period afterward cannot reconstruct it. An operator can choose a higher sampling rate temporarily for a controlled investigation, but should document the added cost and privacy exposure. Head and tail strategies are policy choices, not reliable substitutes for representative testing.

Treat telemetry as a sensitive-data pipeline

Instrumentation can expose URLs, headers, account identifiers and exception details. OpenTelemetry's sensitive-data guidance recommends minimizing collection at source and describes Collector filtering, attribute removal and redaction. Hashing identifiers without a suitable threat model can still permit re-identification.

Apply a deny/allow review to span attributes, metric dimensions, log bodies and baggage. Test sensitive paths, not just normal transactions. Decide who may query detailed traces and logs, where they are retained and which external processors receive them. Sampling lowers volume, but is not a privacy control.

A worked investigation across three signals

Consider a hypothetical checkout system with an API, an inventory service, a payment provider adapter and a queue worker. After a new release, users report intermittent checkout failures. The team begins with a request-error counter split by a stable route template and status class. It confirms a change in the failure rate and identifies the impacted path without retrieving individual customer data.

Next, an engineer opens one retained trace for that path. The parent request span contains a longer-than-usual outbound dependency call; another operation returns quickly. The trace narrows the hypothesis, but a slow span alone cannot distinguish an overloaded service from a network timeout. Correlated structured logs from the adapter show the retry category and timeout type for that trace. A queue-depth gauge indicates whether the failures have begun to delay downstream work.

The investigation checks the deployment version resource, compares the histogram distribution before and after the release, and runs a controlled request with propagation enabled. The result might support rolling back the adapter, adjusting a timeout or investigating connectivity. It cannot prove a single root cause unless the underlying system evidence supports that conclusion. This example illustrates how signal design improves a diagnostic decision; it is not an account of a named production incident.

Failure modes to test explicitly

A telemetry integration check should exercise negative paths, not merely assert that one span arrived:

• Broken context: a request crosses an HTTP or queue boundary and starts a new trace; test carrier propagation and service instrumentation.
• Lost export: a Collector receiver accepts input while its exporter cannot deliver; check queued, failed, sent and dropped telemetry using the available self-observability metrics.
• High cardinality: an instrument emits a user or transaction ID on a metric; verify dimensional rules and cardinality limits.
• Sampling gaps: a known failing trace is discarded under head sampling; make alerts rely on unsampled aggregate signals.
• Schema drift: an SDK upgrade renames an HTTP attribute and dashboards become empty; validate semantic convention changes before rollout.
• Sensitive data: request URLs, exception strings, log bodies or exemplars retain secrets; test the redaction policy using synthetic sensitive values.

For robust checks, query the actual destination, not only the local SDK callback. The OpenTelemetry Demo telemetry tests demonstrate this approach with queries against each signal's backend.

Roll out in slices and test the evidence

An implementation sequence can be incremental:

1. Specify a user journey and the four golden-signal measurements that indicate its health.
2. Assign resource identity and install supported zero-code instrumentation for HTTP and dependencies.
3. Add manual spans for missing domain boundaries and implement propagation across queue or asynchronous interfaces.
4. Add small, bounded metric sets, including histograms for important latency distributions.
5. Emit structured logs and verify they join to spans when context is present.
6. Configure Collector policies and inspect exporter failures, queue pressure and sensitive data.
7. Exercise a controlled slow dependency, failure and retry; inspect the three signals in the real backends.

The official OpenTelemetry Demo supplies a reproducible multi-service reference with mixed automatic and manual instrumentation. Its telemetry tests query trace, metric and log backends to check expected emissions. This is an implementation example, not evidence that every organization will achieve a particular incident-response improvement.

The end state is not a mandate to collect more data. It is a narrower, testable ability to recognize a degraded service, localize a suspicious operation, examine the supporting events and understand when the telemetry itself has gaps.