High-Cardinality Metrics: How to Cut Observability Costs

High-Cardinality Metrics: How to Cut Observability Costs
High-Cardinality Metrics: How to Cut Observability Costs

Metrics become expensive when useful dimensions turn into uncontrolled combinations of labels. A request counter divided by service, method, route and outcome can answer operational questions. The same counter carrying a customer ID, unique request ID and full URL may create a stream of short-lived series that consumes storage and indexing capacity without improving a single alert.

Reducing that cost is not simply a matter of deleting metrics. The objective is to preserve information needed for service-level decisions while moving one-off diagnostic details into traces or logs, where those details are often more appropriate. That requires a clear model of cardinality, an inventory of actual usage, and changes at the right stage of the telemetry pipeline.

What high cardinality actually means

A metric time series is identified by its name and the complete combination of label values attached to it. Prometheus explains that each additional label set creates a time series, with implications for memory, CPU, disk and network.

Consider an illustrative request counter with labels for service, HTTP method, route template and response class. If the possible values are independent, their counts can multiply to a large potential number of combinations. In practice, only observed combinations produce series. A route that cannot accept a particular method does not automatically create a corresponding time series.

Now add a label such as request_id, whose value changes with nearly every request. The counter stops acting like a useful aggregate and approaches one distinct series for each event. Even when old series become inactive, continuous creation creates churn: fresh index entries, new storage work and a shifting diagnostic footprint.

Not all high cardinality is bad. A stable dimension representing an infrastructure instance may be essential for diagnosing a failed node. The question is whether a label's diagnostic value justifies the observed series and the contractual or infrastructure cost.

Measure the problem before changing instrumentation

Start with a baseline rather than an arbitrary series limit. Record active series, the largest metric families, labels with many distinct values, and the rate at which new series appear. Separate these from sample rate, retained history and query load; they influence different parts of the system.

In Prometheus, the documented /api/v1/status/tsdb endpoint reports data including the current head-series count, series by metric name, and distinct label-value counts. It is a starting point for identifying unusually large families. A snapshot cannot tell you whether a series only appeared during a brief deployment, so compare equivalent windows over time.

For Grafana Mimir, the cardinality endpoints can report label-name and label-value information. Those endpoints must be enabled and their in-memory or active counting methods need to be interpreted correctly. A backend's active series is not identical to every series retained in historical blocks.

Managed observability products add billing views. Grafana Cloud's cardinality dashboards expose active metric and label usage, but the documented dashboards focus on a recent two-hour period rather than the complete historical catalog.

Build a working inventory containing the metric owner, label set, series count, recent growth, dashboards, alert rules and known debugging use. An apparently unused metric may still support an infrequent incident investigation. That possibility should trigger review, not indefinite exemption.

Calculate a useful upper bound without mistaking it for a forecast

Suppose a hypothetical counter has four services, five route templates, three methods and two outcome classes. Their product is 120 possible combinations before considering instance and deployment labels. This is a design-space estimate, not a measurement: many possible combinations may never occur. With a per-request identifier added, however, the set of observed combinations can keep expanding with traffic.

A more useful cost worksheet records the observed active series, new series per interval, samples ingested per interval, retention period and the chargeable units in the actual contract. Compare this worksheet before and after a controlled rollout. It is misleading to multiply an advertised per-series rate by a theoretical maximum if the provider bills on a different basis.

Remove unbounded identifiers at the source

The most effective design change is often to replace labels that describe individual events with dimensions that describe groups of events.

Instead of path="/orders/48219", emit a route template such as route="/orders/{id}". Instead of an unbounded user_id label, retain labels such as service, route, status class or a deliberately small workload class when they answer a specific question. Trace IDs, account identifiers and full URLs can remain in appropriately governed diagnostic records.

A small set of deliberate dimensions is easier to maintain than a global rule permitting every available field as a metric label. Before adding a new label, an instrumentation owner should be able to name the alert, SLO, dashboard or operational question it supports.

Do not confuse individual label cardinalities with the cardinality of their combinations. Adding a label does not necessarily multiply every existing series; the actual observed combinations matter. But dimensions with rapidly changing values deserve special scrutiny because their cost may arise from churn even when the instantaneous series count looks ordinary.

Control attributes in OpenTelemetry

If a service uses OpenTelemetry, the Metrics SDK specification provides Views that can retain an allowlist of attribute keys before metric aggregation. This is generally preferable to emitting unbounded combinations and trying to repair their cost after storage.

OpenTelemetry also defines metric cardinality limits. Where neither a View nor reader specifies another value, the specification recommends a default of 2,000 data points for an aggregation. When the limit is reached, measurements beyond the individually tracked combinations are aggregated into a point marked otel.metric.overflow=true.

That overflow point preserves measurements in an aggregate but discards their original attribute combinations. The OpenTelemetry practical explanation from 2026 makes the consequence clear: a filter for an otherwise useful label such as success=false can omit events that were absorbed into overflow. A cap protects resources but is not a substitute for choosing the right attributes.

Create a test with representative traffic, verify the retained attributes, and monitor for overflow before treating a new instrument as production ready. SDK implementation and version behavior still need validation in the deployed language.

Filter or aggregate before storage

For Prometheus scrape targets, metric_relabel_configs can remove costly samples before they are ingested. This is useful for dropping a clearly unnecessary metric family or a narrowly selected set of series. It does not eliminate the work an exporter already performed to produce those metrics.

Removing a label is not automatically the same as correct aggregation. Two counters that differ only by a removed label can become indistinguishable, yet their values cannot be safely summed by blind relabeling. Correct aggregation must respect the metric type and semantics.

An OpenTelemetry Collector can provide a central transformation point. The Collector transform processor documentation describes aggregation by selected attributes, including supported functions and metric types. Such transformations should be tested against the intended totals and distribution semantics.

Prometheus scrape limits also need care. A configured sample_limit is checked after metric relabeling; exceeding it can cause the entire scrape to fail. Limits are safety guardrails, not a substitute for repairing instrumentation.

Example: drop an explicitly disposable metric family

A Prometheus configuration can remove a metric family at ingestion time when its owner has confirmed that it is not needed for operational queries:

scrape_configs:
  - job_name: api
    static_configs:
      - targets: ['api:9090']
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'debug_request_by_id_total'
        action: drop

This is an illustrative configuration, not a blanket recommendation to remove request diagnostics. The exporter still computes and serves the metric, and other scrape jobs could still collect it. A local test should confirm that the target remains healthy and no downstream rule relies on that family. When an important counter needs fewer dimensions rather than complete removal, aggregate by meaningful dimensions in the instrument or an explicitly compatible processing stage.

Avoid treating recording rules as a collection fix

A recording rule can make an expensive dashboard faster by saving a precomputed aggregation. Prometheus recording rules add new series containing query results. Unless the original raw series is also stopped or reduced, this does not remove the ingestion and storage expense of those original series.

A useful pattern is to define the service-level aggregate that should be kept, validate that all alerts and dashboards can use it, and then aggregate before ingestion or drop redundant raw detail once the information-loss decision is explicit. Query performance and raw-series cost are related but separate optimization goals.

Histogram representation is another independent choice. Native Prometheus histograms represent bucket information differently from classic histogram bucket series. They can change the storage and query trade-offs, but deployment support, query compatibility and the desired resolution still matter. They do not fix an unbounded customer identifier attached to every observation.

Verify the actual billing model

A lower series count is not the same as a guaranteed invoice reduction. Self-hosted teams pay through infrastructure and operations; managed platforms may charge for stored series, ingested samples, indexed custom metrics, or other contract units.

For example, Datadog's cardinality-based custom metrics billing documentation calculates billable usage from hourly distinct custom metrics. Its Metrics without Limits approach separates ingestion from the subset of tags retained for querying. Removing a queryable tag can reduce indexed cardinality without reducing the originally ingested cardinality.

Datadog also documents an alternative Metric Name pricing model based on metric names and datapoint volumes. A cost plan based on a different contract's SKU can produce false savings estimates.

Before predicting money saved, identify the precise usage unit, any included allowance, the measurement window, and any charges remaining after a tag-indexing change. Report the infrastructure and billing effects separately.

Distinguish savings from a visual improvement

Four outcomes should be recorded independently: raw series reduced, ingestion volume reduced, query workload reduced, and the provider's billable usage reduced. An attractive dashboard that displays fewer lines proves none of the others. Similarly, a longer scrape interval reduces point frequency but not necessarily the number of distinct series, and may degrade short-window alerting.

Treat churn as its own dimension. An instance count can remain stable while ephemeral container names, build IDs or task identifiers continuously introduce new series. A stable active-series graph alone cannot tell whether the backend is doing avoidable index work. Compare representative busy and quiet periods, especially around rollouts.

Make reductions safe for on-call engineering

Start with a candidate metric whose labels have low operational value and clear ownership. Capture baseline cardinality and sample ingestion, then list the dashboards and alerts that consume it. Test the new metric side by side in staging or a controlled rollout. Confirm that service-wide totals, error-rate denominators, latency aggregations and alert grouping retain the required meaning.

Protect dimensions used for incident localization. If the team must answer which region failed, do not remove region solely because it has several values. If it must investigate one customer request, a trace or structured log may be a better path than permanent per-request metrics.

Grafana's documented Adaptive Metrics workflow examines query usage and recommends aggregations, with review and exemptions. That is a useful operational pattern, but product features and claimed savings are not independent evidence that a particular organization will achieve a given percentage reduction.

A staged implementation checklist

Discovery: Identify the most expensive families and labels, their owners, and the critical queries they support. Include a sample of incident-only queries, not only popular dashboards.

Design: Propose replacement dimensions and document which drill-down questions will move to logs or traces. For counters and histograms, verify that any aggregation preserves the intended mathematical meaning. For gauges, decide whether a sum, maximum or other reduction is actually appropriate.

Rollout: Run old and new paths side by side for a representative window. Compare totals and alert results; inspect whether SDK overflow has appeared; test label selectors that dashboards expect. Where possible, make the change reversible.

Validation: Compare the backend-specific active-series and ingestion measures, then the next applicable billing measurement window. Record changes to query latency or memory only if measured. Finally, assign an owner to watch for recurrence.

This sequence costs some engineering attention, but it is substantially safer than dropping a high-cardinality label in production and discovering during an outage that the one necessary slice has disappeared.

A durable operating model

A successful cleanup can be reversed by the next instrumentation change. Establish metric ownership, review of newly introduced labels, alerts on series growth and overflow, and recurring checks against query usage. Keep an exception process for expensive dimensions with a demonstrated incident-response purpose.

A practical reduction sequence is inventory, classify, redesign, validate and measure again. It makes telemetry cost a design constraint without turning cost control into indiscriminate deletion. The right outcome is fewer unnecessary series and the same—or explicitly accepted narrower—ability to answer operational questions.