The OpenTelemetry Collector has a Cardinality Guardian processor, still marked alpha. It watches label growth and acts when a label starts producing values nobody planned for. I wanted to know which of my metrics it would hold before I put it near a pipeline I care about.

So I built a small lab with a Collector, a Prometheus I can count series in, a label generator, and a second Collector without the processor so every number has a baseline.

In tag_only, the default mode, 200 exploding identities stayed 200 series. The processor marked the explosion and every series reached the backend anyway. Of the eight OTLP metric shapes I sent, two get reduced, Delta Sum and Gauge. Histograms, exponential histograms, summaries and cumulative sums leave the processor at full cardinality in all three modes. Upstream documents this. I read past it the first time.

200 -> 200tag_only, defaultDetects the explosion and forwards every series.
200 -> 24Delta Sum, GaugeThe only two shapes reaggregated by enforcing modes.
200 -> 200Six shapes without reaggregationFull cardinality in all three modes.

The setup #

Everything runs locally in Docker. The Collector is pinned to 0.162.0 and verified against its release checksum, Prometheus to 3.15.0 by image digest. Metrics go in over OTLP gRPC with delta temporality, and I count what comes out of the Prometheus exporter.

processors:
  cardinality_guardian:
    max_cardinality_delta_per_epoch: 20
    epoch_duration_seconds: 10
    enforcement_mode: tag_only
    never_drop_labels:
      - region
      - environment
      - service.name
    top_offenders_count: 20
    max_tracker_count: 10000

Twenty new values per ten seconds is the smallest setting that triggers the processor without production traffic. Don’t carry it into a real pipeline.

Nine correctness runs assert exact series, label and marker counts, and 84 benchmark runs compare against the Collector with no processor. Every scenario runs three times because the cardinality estimate underneath is probabilistic.

tag_only only detects #

Each scenario sends 200 data points. The exploding ones carry a unique user.id or a unique order path per point. The control carries four fixed routes.

ModeStable routeuser.id counterRaw route counterHistogram
tag_only4200200200
overflow_attribute42121200
strip_and_reaggregate42121200

In tag_only, 180 of the 200 series carry otel.metric.overflow=true and all 200 still carry the original unbounded label. The upstream README says tag_only doesn’t protect the backend on its own and needs a downstream routing connector to send the tagged metrics somewhere cheap. So the safest mode to turn on is also the one that changes nothing about what you store.

Twenty values arrive before the growth threshold is crossed and keep their own identity. Everything after collapses into one. That’s the 21.

Only two shapes collapse #

Same workload, 200 unique values, every OTLP shape sent together. The six in the last row are Cumulative Sum, delta and cumulative histogram, delta and cumulative exponential histogram, and Summary.

Metric shapebaselinetag_onlyoverflow_attributestrip_and_reaggregate
Delta Sum2002002424
Gauge2002002424
The other six shapes200200200200

Merging data points that now share an identity is only safe for a Delta Sum, where you add the values, and a Gauge, where the newest timestamp wins. Cumulative streams need state the processor doesn’t keep, histogram buckets need aligned boundaries, and quantiles can’t be merged at all. Those types fall back to tagging instead of producing a wrong number, and I’d rather have that. The 24 is twenty survivors plus one collapsed identity per value of the bounded http.method label, which has four.

A classic histogram at 200 identities is 2,200 Prometheus series in this lab once buckets, sum and count are included, and a summary is 1,000. Neither moves in any mode. If your expensive metrics are latency histograms, and in my experience they usually are, this processor doesn’t touch them yet.

The epoch limit only sees growth #

I read max_cardinality_delta_per_epoch and assumed each epoch hands out a fresh allowance of twenty values, so a steady stream of new values would be clipped to about twenty per epoch forever. To check, I sent 200 new values per epoch across three epochs, 600 in total, and compared it with 2,000 unique values inside one epoch.

Workloadbaselinetag_onlyoverflow_attributestrip_and_reaggregate
2,000 unique inside one epoch200020002424
200 new per epoch across three epochs600600424424

The burst lands at 24. The steady workload keeps 424 of its 600 identities. The first epoch is clipped to 24 and epochs two and three pass through untouched, 24 plus 200 plus 200.

The processor enforces when the current epoch’s estimate exceeds the previous one by more than the limit, (currCount - prevCount) > p.getLimit(metricName). My workload swapped 200 identities for 200 it had never seen each epoch, so the estimate stayed near 200 and the difference near zero. The upstream design notes say this is on purpose. A label with fifty thousand stable values is fine, and growth since the last epoch is the signal. My model would have had me expecting protection from a steady drip the processor is designed to ignore.

What I’d do with this #

First I’d list which of my expensive metrics are Delta Sums or Gauges. If they’re mostly latency histograms, this isn’t the control I need yet. If I did deploy it, tag_only would go in with a routing connector from the start, never_drop_labels would be set before the first rollout, and epoch_duration_seconds would be longer than a normal deploy burst so ordinary traffic doesn’t look like growth. The estimated savings counter multiplies a dollar figure I configured by a count, so the number it reports came from my own configuration.

The case I run into most is one exploding label per quarter from the same team. The gate holds each one as a burst, and nothing in it knows the label it just collapsed is the third from that service.

What this lab doesn’t tell you #

It runs on one machine under Docker Compose, so none of it is a capacity model, and the CPU and memory samples can’t be attributed to the processor because the generator shares the host. Upstream has Go microbenchmarks for that.

The estimate uses HyperLogLog++ with roughly 0.81% standard error, so counts near the threshold move a little between runs. Every number here is a median across three repetitions, with the range kept in the raw results.

The Collector’s Prometheus exporter converts exponential histograms to the native format and this lab scrapes them as text, so only the +Inf bucket arrives and none of this covers native histogram handling. The next post turns native scraping on and measures it. Each run starts a fresh stack, so nothing covers drift over hours or restarts, and I measured one threshold and one epoch duration.

Three Collector counters I wanted weren’t exposed at the telemetry level I used. The results record them as null, because a zero would claim the processor passed nothing through.

Run it #

git clone https://github.com/cauebraz/opentelemetry-cardinality-lab
cd opentelemetry-cardinality-lab
just setup
just check
just experiment
just benchmark

just experiment produces the first table and just benchmark the rest. Everything is in opentelemetry-cardinality-lab, including the raw JSON behind every table above and the assertions that fail the run when upstream behavior changes. The processor itself is documented in the contrib repository.

If you run it against a workload shaped differently from mine and the numbers don’t match, open an issue in the lab repository with what you sent and what came out, and I’ll check whether my assertions are wrong.