The last thing I measured in the Cardinality Guardian was histograms, and it doesn’t touch them. Two hundred histogram identities went in and two hundred came out, in every enforcement mode. In this lab that’s 2,200 Prometheus series, because each identity brings nine buckets, a sum and a count.
A counter identity is one series. A histogram identity here is eleven, so the same label explosion costs eleven times more. Prometheus keeps every active series in memory, and most vendors bill per active series or per custom metric.
So I added a suite to the same lab. It sends one delta histogram, where each export carries only what happened since the previous one. It has 200 identities (each unique combination of labels), with an unbounded user.id next to a bounded method and status code. Then it cuts that down seven ways, counts the series, and runs four fixed PromQL queries. How many observations the metric carries, the slowest p95 across HTTP methods, the observations with status 500, and the p95 for one named user. The workload always records 600 observations, so any other total means the strategy changed the answer. The per-user p95 is the query someone runs when one customer says things are slow, and it’s why user.id ended up in a histogram in the first place.
None of these reduces identities without losing that per-user question, because the identities are what tell one user from another. One option leaves every identity in place and makes each one cheaper.
What I measured #
Local Docker. Collector 0.162.0 verified against its release checksum, Prometheus 3.15.0 by image digest, Python SDK 1.45.0. Three repetitions per strategy, and every repetition agreed.
| Strategy | Series | Total count | Status 500 | p95 per method | p95 for one user |
|---|---|---|---|---|---|
| Baseline, no processor | 2,200 | 600 | 120 | answers | answers |
| Cardinality Guardian | 2,200 | 600 | 120 | answers | answers |
transform, delete_key | 220 | 60 | 12 | answers | none |
transform, aggregate_on_attributes | 220 | 600 | 120 | answers | none |
filter on the attribute | 0 | none | none | none | none |
| SDK view limiting attributes | 220 | 600 | 120 | answers | none |
tag_only plus routing connector | 2,200 | 600 | 120 | answers | answers |
| Exponential, scraped as text | 600 | 600 | 120 | none | none |
| Exponential, scraped as native | 200 | 600 | 120 | answers | answers |
The percentile columns say whether the query returns a number, not which one. The values are synthetic and the exact figures are in REPORT.md. The Cardinality Guardian row is the reference from the last post, in strip_and_reaggregate.
delete_key changed my numbers #
Deleting the offending key is the first thing I’d reach for.
transform:
metric_statements:
- delete_key(datapoint.attributes, "user.id")
It cuts the series, 2,200 down to 220. It also turned 600 observations into 60, and the 120 requests with status 500 into 12.
The statement runs once per data point and merges nothing, so two hundred data points reach the Prometheus exporter carrying twenty distinct identities. The exporter’s accumulator.go at v0.162.0 handles a delta histogram by comparing the start of the incoming point with the end of the previous point for the same identity. If they match, it adds them. If they don’t, and the new point isn’t later, it drops the point and logs a warning. After delete_key, ten points from the same burst land on one identity with the same start and end, and nine are dropped. Sixty is twenty identities times three observations.
The Collector log from every delete_key run carries Dropped misaligned histogram datapoint at warn level, eleven lines per run because the Collector samples repeated lines. None of the other ten strategies logs a warning, and the suite counts those lines in REPORT.md. The metric itself looks fine. The dashboard renders and the p95 moves a little, 9.375 against the baseline’s 9.107. Twelve in 60 is 20%, and 120 in 600 is 20%, so a ratio-based SLO would flag nothing. What’s off by ten is absolute volume, the traffic-drop alert, capacity planning, any requests-per-minute panel. The ratio held because the drop was uniform in this workload. In production it depends on which points collide.
The version that works runs in the metric context and aggregates.
transform:
metric_statements:
- context: metric
statements:
- aggregate_on_attributes("sum", ["environment", "region", "http.method", "http.status_code"])
Same 220 series, and the count is back to 600. Upstream documents that only sum is valid for histograms, which makes sense, since a bucket is a count.
filter does a third thing. A condition on an attribute drops the data points that carry it, and a metric with no data points goes too. Every data point here carries user.id, so the whole metric left the pipeline. Zero series, four empty queries. The component is doing its job. It filters on the attribute, and I’d read it as filtering the attribute out.
The view #
A view is a rule in the service’s SDK config that says which attributes an instrument keeps before exporting.
View(
instrument_name="lab.bench.delta_histogram",
attribute_keys={"environment", "region", "http.method", "http.status_code"},
)
Twenty identities, 220 series, 600 observations, and the Collector never receives the other 180 identities. I like that it lives in the code of the team that added user.id and knows why. It also needs a deploy of every service, and a platform team can’t apply it centrally while an explosion is in flight.
Routing #
The upstream README suggests pairing tag_only with a routing connector to send marked metrics somewhere cheap, and I repeated that in the last post without running it. It works and reduces nothing. 2,200 series in, 220 to the main exporter and 1,980 to the second. The expensive backend stops seeing the explosion, and the explosion is still there.
My first condition routed nothing.
routing:
table:
- condition: datapoint.attributes["otel.metric.overflow"] == "true"
No error, valid config. The processor writes the marker with PutBool, and Prometheus renders booleans as the label value true, so I’d written the condition against what looked like a string. A boolean compared with a string is always false. Dropping the quotes fixes it. The lab keeps both versions, because the broken one is the one I’d have shipped.
The exponential histogram #
An exponential histogram replaces explicit bucket boundaries with a scale. Here it comes from the Python SDK, with ExponentialBucketHistogramAggregation and no processor on the Collector.
Scraped as text, the way this lab scrapes by default, the 200 identities produce 600 series and the p95 queries return nothing. The Collector’s Prometheus exporter converts exponential histograms to the native format, a text scrape can’t carry that, and what arrives is a histogram with only its +Inf bucket. I’d have published that row as “exponential histograms don’t help” if I’d stopped there.
With scrape_native_histograms: true on the scrape job, the same data lands as 200 native histograms, where one series carries every bucket inside each sample. One series per identity instead of eleven, and all four queries answer, including the per-user percentile. No identity was removed. Whether that’s a saving depends on the bill. A backend that counts label combinations sees no change, and one that counts series sees eleven become one.
The percentile values also say something about the column the other strategies lose. In the classic histogram, the p95 for one user came out at 4.750 for three observations worth exactly 1.0. histogram_quantile assumes observations spread evenly inside the bucket the quantile lands in, my lowest bucket runs from 0 to 5, and 95% of 5 is 4.75. The native histogram returns 1.000. Per method it’s 9.107 classic against 6.999 native, for observations worth 6 and 7. With buckets this coarse, the per-user p95 that the view and aggregate_on_attributes lose was already nearly five times the real value.
What I’d do with this #
The view prevents the series, and the transform contains one that already exists, which is what I need at 3am when I don’t own the service. I’d put aggregate_on_attributes in the platform pipeline and never delete_key, which would have left my request counts wrong with only an exporter warning to say so. I’d move the expensive histograms to exponential buckets with native scraping, after checking what my real backend does with native histograms, since mine is Prometheus in a lab.
The part I don’t have a tool for is that the fast fix in the platform means the team that owns the service may never hear their histogram was carrying a user id.
What this lab doesn’t tell you #
One machine, Docker Compose, 200 identities. Not a capacity model, and the CPU samples are context only because the generator and the Collector share a host. The histogram values are synthetic and repeat across identities.
Everything is delta temporality. A cumulative histogram takes a different branch in the same exporter, which keeps the point with the latest timestamp and drops the rest without a log line. I didn’t measure delete_key there. The code suggests a similar loss, without the warning.
Native histograms are Prometheus 3.15.0 with scrape_native_histograms on one job. Other backends ingest them differently or not at all, and I didn’t measure the remote write path. The view and exponential rows are the Python SDK only. I didn’t test the strategies in combination.
Run it #
git clone https://github.com/cauebraz/opentelemetry-cardinality-lab
cd opentelemetry-cardinality-lab
just setup
just check
just benchmark-histograms
The suite takes about nine minutes and writes results/histograms/ with the report, the raw JSON behind every cell above and the Collector logs from each run. The strategies are plain config files in config/.
If your workload gives different numbers, open an issue in the lab repository with the strategy and the results.json. I’d like to know which of these rows only hold in my lab.