[9.5] [APM] Omit HDR percentiles when duration histogram is exponential_histogram (#290874) - #291040
Merged
Merged
Conversation
…togram (elastic#290874) ## Summary - Fixes Latency Distribution going blank or 500ing on OTel/mOTLP metrics, where `transaction.duration.histogram` is mapped as `exponential_histogram` and Elasticsearch rejects HDR (`Percentiles algorithm hdr must not be used with exponential_histogram field`). - Before aggregating, field caps checks whether any metric index maps that field as `exponential_histogram` (including empty mappings: HDR fails on mapping even with 0 hits). If so, HDR is omitted; classic `histogram`-only clusters still use HDR. - Closes elastic#289525 ## Test plan ### Unit - [ ] `node scripts/jest x-pack/solutions/observability/plugins/apm/server/routes/correlations/queries/fetch_duration_percentiles.test.ts` - [ ] `node scripts/jest x-pack/solutions/observability/plugins/apm/server/routes/correlations/queries/has_exponential_histogram_mapping.test.ts` ### Manual: synthtrace + exponential_histogram stub `otel_simple_trace` writes traces (and some OTel metrics) but **not** ECS `transaction.duration.histogram` as `exponential_histogram`. The extra Dev Tools steps below are required to hit the bug path. 1. Start the stack (`yarn es snapshot` / `yarn start`, or your usual local ES+Kibana). 2. Ingest OTel traces (gives a service/transaction to open in APM; other latency charts should already have data): ```bash node scripts/synthtrace otel_simple_trace --local --from=now-1h --to=now ``` 3. In Console, create a high-priority template so `metrics-repro.otel-*` is not swallowed by `metrics-otel@template`, then recreate the data stream: ```json PUT /_index_template/repro-exp-histo { "index_patterns": ["metrics-repro.otel-*"], "priority": 1000, "data_stream": {}, "template": { "mappings": { "properties": { "@timestamp": { "type": "date" }, "processor.event": { "type": "keyword" }, "service.name": { "type": "keyword" }, "service.environment": { "type": "keyword" }, "transaction.name": { "type": "keyword" }, "transaction.type": { "type": "keyword" }, "transaction.duration.histogram": { "type": "exponential_histogram" } } } } } DELETE /_data_stream/metrics-repro.otel-default PUT /_data_stream/metrics-repro.otel-default ``` 4. Index a metric doc **inside the APM time picker** (Last 15 minutes / Last 1 hour). Use a UTC timestamp a few minutes ago, not a future time. `service.environment` must be keyword and match the synthtrace environment: ```json POST /metrics-repro.otel-default/_doc?refresh=true { "@timestamp": "<UTC now minus a few minutes>", "processor.event": "metric", "service.name": "sendotlp-otel-native-synth", "service.environment": "Synthtrace: otel_simple_trace", "transaction.name": "oteldemo.AdServiceSynth/GetAds", "transaction.type": "request", "transaction.duration.histogram": { "scale": 0, "sum": 1000, "min": 100, "max": 2000, "zero": { "count": 0 }, "positive": { "indices": [10], "counts": [5] } } } ``` 5. In APM: **Traces** → service `sendotlp-otel-native-synth` → transaction `oteldemo.AdServiceSynth/GetAds` → **Latency Distribution** (same time range as the stub). Environment: `Synthtrace: otel_simple_trace` or All. 6. **Before this PR:** chart is empty or toasts `An error occurred fetching the overall latency distribution.` Inspect `get_duration_percentiles` has `hdr` and may show shard failures. 7. **After this PR:** - Chart shows distribution bars (the failed-transactions overlay can still be empty; the stub has no `event.outcome: failure`). - Inspect `get_duration_field_caps` returns `exponential_histogram` for `transaction.duration.histogram`. - Inspect `get_duration_percentiles` has `"percentiles": { "field": "transaction.duration.histogram" }` **without** `hdr`, `hits.total >= 1`, numeric `95.0`. - Follow-up requests `get_duration_histogram_range_steps` and `get_duration_ranges` run. 8. Optional classic check: without the stub (or on an Elastic-agent synthtrace scenario that only has `histogram` mappings), Inspect should still send `hdr` on `transaction.duration.histogram`. (cherry picked from commit 6a55818)
kibanamachine
enabled auto-merge (squash)
September 15, 2026 08:26
botelastic
Bot
added
the
Team:obs-presentation
2 tasks
|
Pinging @elastic/obs-presentation-team (Team:obs-presentation) |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
kibanamachine
added
the
reviewer:libra
Contributor
Author
Selected for Libra reviewThis PR was selected for Libra review as part of the temporary 50% trial. To opt out permanently, remove the |
Contributor
Author
💛 Build succeeded, but was flaky
Failed CI StepsMetrics [docs]
|
MiriamAparicio
approved these changes
Sep 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport
This will backport the following commits from
mainto9.5:Questions ?
Please refer to the Backport tool documentation