Skip to content

fix(metrics): encode units in V3 series payloads#1761

Merged
rayz merged 1 commit into
tobz/datadog-metrics-v3-payload-supportfrom
rayz/metrics-v3-unit-encoding
May 28, 2026
Merged

fix(metrics): encode units in V3 series payloads#1761
rayz merged 1 commit into
tobz/datadog-metrics-v3-payload-supportfrom
rayz/metrics-v3-unit-encoding

Conversation

@rayz
Copy link
Copy Markdown
Contributor

@rayz rayz commented May 28, 2026

Summary

This pr adds metric units encoding into V3 series payloads to match the Datadog Agent V3 serializer.

This change keeps sketch behavior unchanged: V3 sketches do not encode units, matching the Agent
sketch path.

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

How did you test this PR?

unit tests / ci

References

@rayz rayz changed the base branch from main to tobz/datadog-metrics-v3-payload-support May 28, 2026 19:26
@dd-octo-sts dd-octo-sts Bot added area/components Sources, transforms, and destinations. encoder/datadog-metrics Datadog Metrics encoder. labels May 28, 2026
@rayz rayz marked this pull request as ready for review May 28, 2026 19:26
@rayz rayz requested a review from a team as a code owner May 28, 2026 19:26
Copilot AI review requested due to automatic review settings May 28, 2026 19:26
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds metric unit encoding to V3 series payloads, matching the Datadog Agent V3 serializer. Units are interned into a new dictionary and stored sparsely via a per-metric FLAG_HAS_UNIT bit and a parallel unit_refs column. Sketches deliberately do not encode units, mirroring Agent behavior.

Changes:

  • Introduces FLAG_HAS_UNIT, a unit interner, dictionary bytes, and sparse unit_refs column in V3Writer, plus a V3MetricBuilder::set_unit API.
  • Adds DICT_UNIT_STR_FIELD_NUMBER (25) and UNIT_REFS_FIELD_NUMBER (26) constants and serializes the new fields in finalize.
  • Wires metric.metadata().unit() into the V3 writer for non-sketch metrics and adds unit tests + payload-level integration tests.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
lib/saluki-components/src/encoders/datadog/metrics/v3/constants.rs Adds new V3 protobuf field numbers for unit dictionary and unit refs.
lib/saluki-components/src/encoders/datadog/metrics/v3/writer.rs Implements sparse unit interning, encoding, delta-encoding, and serialization with new set_unit builder API and tests.
lib/saluki-components/src/encoders/datadog/metrics/mod.rs Sets units on V3 builder for non-sketch metrics; adds payload-level tests verifying sparse unit refs and sketch exclusion.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented May 28, 2026

Binary Size Analysis (Agent Data Plane)

Baseline: 1ad7cde · Comparison: 935bba5 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 37.65 MiB (baseline) vs 37.97 MiB (comparison)
Size Change: +328.72 KiB (+0.85%)

✅ Binary size difference within threshold

Changes by Module
Module File Size Symbols
saluki_components::encoders::datadog +70.15 KiB 352
core +53.94 KiB 14504
anyhow +40.05 KiB 1699
alloc +34.54 KiB 2403
hyper +28.29 KiB 580
figment +24.37 KiB 700
saluki_components::sources::otlp -21.89 KiB 236
hashbrown +16.21 KiB 1100
prost -14.46 KiB 515
saluki_components::common::datadog +13.66 KiB 440
chrono -13.08 KiB 23
hyper_util -12.11 KiB 137
http +11.84 KiB 419
[sections] +11.25 KiB 9
tower +10.78 KiB 318
tracing -10.19 KiB 175
axum +9.75 KiB 422
serde_core +9.49 KiB 888
tokio_rustls +9.28 KiB 93
rustls -8.78 KiB 41
Detailed Symbol Changes
    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +1.6%  +327Ki  +1.6%  +258Ki    [46991 Others]
  [NEW]  +136Ki  [NEW]  +136Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::hf265beba717a8ffb
  [NEW] +66.7Ki  [NEW] +66.5Ki    saluki_core::topology::built::BuiltTopology::spawn::_{{closure}}::hef6f38503239c166
  [NEW] +66.1Ki  [NEW] +66.0Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::hdcdf70adbde87ffc
  [NEW] +58.7Ki  [NEW] +58.5Ki    agent_data_plane::internal::env::workload::build_collector::_{{closure}}::ha56b75595a60c965
  [NEW] +58.2Ki  [NEW] +58.0Ki    saluki_core::topology::blueprint::TopologyBlueprint::build::_{{closure}}::hacbbeaf29f1d76dc
  [NEW] +57.1Ki  [NEW] +56.9Ki    agent_data_plane::internal::env::ADPEnvironmentProvider::from_configuration::_{{closure}}::hb77df6cc347f39b1
  [NEW] +56.7Ki  [NEW] +56.5Ki    agent_data_plane::cli::debug::handle_debug_command::_{{closure}}::ha2e6796c57380dae
  [NEW] +56.0Ki  [NEW] +55.8Ki    agent_data_plane::cli::dogstatsd::handle_dogstatsd_command::_{{closure}}::hb45f834f247c7f00
  [NEW] +49.8Ki  [NEW] +49.5Ki    agent_data_plane::main::_{{closure}}::hafe55c655b44bc0f
  [NEW] +49.7Ki  [NEW] +49.6Ki    core::ops::function::FnOnce::call_once::he29f19ec791745c2
  [DEL] -48.9Ki  [DEL] -48.8Ki    core::ops::function::FnOnce::call_once::h669ff363e842dff3
  [DEL] -49.7Ki  [DEL] -49.4Ki    agent_data_plane::main::_{{closure}}::h7d49a9469c34b899
  [DEL] -56.5Ki  [DEL] -56.3Ki    agent_data_plane::cli::dogstatsd::handle_dogstatsd_command::_{{closure}}::he79b0808df75cda1
  [DEL] -56.6Ki  [DEL] -56.4Ki    agent_data_plane::cli::debug::handle_debug_command::_{{closure}}::h7c655b6902e2f89a
  [DEL] -57.1Ki  [DEL] -56.9Ki    agent_data_plane::internal::env::ADPEnvironmentProvider::from_configuration::_{{closure}}::ha0920a2b79b02cce
  [DEL] -57.8Ki  [DEL] -57.6Ki    saluki_core::topology::blueprint::TopologyBlueprint::build::_{{closure}}::h32219546faa6f65a
  [DEL] -58.4Ki  [DEL] -58.2Ki    agent_data_plane::internal::env::workload::build_collector::_{{closure}}::ha8e6b4b93de14a84
  [DEL] -66.2Ki  [DEL] -66.0Ki    saluki_core::topology::built::BuiltTopology::spawn::_{{closure}}::hfe1e9d26bba4c0cd
  [DEL] -66.3Ki  [DEL] -66.1Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::h9611a56813a60eb8
  [DEL]  -136Ki  [DEL]  -136Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::h4399bd59161f275e
  +0.9%  +328Ki  +0.8%  +259Ki    TOTAL

@rayz
Copy link
Copy Markdown
Contributor Author

rayz commented May 28, 2026

/merge

@gh-worker-devflow-routing-ef8351
Copy link
Copy Markdown

gh-worker-devflow-routing-ef8351 Bot commented May 28, 2026

View all feedbacks in Devflow UI.

2026-05-28 19:40:47 UTC ℹ️ Start processing command /merge


2026-05-28 19:40:48 UTC ℹ️ MergeQueue: Stack detected — automatically merging the entire stack.


2026-05-28 19:40:58 UTC ℹ️ Devflow: Merging stacked PRs in the following order:

  1. enhancement(metrics): support for metrics v3 protocol (#1175)
  2. fix(metrics): encode units in V3 series payloads (#1761)

Progress updates will be sent here directly.


2026-05-28 19:40:59 UTC ℹ️ Devflow: Merging enhancement(metrics): support for metrics v3 protocol (#1175)...


2026-05-28 19:41:01 UTC ❌ MergeQueue

merge method 'merge' is not allowed for this repository. Supported methods are: [squash, rebase]

@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented May 28, 2026

Regression Detector (Agent Data Plane)

Run ID: 0dcd4722-02aa-4be6-98e7-a8970e01394a
Baseline: 1ad7cde2 · Comparison: 935bba59 · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (35)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment goal Δ mean % links
dsd_uds_10mb_3k_contexts_cpu (erratic) cpu ⚪ +6.29 metrics profiles logs
otlp_ingest_metrics_5mb_memory memory ⚪ +1.66 metrics profiles logs
quality_gates_rss_idle memory ⚪ +1.14 metrics profiles logs
otlp_ingest_traces_5mb_memory memory ⚪ +0.79 metrics profiles logs
dsd_uds_512kb_3k_contexts_memory memory ⚪ +0.69 metrics profiles logs
quality_gates_rss_dsd_low memory ⚪ +0.65 metrics profiles logs
dsd_uds_100mb_3k_contexts_memory memory ⚪ +0.57 metrics profiles logs
dsd_uds_10mb_3k_contexts_memory memory ⚪ +0.50 metrics profiles logs
quality_gates_rss_dsd_medium memory ⚪ +0.47 metrics profiles logs
dsd_uds_500mb_3k_contexts_memory memory ⚪ +0.39 metrics profiles logs
dsd_uds_1mb_3k_contexts_memory memory ⚪ +0.32 metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_memory memory ⚪ +0.30 metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_throughput throughput ⚪ -0.25 metrics profiles logs
quality_gates_rss_dsd_heavy memory ⚪ +0.21 metrics profiles logs
dsd_uds_500mb_3k_contexts_throughput throughput ⚪ -0.10 metrics profiles logs
dsd_uds_100mb_3k_contexts_cpu (erratic) cpu ⚪ +0.09 metrics profiles logs
otlp_ingest_metrics_5mb_throughput throughput ⚪ -0.02 metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_cpu (erratic) cpu ⚪ +0.01 metrics profiles logs
dsd_uds_10mb_3k_contexts_throughput throughput ⚪ -0.00 metrics profiles logs
dsd_uds_512kb_3k_contexts_throughput throughput ⚪ -0.00 metrics profiles logs
dsd_uds_1mb_3k_contexts_throughput throughput ⚪ -0.00 metrics profiles logs
otlp_ingest_logs_5mb_throughput (ignored) throughput ⚪ -0.00 metrics profiles logs
dsd_uds_100mb_3k_contexts_throughput throughput ⚪ +0.00 metrics profiles logs
quality_gates_rss_dsd_ultraheavy memory ⚪ -0.29 metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_memory memory ⚪ -0.30 metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_throughput throughput ⚪ +0.44 metrics profiles logs
otlp_ingest_traces_5mb_cpu (erratic) cpu ⚪ -0.57 metrics profiles logs
otlp_ingest_logs_5mb_cpu (ignored) cpu ⚪ -0.85 metrics profiles logs
dsd_uds_500mb_3k_contexts_cpu (erratic) cpu ⚪ -0.91 metrics profiles logs
otlp_ingest_traces_5mb_throughput throughput ⚪ +0.93 metrics profiles logs
otlp_ingest_logs_5mb_memory (ignored) memory ⚪ -1.25 metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_cpu (erratic) cpu ⚪ -1.95 metrics profiles logs
otlp_ingest_metrics_5mb_cpu (erratic) cpu ⚪ -2.33 metrics profiles logs
dsd_uds_512kb_3k_contexts_cpu (erratic) cpu ⚪ -3.45 metrics profiles logs
dsd_uds_1mb_3k_contexts_cpu (erratic) cpu 🟢 -11.83 metrics profiles logs
Bounds Checks: ✅ Passed (5)
experiment check replicates observed links
quality_gates_rss_dsd_heavy memory_usage 10/10 ✅ 123 MiB ≤ 140 MiB metrics profiles logs
quality_gates_rss_dsd_low memory_usage 10/10 ✅ 40.1 MiB ≤ 50 MiB metrics profiles logs
quality_gates_rss_dsd_medium memory_usage 10/10 ✅ 61.7 MiB ≤ 75 MiB metrics profiles logs
quality_gates_rss_dsd_ultraheavy memory_usage 10/10 ✅ 178 MiB ≤ 200 MiB metrics profiles logs
quality_gates_rss_idle memory_usage 10/10 ✅ 27.3 MiB ≤ 40 MiB metrics profiles logs
Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

@rayz rayz merged commit a2ca1c3 into tobz/datadog-metrics-v3-payload-support May 28, 2026
79 checks passed
@rayz rayz deleted the rayz/metrics-v3-unit-encoding branch May 28, 2026 19:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/components Sources, transforms, and destinations. encoder/datadog-metrics Datadog Metrics encoder.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants