enhancement(metrics): support for metrics v3 protocol

[tobz]

Summary

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

How did you test this PR?

References

[tobz]

• enhancement(metrics): support for metrics v3 protocol #1175 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[pr-commenter]

Binary Size Analysis (Agent Data Plane)

Baseline: 1ad7cde · Comparison: f27ed86 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 37.65 MiB (baseline) vs 37.97 MiB (comparison)
Size Change: +327.95 KiB (+0.85%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
saluki_components::encoders::datadog	+83.20 KiB	356
core	+53.09 KiB	14500
anyhow	+39.34 KiB	1699
alloc	+37.12 KiB	2408
hyper	+27.84 KiB	584
saluki_components::sources::otlp	-21.76 KiB	236
hashbrown	+21.32 KiB	1115
figment	+20.92 KiB	700
[sections]	+17.78 KiB	9
piecemeal	-14.05 KiB	31
axum	+13.12 KiB	428
chrono	-13.08 KiB	23
tokio	+11.44 KiB	4515
http	+11.34 KiB	418
serde_core	+10.89 KiB	889
hyper_util	-10.21 KiB	139
http_body_util	-9.12 KiB	208
h2	+9.03 KiB	779
saluki_components::common::datadog	+8.70 KiB	440
tracing	-8.11 KiB	175

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +1.6%  +327Ki  +1.6%  +259Ki    [47045 Others]
  [NEW]  +136Ki  [NEW]  +136Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::hf265beba717a8ffb
  [NEW] +66.7Ki  [NEW] +66.5Ki    saluki_core::topology::built::BuiltTopology::spawn::_{{closure}}::hef6f38503239c166
  [NEW] +66.1Ki  [NEW] +66.0Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::hdcdf70adbde87ffc
  [NEW] +58.7Ki  [NEW] +58.5Ki    agent_data_plane::internal::env::workload::build_collector::_{{closure}}::ha56b75595a60c965
  [NEW] +58.2Ki  [NEW] +58.0Ki    saluki_core::topology::blueprint::TopologyBlueprint::build::_{{closure}}::hacbbeaf29f1d76dc
  [NEW] +57.1Ki  [NEW] +56.9Ki    agent_data_plane::internal::env::ADPEnvironmentProvider::from_configuration::_{{closure}}::hb77df6cc347f39b1
  [NEW] +56.7Ki  [NEW] +56.5Ki    agent_data_plane::cli::debug::handle_debug_command::_{{closure}}::ha2e6796c57380dae
  [NEW] +56.0Ki  [NEW] +55.8Ki    agent_data_plane::cli::dogstatsd::handle_dogstatsd_command::_{{closure}}::hb45f834f247c7f00
  [NEW] +49.8Ki  [NEW] +49.5Ki    agent_data_plane::main::_{{closure}}::hafe55c655b44bc0f
  [NEW] +49.7Ki  [NEW] +49.6Ki    core::ops::function::FnOnce::call_once::he29f19ec791745c2
  [DEL] -48.9Ki  [DEL] -48.8Ki    core::ops::function::FnOnce::call_once::h669ff363e842dff3
  [DEL] -49.7Ki  [DEL] -49.4Ki    agent_data_plane::main::_{{closure}}::h7d49a9469c34b899
  [DEL] -56.5Ki  [DEL] -56.3Ki    agent_data_plane::cli::dogstatsd::handle_dogstatsd_command::_{{closure}}::he79b0808df75cda1
  [DEL] -56.6Ki  [DEL] -56.4Ki    agent_data_plane::cli::debug::handle_debug_command::_{{closure}}::h7c655b6902e2f89a
  [DEL] -57.1Ki  [DEL] -56.9Ki    agent_data_plane::internal::env::ADPEnvironmentProvider::from_configuration::_{{closure}}::ha0920a2b79b02cce
  [DEL] -57.8Ki  [DEL] -57.6Ki    saluki_core::topology::blueprint::TopologyBlueprint::build::_{{closure}}::h32219546faa6f65a
  [DEL] -58.4Ki  [DEL] -58.2Ki    agent_data_plane::internal::env::workload::build_collector::_{{closure}}::ha8e6b4b93de14a84
  [DEL] -66.2Ki  [DEL] -66.0Ki    saluki_core::topology::built::BuiltTopology::spawn::_{{closure}}::hfe1e9d26bba4c0cd
  [DEL] -66.3Ki  [DEL] -66.1Ki    agent_data_plane::cli::run::create_topology::_{{closure}}::h9611a56813a60eb8
  [DEL]  -136Ki  [DEL]  -136Ki    agent_data_plane::cli::run::handle_run_command::_{{closure}}::h4399bd59161f275e
  +0.9%  +327Ki  +0.8%  +260Ki    TOTAL

[pr-commenter]

Regression Detector (Agent Data Plane)

Run ID: 52ed90b6-94fd-4b10-bd0a-aa35ba12811f
Baseline: 1ad7cde2 · Comparison: f27ed862 · diff

Optimization Goals: ✅ No significant changes detected

Fine details of change detection per experiment (35)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
dsd_uds_512kb_3k_contexts_cpu (erratic)	cpu	⚪ +6.03	metrics profiles logs
dsd_uds_10mb_3k_contexts_cpu (erratic)	cpu	⚪ +4.94	metrics profiles logs
dsd_uds_100mb_3k_contexts_cpu (erratic)	cpu	⚪ +1.44	metrics profiles logs
otlp_ingest_logs_5mb_cpu (ignored)	cpu	⚪ +1.38	metrics profiles logs
dsd_uds_1mb_3k_contexts_memory	memory	⚪ +1.37	metrics profiles logs
quality_gates_rss_idle	memory	⚪ +1.33	metrics profiles logs
dsd_uds_100mb_3k_contexts_memory	memory	⚪ +0.98	metrics profiles logs
otlp_ingest_traces_5mb_cpu (erratic)	cpu	⚪ +0.95	metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_cpu (erratic)	cpu	⚪ +0.93	metrics profiles logs
quality_gates_rss_dsd_low	memory	⚪ +0.91	metrics profiles logs
quality_gates_rss_dsd_heavy	memory	⚪ +0.85	metrics profiles logs
dsd_uds_10mb_3k_contexts_memory	memory	⚪ +0.79	metrics profiles logs
dsd_uds_512kb_3k_contexts_memory	memory	⚪ +0.76	metrics profiles logs
dsd_uds_500mb_3k_contexts_throughput	throughput	⚪ -0.69	metrics profiles logs
otlp_ingest_traces_5mb_memory	memory	⚪ +0.48	metrics profiles logs
dsd_uds_500mb_3k_contexts_memory	memory	⚪ +0.35	metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_memory	memory	⚪ +0.25	metrics profiles logs
quality_gates_rss_dsd_medium	memory	⚪ +0.23	metrics profiles logs
dsd_uds_500mb_3k_contexts_cpu (erratic)	cpu	⚪ +0.21	metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_memory	memory	⚪ +0.16	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory	⚪ +0.13	metrics profiles logs
otlp_ingest_logs_5mb_throughput (ignored)	throughput	⚪ -0.03	metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_throughput	throughput	⚪ -0.03	metrics profiles logs
otlp_ingest_metrics_5mb_throughput	throughput	⚪ -0.01	metrics profiles logs
dsd_uds_1mb_3k_contexts_throughput	throughput	⚪ -0.00	metrics profiles logs
dsd_uds_512kb_3k_contexts_throughput	throughput	⚪ -0.00	metrics profiles logs
dsd_uds_100mb_3k_contexts_throughput	throughput	⚪ +0.00	metrics profiles logs
otlp_ingest_traces_5mb_throughput	throughput	⚪ +0.01	metrics profiles logs
dsd_uds_10mb_3k_contexts_throughput	throughput	⚪ +0.01	metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_throughput	throughput	⚪ +0.06	metrics profiles logs
dsd_uds_1mb_3k_contexts_cpu (erratic)	cpu	⚪ -0.46	metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_cpu (erratic)	cpu	⚪ -1.24	metrics profiles logs
otlp_ingest_metrics_5mb_cpu (erratic)	cpu	⚪ -1.65	metrics profiles logs
otlp_ingest_metrics_5mb_memory	memory	⚪ -3.21	metrics profiles logs
otlp_ingest_logs_5mb_memory (ignored)	memory	⚪ -3.78	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 125 MiB ≤ 140 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 40 MiB ≤ 50 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 60.2 MiB ≤ 75 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 176 MiB ≤ 200 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 27.1 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.

[tobz]

This is temporarily blocked on there being a version of the Datadog Agent for us to test against in correctness tests that has up-to-date v3 metrics support.

Currently, we're hitting an issue related to rate intervals being delta encoded when they shouldn't be. That bug is fixed in DataDog/datadog-agent#45825 but won't be released until 7.77: roughly 2 weeks from now before an RC is available to use. We can potentially do a hacky image build or something for keep going in the meantime and then switch back to a proper Agent version once available, we'll see.

[tobz]

We've temporarily handled the issue of correctness tests by using a "dev" container image (datadoghq/agent-dev) based on the latest fix for V3 support in the Datadog Agent. With this in place, we're now currently passing for both dsd-plain (V2 payloads) and dsd-plain-v3 (new, V3 payloads only).

We can't merge this as-is: we need to wait for at least an RC build of Datadog Agent 7.77 so we can pin to a non-development image. In the meantime, I'm going to work on making sure we've integrated all of the same small fixes/changes that have been steadily being made upstream in the Datadog Agent repository for V3 support.

[vickenty]

This doesn't seem to observe any intake payload size limits, or am I missing anything?

[tobz]

That's correct.

Right now, we're either flushing with the V2 encoder determines it needs to flush (so that we generate an equivalent payload in terms of the contained metrics between the two) or if we exceed the configured maximum metrics per payload limit.

In V2/V3 mode, I suppose it's entirely possible to have the V3 payload exceed the payload limits, although it would be incredibly unlikely. In V3 only mode, it's obviously a much more likely risk.

My thought process was that we would improve this -- make V3 encoding aware of the payload limits -- at the same time we added incremental compression to match the behavior of the Agent... since back when this was originally written many weeks ago, it seemed like we'd have enough time between then and "V3 only in production / for customers" to do the follow-up work.

I guess the question I have is: do you feel like we still have that sort of time before we want to be running V3 only?

[vickenty]

Is it intentional that batch_seq and batch_len are hard-coded as 0 and 1?

[tobz]

It is intentional, but only in the context of us not currently splitting V3 payloads: there can literally only ever be a single V3 payload in each batch.

(Mostly related to the question you left about not obeying intake payload size limits.)

[vickenty]

That doesn't sound right: there can be multiple v2 payloads constructed from even a single event buffer, so even if they mach one to one, there should be multiple v3 ones. Or are we using word batch differently here?

[tobz]

There's probably some missing context/mismatched terminology here, yeah.

Every time we receive an "event buffer", we process all of the metrics in the event buffer, incrementally encoding them via the request builder. As we're doing that, we might reach the point where a payload is "full" (aka adding another metric would cause the final payload to exceed the (un)compressed size limits) and we have to flush it before encoding the next metric in the event buffer. That scenario is what I'm referring to here.

Unlike the Core Agent, we don't immediately emit partial payloads for the remainder of an event buffer: if we finish processing an event buffer, and there's still remaining metrics in the request builder, we wait for a period of time for additional event buffers to come in and eventually time out and flush that partial payload.