feat(datadog): support forwarder_outdated_file_in_days for stale retry file cleanup

[jszwedko]

Summary

Adds support for forwarder_outdated_file_in_days (default 10), matching the core Agent's startup behavior. When disk persistence is enabled, ADP now scans forwarder_storage_path at startup and removes retry-*.json files whose filesystem mtime exceeds the configured age, preventing unbounded disk growth from stale retry data after extended outages. Set to 0 to disable cleanup. The config key is moved from the unsupported registry to the supported forwarder registry.

Closes #1360

Test plan

• cargo check --workspace && cargo check --workspace --tests passes
• cargo test -p saluki-components --lib outdated passes (3 new tests)
• cargo test -p saluki-components --lib config_registry passes
• With forwarder_storage_max_size_in_bytes set and forwarder_outdated_file_in_days: 10, old retry files are removed at startup; recent files and non-retry files are untouched
• With forwarder_outdated_file_in_days: 0, no files are removed

🤖 Generated with Claude Code

[datadog-datadog-prod-us1]

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 2 Pipeline jobs failed

Semantic PR Title Check | Check For Semantic PR Title     

🛟 This job is unlikely to succeed on retry. Please review your pipeline configuration.

Unknown scope 'forwarder' in pull request title. Valid scopes required: agent-data-plane, aggregate, airlock, etc.

DataDog/saluki | run-benchmarks-adp     

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: d24e244 | Docs | Datadog PR Page | Give us feedback!}

[jszwedko]

This matches the Agent's behavior of only checking at start-up.

[pr-commenter]

Binary Size Analysis (Agent Data Plane)

Baseline: b078b0d · Comparison: d24e244 · diff
Analysis Configuration: stripped binaries · Pass/Fail Threshold: +5%
Sizes: 37.93 MiB (baseline) vs 38.04 MiB (comparison)
Size Change: +111.26 KiB (+0.29%)

✅ Binary size difference within threshold

Changes by Module

Module	File Size	Symbols
anyhow	+27.99 KiB	276
piecemeal	+13.00 KiB	15
tokio	+12.78 KiB	226
figment	+10.09 KiB	93
saluki_components::common::datadog	+8.68 KiB	32
core	+6.79 KiB	1317
saluki_common::cache::CacheBuilder<K,V,W,H>	+6.52 KiB	3
saluki_components::encoders::datadog	-6.18 KiB	19
saluki_components::sources::otlp	-5.35 KiB	19
serde_json	+5.33 KiB	13
bytes	+5.32 KiB	23
&mut serde_json	-4.90 KiB	13
saluki_io::net::util	+4.71 KiB	24
serde	+4.57 KiB	6
hyper_util	-4.21 KiB	13
otlp_protos::otlp_include::opentelemetry	-4.12 KiB	78
h2	+4.07 KiB	79
[sections]	-4.05 KiB	8
anon.d933a6d672e016e78db8d7364bd54640.56.llvm.2344071001195518020	+3.99 KiB	1
anon.74ec343e7df4c5afe576d99fcc874370.67.llvm.13153431456681701250	-3.99 KiB	1

Detailed Symbol Changes

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +2.0%  +130Ki  +1.6% +80.6Ki    [6759 Others]
 +12e2% +6.92Ki  [ = ]       0    core::ptr::drop_in_place<http_body_util::combinators::map_err::MapErr<tonic::body::Body,axum_core::error::Error::new<tonic::status::Status>>>::h6b978571211c4eed
  [NEW] +6.88Ki  [NEW] +6.78Ki    h2::proto::connection::Connection<T,P,B>::poll::h44139f071de446d5
  [NEW] +6.63Ki  [NEW] +6.53Ki    h2::proto::connection::Connection<T,P,B>::poll::h5a36798488cd5681
  [NEW] +6.63Ki  [NEW] +6.53Ki    h2::proto::connection::Connection<T,P,B>::poll::h7f780e2c1564a908
  [NEW] +6.45Ki  [NEW]    +340    core::ptr::drop_in_place<http_body_util::combinators::map_err::MapErr<http_body_util::combinators::map_frame::MapFrame<tonic::body::Body,tonic::codec::decode::Streaming<datadog_protos::checks_include::datadog::checks::v1::SendCheckPayloadRequest>::new<tonic::body::Body,tonic_prost::codec::ProstDecoder<datadog_protos::checks_include::datadog::checks::v1::SendCheckPayloadRequest>>::{{closure}}>,tonic::codec::decode::Streaming<datadog_protos::checks_include::datadog::checks::v1::SendCheckPayloadRequest>::new<tonic::body::Body,tonic_prost::codec::ProstDecoder<datadog_protos::checks_include::datadog::checks::v1::SendCheckPayloadRequest>>::{{closure}}>>::h57e6d59f419e6880
  +878% +5.68Ki +11e2% +5.68Ki    saluki_components::sources::dogstatsd::replay::capture::DogStatsDCaptureControl::start_capture::h80b33ec2e2c1b5d8
   +13% +5.62Ki   +13% +5.62Ki    saluki_components::common::datadog::io::run_endpoint_io_loop::_{{closure}}::h4c31dc59c83ba86c
  [NEW] +4.68Ki  [NEW] +4.52Ki    _<core::slice::iter::Iter<T> as core::iter::traits::iterator::Iterator>::fold::h1210d20d3cee0b37
  [NEW] +4.47Ki  [NEW] +4.30Ki    _<saluki_io::deser::framing::NestedFramer<Inner,Outer> as saluki_io::deser::framing::Framer>::next_frame::hab58ad7434044a92
 -82.1% -4.41Ki -85.7% -4.41Ki    saluki_components::common::datadog::transaction::_::_<impl serde_core::ser::Serialize for saluki_components::common::datadog::transaction::Transaction<B>>::serialize::hdd0848d8474f34d4
  [DEL] -4.68Ki  [DEL] -4.52Ki    core::ops::function::impls::_<impl core::ops::function::FnMut<A> for &mut F>::call_mut::h6f18fe8453bdd4e1
 -86.3% -5.20Ki -88.0% -5.20Ki    saluki_components::sources::otlp::metrics::cache::PointsCache::from_config::hc0fcdd066e6ef178
  [DEL] -5.35Ki  [DEL] -5.22Ki    saluki_components::sources::dogstatsd::replay::writer::TrafficCaptureWriter::start_capture::h1c1a09bbc468b634
 -34.1% -5.60Ki -34.6% -5.60Ki    _<saluki_components::encoders::datadog::traces::TraceEndpointEncoder as saluki_components::common::datadog::request_builder::EndpointEncoder>::encode::h854bf035420c3910
  -3.7% -6.26Ki  -3.7% -6.26Ki    [section .rodata]
 -83.5% -6.71Ki -25.0%    -340    core::ptr::drop_in_place<tonic::body::Body>::h62f35fa9bcf1e4d2
  [DEL] -7.25Ki  [DEL]    -341    core::ptr::drop_in_place<http_body_util::combinators::map_err::MapErr<http_body_util::combinators::map_err::MapErr<http_body_util::combinators::map_frame::MapFrame<tonic::body::Body,tonic::codec::decode::Streaming<datadog_protos::checks_include::datadog::checks::v1::SendCheckPayloadRequest>::new<tonic::body::Body,tonic_prost::codec::ProstDecoder<datadog_protos::checks_include::datadog::checks::v1::SendCheckPayloadRequest>>::{{closure}}>,tonic::codec::decode::Streaming<datadog_protos::checks_include::datadog::checks::v1::SendCheckPayloadRequest>::new<tonic::body::Body,tonic_prost::codec::ProstDecoder<datadog_protos::checks_include::datadog::checks::v1::SendCheckPayloadRequest>>::{{closure}}>,tonic::status::Status::map_error<tonic::status::Status>>>::h45015946cfdf719a
  [DEL] -9.01Ki  [DEL] -8.92Ki    h2::server::Connection<T,B>::poll_closed::hae1db10ac11f7523
  [DEL] -9.11Ki  [DEL] -9.01Ki    h2::server::Connection<T,B>::poll_closed::h7fc5e50f37a1a1fe
  [DEL] -9.51Ki  [DEL] -9.42Ki    h2::server::Connection<T,B>::poll_closed::h3f3ed7a68096333d
  +0.3%  +111Ki  +0.2% +61.7Ki    TOTAL

[pr-commenter]

Regression Detector (Agent Data Plane)

Run ID: f0b37c5e-358c-4107-9c76-8ef694e90893
Baseline: b078b0db · Comparison: d24e244c · diff

Optimization Goals: ❌ 2 regressions detected

experiment	goal	Δ mean %	links
otlp_ingest_metrics_5mb_memory	memory	🔴 +8.18	metrics profiles logs
dsd_uds_100mb_3k_contexts_cpu (erratic)	cpu	🔴 +6.50	metrics profiles logs

Fine details of change detection per experiment (33)

Experiments configured erratic: true are tagged (ignored) and skipped when determining which experiments regressed or improved. Experiments which are detected as erratic at runtime are tagged (erratic) to flag that the run's sample dispersion was high, but their regression / improvement signal still counts.

experiment	goal	Δ mean %	links
dsd_uds_10mb_3k_contexts_cpu (erratic)	cpu	⚪ +3.90	metrics profiles logs
otlp_ingest_logs_5mb_memory (ignored)	memory	⚪ +2.62	metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_cpu (erratic)	cpu	⚪ +1.91	metrics profiles logs
otlp_ingest_traces_5mb_throughput	throughput	⚪ -1.78	metrics profiles logs
otlp_ingest_logs_5mb_cpu (ignored)	cpu	⚪ +1.46	metrics profiles logs
dsd_uds_500mb_3k_contexts_throughput	throughput	⚪ -1.19	metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_throughput	throughput	⚪ -1.02	metrics profiles logs
dsd_uds_100mb_3k_contexts_memory	memory	⚪ +0.42	metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_memory	memory	⚪ +0.37	metrics profiles logs
quality_gates_rss_dsd_heavy	memory	⚪ +0.30	metrics profiles logs
quality_gates_rss_dsd_medium	memory	⚪ +0.27	metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_memory	memory	⚪ +0.17	metrics profiles logs
dsd_uds_500mb_3k_contexts_memory	memory	⚪ +0.16	metrics profiles logs
dsd_uds_1mb_3k_contexts_memory	memory	⚪ +0.15	metrics profiles logs
quality_gates_rss_dsd_low	memory	⚪ +0.11	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory	⚪ +0.11	metrics profiles logs
dsd_uds_10mb_3k_contexts_throughput	throughput	⚪ -0.01	metrics profiles logs
dsd_uds_512kb_3k_contexts_throughput	throughput	⚪ -0.00	metrics profiles logs
dsd_uds_1mb_3k_contexts_throughput	throughput	⚪ +0.00	metrics profiles logs
dsd_uds_100mb_3k_contexts_throughput	throughput	⚪ +0.00	metrics profiles logs
otlp_ingest_logs_5mb_throughput (ignored)	throughput	⚪ +0.01	metrics profiles logs
otlp_ingest_traces_ottl_filtering_5mb_throughput	throughput	⚪ +0.01	metrics profiles logs
otlp_ingest_traces_5mb_memory	memory	⚪ -0.02	metrics profiles logs
otlp_ingest_metrics_5mb_throughput	throughput	⚪ +0.03	metrics profiles logs
dsd_uds_512kb_3k_contexts_memory	memory	⚪ -0.03	metrics profiles logs
quality_gates_rss_idle	memory	⚪ -0.14	metrics profiles logs
dsd_uds_10mb_3k_contexts_memory	memory	⚪ -0.28	metrics profiles logs
dsd_uds_500mb_3k_contexts_cpu (erratic)	cpu	⚪ -0.44	metrics profiles logs
otlp_ingest_traces_5mb_cpu (erratic)	cpu	⚪ -0.64	metrics profiles logs
otlp_ingest_traces_ottl_transform_5mb_cpu (erratic)	cpu	⚪ -1.36	metrics profiles logs
dsd_uds_512kb_3k_contexts_cpu (erratic)	cpu	⚪ -2.35	metrics profiles logs
otlp_ingest_metrics_5mb_cpu (erratic)	cpu	🟢 -5.76	metrics profiles logs
dsd_uds_1mb_3k_contexts_cpu (erratic)	cpu	🟢 -6.88	metrics profiles logs

Bounds Checks: ✅ Passed (5)

experiment	check	replicates	observed	links
quality_gates_rss_dsd_heavy	memory_usage	10/10	✅ 126 MiB ≤ 140 MiB	metrics profiles logs
quality_gates_rss_dsd_low	memory_usage	10/10	✅ 40 MiB ≤ 50 MiB	metrics profiles logs
quality_gates_rss_dsd_medium	memory_usage	10/10	✅ 61.9 MiB ≤ 75 MiB	metrics profiles logs
quality_gates_rss_dsd_ultraheavy	memory_usage	10/10	✅ 182 MiB ≤ 200 MiB	metrics profiles logs
quality_gates_rss_idle	memory_usage	10/10	✅ 26.7 MiB ≤ 40 MiB	metrics profiles logs

Explanation

A change is flagged as a regression when |Δ mean %| > 5.00% in the regressing direction for its optimization goal AND SMP marks the experiment as a regression (is_regression: true). Improvements use the matching criteria for the improving direction. Experiments configured erratic: true (tagged (ignored)) are skipped outright; experiments detected as erratic at runtime (tagged (erratic)) still count, since that flag describes sample dispersion rather than directional certainty. The Δ mean % cell is colored accordingly: 🟢 = improvement, 🔴 = regression, ⚪ = neutral. Reduction in CPU or memory is an improvement; reduction in ingress throughput is a regression.