From 2579a6245b5929fde7eaff17116c7f23546d75a1 Mon Sep 17 00:00:00 2001 From: Jesse Szwedko Date: Fri, 29 May 2026 14:25:26 -0700 Subject: [PATCH 1/7] docs(dogstatsd): document dogstatsd_mem_based_rate_limiter.* as architectural divergence Expands the single .enabled entry in the Not Planned table to cover all 11 keys under the prefix, and adds an explanation section describing why these Go GC-specific mechanisms have no Rust equivalent in ADP and what ADP uses instead (MemoryBounds, MemoryLimiter, bounded channels). Closes #1351 Co-Authored-By: Claude Sonnet 4.6 --- .../configuration/dogstatsd.md | 27 +++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-) diff --git a/docs/agent-data-plane/configuration/dogstatsd.md b/docs/agent-data-plane/configuration/dogstatsd.md index c6851c7265..fd27e7dbb5 100644 --- a/docs/agent-data-plane/configuration/dogstatsd.md +++ b/docs/agent-data-plane/configuration/dogstatsd.md @@ -1,6 +1,6 @@ # Configuring DogStatsD on Agent Data Plane - + The DogStatsD implementation on ADP has been redesigned in Rust for better resource guarantees and efficiency. Because the architecture is different from the original implementation, certain @@ -45,7 +45,7 @@ architecture is fundamentally different or the feature is platform-specific. | Config Key | Description | Reason | | ---------------------------------------------- | ---------------------------------- | ------------------------------------------------------------ | | `dogstatsd_host_socket_path` | Host UDS socket dir for DSD | Not read by DSD server; admission controller only | -| `dogstatsd_mem_based_rate_limiter.enabled` | Enable memory rate limiter | Go GC specific; use `memory_limit` | +| `dogstatsd_mem_based_rate_limiter.*` | Memory-based rate limiter (11 keys) | Go GC–specific; ADP uses `memory_limit` (see below) | | `dogstatsd_no_aggregation_pipeline_batch_size` | No-aggregation pipeline batch size | Fixed in ADP topology | | `dogstatsd_packet_buffer_flush_timeout` | Packet buffer flush timeout | ADP decodes inline | | `dogstatsd_packet_buffer_size` | Datagrams per packet buffer | ADP decodes inline | @@ -56,6 +56,29 @@ architecture is fundamentally different or the feature is platform-specific. | `dogstatsd_workers_count` | Number of DSD processing workers | ADP uses async tasks | | `use_dogstatsd` | Master DogStatsD enable toggle | Core Agent evaluates and sets `data_plane.dogstatsd.enabled` | +### Memory-based rate limiter (`dogstatsd_mem_based_rate_limiter.*`) + +The core agent exposes 11 keys under this prefix to apply backpressure when the Go process +approaches its memory limit. They work by manipulating Go's garbage collector +(`debug.SetGCPercent`, `debug.FreeOSMemory`), allocating a large heap ballast to adjust GC +heuristics, and blocking goroutines to slow packet ingestion. None of these mechanisms have +an equivalent in Rust, and ADP does not use a Go runtime. + +ADP takes a different approach to the same problem: + +- **Static bounds**: components declare their memory footprint at startup via `MemoryBounds`. + ADP refuses to start if declared bounds exceed the configured `memory_limit`, preventing + over-commitment before any traffic arrives. +- **Dynamic limiting**: a `MemoryLimiter` polls the process RSS every 250 ms. When usage + exceeds 95 % of the effective limit it applies proportional async backpressure (1–25 ms) + to ingestion tasks. +- **Structural backpressure**: bounded channels between components provide back-pressure + independently of memory monitoring. + +To set a process memory limit in ADP, use `memory_limit` (bytes). The `memory_slop_factor` +setting reserves a headroom fraction to account for allocations not tracked by ADP's internal +accounting. All 11 `dogstatsd_mem_based_rate_limiter.*` keys are ignored by ADP. + ## Behavioral Differences From 4b6dcefe13e790bc6d08947d68277fed24c0feb3 Mon Sep 17 00:00:00 2001 From: Jesse Szwedko Date: Fri, 29 May 2026 14:27:38 -0700 Subject: [PATCH 2/7] docs(dogstatsd): fix table column alignment in mem_based_rate_limiter row Co-Authored-By: Claude Sonnet 4.6 --- docs/agent-data-plane/configuration/dogstatsd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/agent-data-plane/configuration/dogstatsd.md b/docs/agent-data-plane/configuration/dogstatsd.md index fd27e7dbb5..62b1782a0d 100644 --- a/docs/agent-data-plane/configuration/dogstatsd.md +++ b/docs/agent-data-plane/configuration/dogstatsd.md @@ -45,7 +45,7 @@ architecture is fundamentally different or the feature is platform-specific. | Config Key | Description | Reason | | ---------------------------------------------- | ---------------------------------- | ------------------------------------------------------------ | | `dogstatsd_host_socket_path` | Host UDS socket dir for DSD | Not read by DSD server; admission controller only | -| `dogstatsd_mem_based_rate_limiter.*` | Memory-based rate limiter (11 keys) | Go GC–specific; ADP uses `memory_limit` (see below) | +| `dogstatsd_mem_based_rate_limiter.*` | Memory-based rate limiter | Go GC–specific; ADP uses `memory_limit` (see below) | | `dogstatsd_no_aggregation_pipeline_batch_size` | No-aggregation pipeline batch size | Fixed in ADP topology | | `dogstatsd_packet_buffer_flush_timeout` | Packet buffer flush timeout | ADP decodes inline | | `dogstatsd_packet_buffer_size` | Datagrams per packet buffer | ADP decodes inline | From 4463fa8ed8ff0e908ee6e5cf41c3e9ad3ec53470 Mon Sep 17 00:00:00 2001 From: Jesse Szwedko Date: Fri, 29 May 2026 14:29:46 -0700 Subject: [PATCH 3/7] docs(dogstatsd): fix trailing whitespace in mem_based_rate_limiter table row Co-Authored-By: Claude Sonnet 4.6 --- docs/agent-data-plane/configuration/dogstatsd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/agent-data-plane/configuration/dogstatsd.md b/docs/agent-data-plane/configuration/dogstatsd.md index 62b1782a0d..cb9aa2f0fb 100644 --- a/docs/agent-data-plane/configuration/dogstatsd.md +++ b/docs/agent-data-plane/configuration/dogstatsd.md @@ -45,7 +45,7 @@ architecture is fundamentally different or the feature is platform-specific. | Config Key | Description | Reason | | ---------------------------------------------- | ---------------------------------- | ------------------------------------------------------------ | | `dogstatsd_host_socket_path` | Host UDS socket dir for DSD | Not read by DSD server; admission controller only | -| `dogstatsd_mem_based_rate_limiter.*` | Memory-based rate limiter | Go GC–specific; ADP uses `memory_limit` (see below) | +| `dogstatsd_mem_based_rate_limiter.*` | Memory-based rate limiter | Go GC–specific; ADP uses `memory_limit` (see below) | | `dogstatsd_no_aggregation_pipeline_batch_size` | No-aggregation pipeline batch size | Fixed in ADP topology | | `dogstatsd_packet_buffer_flush_timeout` | Packet buffer flush timeout | ADP decodes inline | | `dogstatsd_packet_buffer_size` | Datagrams per packet buffer | ADP decodes inline | From e608cac6f76da3aed67d54bc8886873c2039db2b Mon Sep 17 00:00:00 2001 From: Jesse Szwedko Date: Fri, 29 May 2026 14:32:29 -0700 Subject: [PATCH 4/7] docs(dogstatsd): remove hardcoded key count from rate limiter explanation Co-Authored-By: Claude Sonnet 4.6 --- docs/agent-data-plane/configuration/dogstatsd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/agent-data-plane/configuration/dogstatsd.md b/docs/agent-data-plane/configuration/dogstatsd.md index cb9aa2f0fb..89545ea977 100644 --- a/docs/agent-data-plane/configuration/dogstatsd.md +++ b/docs/agent-data-plane/configuration/dogstatsd.md @@ -58,7 +58,7 @@ architecture is fundamentally different or the feature is platform-specific. ### Memory-based rate limiter (`dogstatsd_mem_based_rate_limiter.*`) -The core agent exposes 11 keys under this prefix to apply backpressure when the Go process +The core agent exposes configuration under this prefix to apply backpressure when the Go process approaches its memory limit. They work by manipulating Go's garbage collector (`debug.SetGCPercent`, `debug.FreeOSMemory`), allocating a large heap ballast to adjust GC heuristics, and blocking goroutines to slow packet ingestion. None of these mechanisms have From 1c98dfa2bdc6cc5c340b46f2068c8d6344f5383f Mon Sep 17 00:00:00 2001 From: Jesse Szwedko Date: Mon, 1 Jun 2026 10:24:03 -0700 Subject: [PATCH 5/7] docs(dogstatsd): move memory architecture to dedicated page Extract the ADP memory management explanation from dogstatsd.md into a new docs/agent-data-plane/memory.md page, covering configuration, memory_mode values, how static/dynamic/structural backpressure work, and comparison with the core Agent's GC-based approach. Leave a brief summary and link in dogstatsd.md. Co-Authored-By: Claude Sonnet 4.6 --- .../configuration/dogstatsd.md | 17 ++---- docs/agent-data-plane/memory.md | 53 +++++++++++++++++++ 2 files changed, 56 insertions(+), 14 deletions(-) create mode 100644 docs/agent-data-plane/memory.md diff --git a/docs/agent-data-plane/configuration/dogstatsd.md b/docs/agent-data-plane/configuration/dogstatsd.md index 89545ea977..157f175f3c 100644 --- a/docs/agent-data-plane/configuration/dogstatsd.md +++ b/docs/agent-data-plane/configuration/dogstatsd.md @@ -64,20 +64,9 @@ approaches its memory limit. They work by manipulating Go's garbage collector heuristics, and blocking goroutines to slow packet ingestion. None of these mechanisms have an equivalent in Rust, and ADP does not use a Go runtime. -ADP takes a different approach to the same problem: - -- **Static bounds**: components declare their memory footprint at startup via `MemoryBounds`. - ADP refuses to start if declared bounds exceed the configured `memory_limit`, preventing - over-commitment before any traffic arrives. -- **Dynamic limiting**: a `MemoryLimiter` polls the process RSS every 250 ms. When usage - exceeds 95 % of the effective limit it applies proportional async backpressure (1–25 ms) - to ingestion tasks. -- **Structural backpressure**: bounded channels between components provide back-pressure - independently of memory monitoring. - -To set a process memory limit in ADP, use `memory_limit` (bytes). The `memory_slop_factor` -setting reserves a headroom fraction to account for allocations not tracked by ADP's internal -accounting. All 11 `dogstatsd_mem_based_rate_limiter.*` keys are ignored by ADP. +ADP takes a different approach to the same problem using explicit static memory accounting and a +process-level RSS limit. All 11 `dogstatsd_mem_based_rate_limiter.*` keys are ignored. See +[Memory Management](../memory.md) for details. ## Behavioral Differences diff --git a/docs/agent-data-plane/memory.md b/docs/agent-data-plane/memory.md new file mode 100644 index 0000000000..ad5760b447 --- /dev/null +++ b/docs/agent-data-plane/memory.md @@ -0,0 +1,53 @@ +# Memory Management + +ADP takes a different approach to memory management than the core Agent, which relies on Go's +garbage collector to reclaim memory and uses runtime GC tuning as backpressure. ADP instead uses +explicit, static memory accounting and a process-level limit. + +## Configuration + +| Config Key | Description | +| ----------------------- | -------------------------------------------------------- | +| `memory_limit` | Process memory limit (bytes, or SI string e.g. `512MB`) | +| `memory_slop_factor` | Fraction of `memory_limit` held back as headroom | +| `memory_mode` | Bounds validation and global limiter behavior | +| `enable_global_limiter` | Toggle dynamic RSS-based backpressure (default: `true`) | + +### `memory_mode` + +| Value | Bounds validation failure | Global limiter active? | +| ------------ | ------------------------- | ------------------------------------- | +| `disabled` | Skipped (default) | No | +| `permissive` | Non-fatal (logged) | Yes, if `memory_limit` is set | +| `strict` | Fatal (ADP won't start) | Yes, if `memory_limit` is set | + +### `memory_slop_factor` + +`memory_slop_factor` is applied as a reduction to `memory_limit` before bounds validation. +A factor of `0.25` withholds 25 % of the limit — for example, a 100 MB limit becomes an +effective 75 MB budget. This accounts for allocations that ADP's static accounting doesn't +track. + +## How it works + +ADP enforces memory bounds through three complementary mechanisms: + +- **Static bounds**: components declare their memory footprint at startup via `MemoryBounds`. + ADP validates that declared bounds fit within the effective limit (`memory_limit` × + (1 − `memory_slop_factor`)). Under `strict` mode, ADP refuses to start if bounds are + exceeded, preventing over-commitment before any traffic arrives. This covers interners + (context strings, tag strings, OTLP strings), aggregation state, and other bounded caches. +- **Dynamic limiting**: a `MemoryLimiter` polls the process RSS every 250 ms. When usage + exceeds 95 % of the effective limit it applies proportional async backpressure (1–25 ms) + to ingestion tasks. Disable with `enable_global_limiter: false`. +- **Structural backpressure**: bounded channels between components provide back-pressure + independently of memory monitoring. + +## Comparison with core Agent + +The core Agent exposes `dogstatsd_mem_based_rate_limiter.*` settings to apply backpressure when +the Go process approaches its memory limit. Those settings work by manipulating Go's garbage +collector (`debug.SetGCPercent`, `debug.FreeOSMemory`), allocating a large heap ballast to +adjust GC heuristics, and blocking goroutines to slow packet ingestion. None of these mechanisms +have an equivalent in Rust, and ADP does not use a Go runtime. All 11 +`dogstatsd_mem_based_rate_limiter.*` keys are ignored by ADP. From 2ee0edccffae496ac635cda947e211846489022c Mon Sep 17 00:00:00 2001 From: Jesse Szwedko Date: Mon, 1 Jun 2026 11:02:45 -0700 Subject: [PATCH 6/7] docs(dogstatsd): fix Vale linting errors in memory.md Co-Authored-By: Claude Sonnet 4.6 --- docs/agent-data-plane/memory.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/agent-data-plane/memory.md b/docs/agent-data-plane/memory.md index ad5760b447..8971a20b4d 100644 --- a/docs/agent-data-plane/memory.md +++ b/docs/agent-data-plane/memory.md @@ -8,7 +8,7 @@ explicit, static memory accounting and a process-level limit. | Config Key | Description | | ----------------------- | -------------------------------------------------------- | -| `memory_limit` | Process memory limit (bytes, or SI string e.g. `512MB`) | +| `memory_limit` | Process memory limit (bytes, or SI string, for example `512MB`) | | `memory_slop_factor` | Fraction of `memory_limit` held back as headroom | | `memory_mode` | Bounds validation and global limiter behavior | | `enable_global_limiter` | Toggle dynamic RSS-based backpressure (default: `true`) | @@ -24,7 +24,7 @@ explicit, static memory accounting and a process-level limit. ### `memory_slop_factor` `memory_slop_factor` is applied as a reduction to `memory_limit` before bounds validation. -A factor of `0.25` withholds 25 % of the limit — for example, a 100 MB limit becomes an +A factor of `0.25` withholds 25 % of the limit. For example, a 100 MB limit becomes an effective 75 MB budget. This accounts for allocations that ADP's static accounting doesn't track. @@ -35,8 +35,9 @@ ADP enforces memory bounds through three complementary mechanisms: - **Static bounds**: components declare their memory footprint at startup via `MemoryBounds`. ADP validates that declared bounds fit within the effective limit (`memory_limit` × (1 − `memory_slop_factor`)). Under `strict` mode, ADP refuses to start if bounds are - exceeded, preventing over-commitment before any traffic arrives. This covers interners - (context strings, tag strings, OTLP strings), aggregation state, and other bounded caches. + exceeded, preventing over-commitment before any traffic arrives. This covers string + interning caches (context strings, tag strings, OTLP strings), aggregation state, and + other bounded allocations. - **Dynamic limiting**: a `MemoryLimiter` polls the process RSS every 250 ms. When usage exceeds 95 % of the effective limit it applies proportional async backpressure (1–25 ms) to ingestion tasks. Disable with `enable_global_limiter: false`. From aa2da3ab7753cb8b3a74fd5204cfd11eaff8361b Mon Sep 17 00:00:00 2001 From: Jesse Szwedko Date: Mon, 1 Jun 2026 13:05:38 -0700 Subject: [PATCH 7/7] docs(dogstatsd): capitalize Core Agent consistently Co-Authored-By: Claude Sonnet 4.6 --- docs/agent-data-plane/configuration/dogstatsd.md | 2 +- docs/agent-data-plane/memory.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/agent-data-plane/configuration/dogstatsd.md b/docs/agent-data-plane/configuration/dogstatsd.md index 157f175f3c..79b0383b3a 100644 --- a/docs/agent-data-plane/configuration/dogstatsd.md +++ b/docs/agent-data-plane/configuration/dogstatsd.md @@ -58,7 +58,7 @@ architecture is fundamentally different or the feature is platform-specific. ### Memory-based rate limiter (`dogstatsd_mem_based_rate_limiter.*`) -The core agent exposes configuration under this prefix to apply backpressure when the Go process +The Core Agent exposes configuration under this prefix to apply backpressure when the Go process approaches its memory limit. They work by manipulating Go's garbage collector (`debug.SetGCPercent`, `debug.FreeOSMemory`), allocating a large heap ballast to adjust GC heuristics, and blocking goroutines to slow packet ingestion. None of these mechanisms have diff --git a/docs/agent-data-plane/memory.md b/docs/agent-data-plane/memory.md index 8971a20b4d..d6c8c697c1 100644 --- a/docs/agent-data-plane/memory.md +++ b/docs/agent-data-plane/memory.md @@ -1,6 +1,6 @@ # Memory Management -ADP takes a different approach to memory management than the core Agent, which relies on Go's +ADP takes a different approach to memory management than the Core Agent, which relies on Go's garbage collector to reclaim memory and uses runtime GC tuning as backpressure. ADP instead uses explicit, static memory accounting and a process-level limit. @@ -44,9 +44,9 @@ ADP enforces memory bounds through three complementary mechanisms: - **Structural backpressure**: bounded channels between components provide back-pressure independently of memory monitoring. -## Comparison with core Agent +## Comparison with Core Agent -The core Agent exposes `dogstatsd_mem_based_rate_limiter.*` settings to apply backpressure when +The Core Agent exposes `dogstatsd_mem_based_rate_limiter.*` settings to apply backpressure when the Go process approaches its memory limit. Those settings work by manipulating Go's garbage collector (`debug.SetGCPercent`, `debug.FreeOSMemory`), allocating a large heap ballast to adjust GC heuristics, and blocking goroutines to slow packet ingestion. None of these mechanisms