fix(stats): don't feed inferred proxy spans to client-side stats concentrator#3823
Draft
fix(stats): don't feed inferred proxy spans to client-side stats concentrator#3823
Conversation
…entrator Inferred spans (API Gateway Tracing) were being fed to the stats concentrator in the early-exit path, creating a separate aggregation bucket with empty http.method/http.endpoint. This doubled the hit count in client-side stats, breaking the system test assertion ok_hits == 5. Inferred spans are proxy-level spans that don't represent real service calls and should not be included in client-side stats.
🎉 All green!❄️ No new flaky tests detected 🎯 Code Coverage (details) 🔗 Commit SHA: 00acf4c | Docs | Datadog PR Page | Give us feedback! |
Benchmarks [ tracer ]Benchmark execution time: 2026-04-24 12:12:12 Comparing candidate commit 00acf4c in PR branch Found 0 performance improvements and 4 performance regressions! Performance is the same for 188 metrics, 2 unstable metrics. scenario:EmptyFileBench/benchEmptyFileBaseline
scenario:PDOBench/benchPDOOverhead-opcache
scenario:PDOBench/benchPDOOverheadWithDBM-opcache
scenario:PHPRedisBench/benchRedisOverhead-opcache
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
System Tests: [tracer-release]CI job was failing because client-side stats were reporting 10 hits instead of 5 for 5 HTTP requests. This broke the system test assertionok_hits == ok_top_hits == 5.Root Cause
The early-exit stats computation block in
ext/serializer.cwas feeding both the root span AND the inferred proxy span (API Gateway Tracing) to the stats concentrator. The inferred span has nohttp.methodorhttp.endpointin its meta, creating a separate aggregation bucket. With 5 requests:Total: 10 hits instead of 5.
Fix
Remove the inferred span from the early-exit stats path. Inferred spans are proxy-level spans (API Gateway Tracing) that don't represent real service calls and should not be included in client-side stats. The root span already captures the full request statistics.
Analysis
The
if (inferred_span)block was added in #3756 (Implement stats computation) with the comment "Inferred span won't be serialized, so feed it to the concentrator here." The intent was to compensate for the fact that the early-exit path returnsNULLbefore the normal inferred span serialization at line 1829. However, inferred spans should not be in the concentrator at all — they are proxy-level spans without meaningfulhttp.method/http.endpointtags, and feeding them creates a spurious aggregation bucket.Testing
The system test
Test_Client_Stats::test_client_statsin~/git/system-tests/tests/stats/test_stats.pyshould now pass withok_hits == 5.