Skip to content

fix(stats): don't feed inferred proxy spans to client-side stats concentrator#3823

Draft
Leiyks wants to merge 1 commit intomasterfrom
leiyks/fix-ci-stats-inferred-span
Draft

fix(stats): don't feed inferred proxy spans to client-side stats concentrator#3823
Leiyks wants to merge 1 commit intomasterfrom
leiyks/fix-ci-stats-inferred-span

Conversation

@Leiyks
Copy link
Copy Markdown
Contributor

@Leiyks Leiyks commented Apr 24, 2026

Problem

The System Tests: [tracer-release] CI job was failing because client-side stats were reporting 10 hits instead of 5 for 5 HTTP requests. This broke the system test assertion ok_hits == ok_top_hits == 5.

Root Cause

The early-exit stats computation block in ext/serializer.c was feeding both the root span AND the inferred proxy span (API Gateway Tracing) to the stats concentrator. The inferred span has no http.method or http.endpoint in its meta, creating a separate aggregation bucket. With 5 requests:

  • Root span bucket: 5 hits (http.method=GET, http.endpoint=/stats-unique)
  • Inferred span bucket: 5 hits (empty http.method, empty http.endpoint)

Total: 10 hits instead of 5.

Fix

Remove the inferred span from the early-exit stats path. Inferred spans are proxy-level spans (API Gateway Tracing) that don't represent real service calls and should not be included in client-side stats. The root span already captures the full request statistics.

Analysis

The if (inferred_span) block was added in #3756 (Implement stats computation) with the comment "Inferred span won't be serialized, so feed it to the concentrator here." The intent was to compensate for the fact that the early-exit path returns NULL before the normal inferred span serialization at line 1829. However, inferred spans should not be in the concentrator at all — they are proxy-level spans without meaningful http.method/http.endpoint tags, and feeding them creates a spurious aggregation bucket.

Testing

The system test Test_Client_Stats::test_client_stats in ~/git/system-tests/tests/stats/test_stats.py should now pass with ok_hits == 5.

…entrator

Inferred spans (API Gateway Tracing) were being fed to the stats
concentrator in the early-exit path, creating a separate aggregation
bucket with empty http.method/http.endpoint. This doubled the hit count
in client-side stats, breaking the system test assertion ok_hits == 5.

Inferred spans are proxy-level spans that don't represent real service
calls and should not be included in client-side stats.
@datadog-datadog-prod-us1
Copy link
Copy Markdown

datadog-datadog-prod-us1 Bot commented Apr 24, 2026

Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 60.68% (+0.03%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 00acf4c | Docs | Datadog PR Page | Give us feedback!

@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented Apr 24, 2026

Benchmarks [ tracer ]

Benchmark execution time: 2026-04-24 12:12:12

Comparing candidate commit 00acf4c in PR branch leiyks/fix-ci-stats-inferred-span with baseline commit b7eb412 in branch master.

Found 0 performance improvements and 4 performance regressions! Performance is the same for 188 metrics, 2 unstable metrics.

scenario:EmptyFileBench/benchEmptyFileBaseline

  • 🟥 execution_time [+94.180µs; +352.360µs] or [+3.073%; +11.498%]

scenario:PDOBench/benchPDOOverhead-opcache

  • 🟥 execution_time [+7.630µs; +10.000µs] or [+3.158%; +4.140%]

scenario:PDOBench/benchPDOOverheadWithDBM-opcache

  • 🟥 execution_time [+8.964µs; +11.136µs] or [+3.721%; +4.622%]

scenario:PHPRedisBench/benchRedisOverhead-opcache

  • 🟥 execution_time [+30.696µs; +39.579µs] or [+3.088%; +3.981%]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant