Skip to content

feat(cli): instrument fern automations generate telemetry#16011

Open
FedeZara wants to merge 3 commits into
mainfrom
FedeZara/feat/automations-generate-telemetry
Open

feat(cli): instrument fern automations generate telemetry#16011
FedeZara wants to merge 3 commits into
mainfrom
FedeZara/feat/automations-generate-telemetry

Conversation

@FedeZara
Copy link
Copy Markdown
Contributor

@FedeZara FedeZara commented May 20, 2026

Summary

Adds Tier 1 automation telemetry for fern automations generate, including run-level lifecycle events and per-generator success/failure/skip events gated on FERN_AUTOMATION=true.

Closes FER-10684

Changes

  • Add generation_started, generation_completed, generation_failed, generator_completed, generator_failed, and generator_skipped to the automation event registry with attribute helpers
  • Wire telemetry emits through GeneratorRunCollector and run-level events in executeAutomationsGenerate
  • Add skipErrorReporting on Fiddle failAndThrow so container failures are not double-reported to CLI Sentry
  • Classify generator_failed with failure_source: "container" | "cli": suppress CLI Sentry when the generator failed inside the remote container (TaskAbortSignal from failAndThrow); correlate via fern_run_id
  • Add unit tests for registry, attributes, collector, and AutomationTelemetryManager Sentry gating

Test plan

  • Unit tests added/updated
  • pnpm turbo run compile --filter @fern-api/cli
  • Manual smoke in GitHub Actions with FERN_AUTOMATION=true

@FedeZara FedeZara requested a review from amckinney as a code owner May 20, 2026 15:56
@FedeZara FedeZara self-assigned this May 20, 2026
Copy link
Copy Markdown

@claude claude Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

Tip: disable this comment in your organization's Code Review settings.

Emit Tier 1 automation events for fern automations generate with
per-generator outcomes and run-level lifecycle tracking. Dedupe Sentry
for Fiddle failures via skipErrorReporting and failure_source tagging.

Co-authored-by: Cursor <cursoragent@cursor.com>
@FedeZara FedeZara force-pushed the FedeZara/feat/automations-generate-telemetry branch from 9a8a30c to cef4c78 Compare May 20, 2026 15:58
Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 20, 2026

Docs Generation Benchmark Results

Comparing PR branch against median of 5 nightly run(s) on main (latest: 2026-05-22T05:19:54Z).

Fixture main PR Delta
docs 211.4s (n=5) 227.5s (35 versions) +16.1s (+7.6%)

Docs generation runs fern generate --docs --preview end-to-end against the benchmark fixture with 35 API versions (each version: markdown processing + OpenAPI-to-IR + FDR upload).
Delta is computed against the nightly baseline on main.
Baseline from nightly run(s) on main (latest: 2026-05-22T05:19:54Z). Trigger benchmark-baseline to refresh.
Last updated: 2026-05-22 16:54 UTC

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 20, 2026

SDK Generation Benchmark Results

Comparing PR branch against median of 5 nightly run(s) on main (latest: 2026-05-22T05:19:54Z).

Full benchmark table (click to expand)
Generator Spec main (generator) main (E2E) PR (generator) Delta
csharp-sdk square 73s (n=5) 109s (n=5) 65s -8s (-11.0%)
go-sdk square 136s (n=5) 294s (n=5) 125s -11s (-8.1%)
java-sdk square 209s (n=5) 286s (n=5) 199s -10s (-4.8%)
php-sdk square 62s (n=5) 80s (n=5) 53s -9s (-14.5%)
python-sdk square 128s (n=5) 229s (n=5) 130s +2s (+1.6%)
ruby-sdk-v2 square 92s (n=5) 118s (n=5) 81s -11s (-12.0%)
rust-sdk square 170s (n=5) 164s (n=5) 158s -12s (-7.1%)
swift-sdk square 63s (n=5) 746s (n=5) 53s -10s (-15.9%)
ts-sdk square 234s (n=5) 240s (n=5) 220s -14s (-6.0%)

main (generator): generator-only time via --skip-scripts (includes Docker image build, container startup, IR parsing, and code generation — this is the same Docker-based flow customers use via fern generate). main (E2E): full customer-observable time including build/test scripts (nightly baseline, informational). Delta is computed against generator-only baseline.
⚠️ = generation exited with a non-zero exit code (timing may not reflect a successful run).
Baseline from nightly runs on main (latest: 2026-05-22T05:19:54Z). Trigger benchmark-baseline to refresh.
Last updated: 2026-05-22 16:55 UTC

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant