fix(python): bypass expensive dealiasing in parse_sse_obj for ~28x SSE speedup#16074
fix(python): bypass expensive dealiasing in parse_sse_obj for ~28x SSE speedup#16074tstanmay13 wants to merge 1 commit into
Conversation
…E speedup parse_sse_obj was calling parse_obj_as which runs convert_and_respect_annotation_metadata on every SSE event. This function recursively introspects all union variants and accounts for ~75% of total parse time on large discriminated unions like V2ChatStreamResponse. SSE data arrives as wire-format JSON and does not need the TypedDict dealiasing pass. This change: - Adds _validate_sse_data() that calls TypeAdapter.validate_python directly - Caches discriminator/variant analysis per type - Replaces dataclasses.asdict() with direct attribute access Benchmark (100k events, Cohere V2ChatStreamResponse): BEFORE: 4059 us/event (405.9s total) AFTER: 147 us/event (14.7s total) Speedup: 27.6x Co-Authored-By: tanmay.singh <tstanmay13@gmail.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.
Tip: disable this comment in your organization's Code Review settings.
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
SDK Generation Benchmark ResultsComparing PR branch against median of 5 nightly run(s) on Full benchmark table (click to expand)
main (generator): generator-only time via --skip-scripts (includes Docker image build, container startup, IR parsing, and code generation — this is the same Docker-based flow customers use via |
1 participant
Description
Fixes the major SSE streaming performance regression reported by Cohere. The previous TypeAdapter caching fix (PR #16013) addressed only ~17% of the cost. This PR eliminates the remaining ~75% bottleneck.
Root Cause
parse_sse_obj→parse_obj_as→convert_and_respect_annotation_metadataruns expensive recursive type introspection on every single SSE event. For a large discriminated union likeV2ChatStreamResponse, this function iterates through all union variants and calls_convert_mappingfor each Pydantic model variant — O(variants × fields) per event.This dealiasing pass is designed for TypedDict aliasing support and is completely unnecessary for SSE data, which arrives as wire-format JSON and maps directly to Pydantic models.
Profiling breakdown (cohere 6.1.0, 10K events)
convert_and_respect_annotation_metadataTypeAdapter()creation (uncached)TypeAdapter.validate_pythonasdict/ discriminator / json.loadsChanges Made
_validate_sse_data()helper that callsTypeAdapter.validate_python()directly, bypassing the unnecessaryconvert_and_respect_annotation_metadatadealiasing step_get_sse_type_info()cache for discriminator/variant analysis per type (avoids repeated type introspection)dataclasses.asdict(sse)with direct attribute access (sse.data,sse.event, etc.)pydantic_utilities.pyvariants:shared/pydantic_utilities.py(pydantic v2 native)shared/with_pydantic_aliases/pydantic_utilities.py(pydantic v2 with aliases)shared/with_pydantic_v1_on_v2/pydantic_utilities.py(pydantic v1 compat)shared/with_pydantic_v1_on_v2/with_aliases/pydantic_utilities.py(v1 compat with aliases)Cohere SDK Benchmark Results
Comprehensive benchmark (100K events,
V2ChatStreamResponse):json.loadsonlyconstruct_type)parse_sse_objparse_sse_objoriginal)The ~20x overhead vs
json.loads-only is the irreducible cost of Pydantic'sTypeAdapter.validate_python()which provides proper typing, discriminated union dispatch, and field validation. This is 3x faster than the old 5.20.4construct_typepath that preceded theparse_sse_objregression.Additional per-version results
cohere 7.0.0 (latest):
cohere 6.1.0:
Testing
construct_type(161 vs 511 µs)test_http_sse.py)parse_obj_astests pass (test_parse_obj_as.py)poetry run pre-commit run -apasses cleanlyLink to Devin session: https://app.devin.ai/sessions/44e42c69b1c240cc8d38f1b9416aaa3a
Requested by: @tstanmay13