feat(fireworks): add prompt caching middleware#38823
feat(fireworks): add prompt caching middleware#38823Mason Daugherty (mdrxy) wants to merge 1 commit into
Conversation
github-actions
Bot
added
dependencies
![open-swe[bot]](https://avatars.githubusercontent.com/in/1395372?s=60&v=4){kind=small}
| if not _supports_fireworks_prompt_cache(fallback_model): | ||
| model_settings, fireworks_cache_changed = _without_fireworks_prompt_cache(model_settings) | ||
| model_settings_changed = model_settings_changed or fireworks_cache_changed |
There was a problem hiding this comment.
🔵 Preserve OpenAI prompt cache keys
prompt_cache_key is not Fireworks-specific: ChatOpenAI also exposes it as a supported per-invocation setting for cache routing. Because every non-Fireworks fallback now passes through this branch, an OpenAI primary/fallback request with model_settings={"prompt_cache_key": ...} silently loses the caller's valid key after the primary fails, reducing cache affinity and changing the fallback request. The sanitizer needs to distinguish affinity injected for Fireworks from a provider setting that the selected fallback supports, rather than removing this shared key from every non-Fireworks model.
(Refers to lines 127-129)
Your feedback helps Open SWE learn. React with 👍 or 👎 to tell us if this review comment was useful.
1 participant
Fireworks agents can now use
FireworksPromptCachingMiddlewareto automatically set prompt-cache session affinity from the active thread ID. Cross-provider model fallbacks remove those Fireworks-specific settings before invoking the fallback model.Fireworks prompt caching is most effective when requests from the same agent thread reach the same model replica. Agents currently need to manage that routing affinity themselves, which makes cache reuse easy to miss.
This adds
FireworksPromptCachingMiddleware, which derives session affinity fromconfig.configurable.thread_idwhile respecting affinity settings supplied by the caller. It also updatesModelFallbackMiddlewareso Fireworks-only cache settings are retained for Fireworks fallbacks and removed before requests are sent to another provider.