fix: skip prompt cache hints for Bedrock Nova and add per-model override

[MingyuLiNCU]

Summary

Fixes #2332 — Bedrock Nova models fail with extraneous key [cachePoint] is not permitted on every completion call, because the prompt-cache hint that
LiteLLM translates into the Converse cachePoint field was only skipped for
Gemini routes (#2069) and never for Nova.

Root cause

cache_control_injection_points is injected unconditionally for every
non-Gemini route in holmes/core/llm.py:

if not _is_gemini_route(litellm_model_name):
    cache_kwargs["cache_control_injection_points"] = [...]

On Bedrock, LiteLLM sends this hint as the Converse API's cachePoint field,
which the Amazon Nova family rejects. Bedrock Claude (which supports
cachePoint) was never affected, which is why the exception list stayed
Gemini-only.

Changes

Two complementary parts, both in holmes/core/llm.py:

1. A single capability check instead of per-model exceptions —
   new _supports_prompt_caching(litellm_model_name) returns False for the
   Gemini routes (unchanged behavior) and for Bedrock amazon.nova* models.
   All route-specific decisions now live in one place, so the next
   incompatible model is a one-line addition rather than another if at the
   call site.

2. An explicit per-model override — ModelEntry gains a
   cache_control: Optional[bool] field. Users can force prompt caching
   on/off from model_list.yaml, covering models neither the defaults nor
   LiteLLM know about yet (same shape as the temperature issue raised for
   Bedrock Opus 4.7).

Tests

Extended tests/core/test_llm_completion_cache_control.py:

• Bedrock Nova routes (us.amazon.nova-pro/lite/micro-v1:0, plus a regional
  variant) get no cache_control_injection_points kwarg
• Bedrock Claude and all other non-Gemini routes still get it (regression)
• cache_control: true|false override wins in both directions, including
  forcing the hint on for Nova
• cache_control never leaks from ModelEntry/DefaultLLM.args into the
  litellm call

All tests are offline (mocked litellm.completion), following the existing
pattern from the #2069 tests.

Verification

• pytest tests/core/test_llm_completion_cache_control.py and the LLM-related
  regression suite: all pass locally

Summary by CodeRabbit

• New Features

  • Added automatic cache-control handling based on the selected model route.
  • Added per-model settings to force or suppress cache-control hints.
  • Cache-control settings are handled internally and are not passed to the underlying service.

• Bug Fixes

  • Prevented unsupported cache hints from being sent to Gemini and Bedrock Nova routes.
  • Preserved cache hints for compatible routes to maintain expected caching behavior.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: MingyuLiNCU / name: Mingyu Li (802b80b)
• ✅ login: MingyuLiNCU / name: 掌心扑火 (4b6bdbe)

[netlify]

✅ Deploy Preview for holmes-docs ready!

Name	Link
🔨 Latest commit	4fb80f4
🔍 Latest deploy log	https://app.netlify.com/projects/holmes-docs/deploys/6a70b6cef8ece70009725a7a
😎 Deploy Preview	https://deploy-preview-2348--holmes-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 73d205fe-7613-4a6c-acbe-2d6e6d7bf1f7

📥 Commits

Reviewing files that changed from the base of the PR and between 802b80b and 400ab81.

📒 Files selected for processing (1)

• tests/core/test_llm_completion_cache_control.py

🚧 Files skipped from review as they are similar to previous changes (1)

• tests/core/test_llm_completion_cache_control.py

---

Walkthrough

The LLM now detects prompt-cache support by route. Gemini and Bedrock Nova disable cache hints by default. A per-model cache_control setting overrides detection and is removed before LiteLLM calls. Tests cover defaults, overrides, and argument handling.

Changes

Prompt-caching route control

Layer / File(s)	Summary
Route-aware cache-control handling holmes/core/llm.py, tests/core/test_llm_completion_cache_control.py	ModelEntry and DefaultLLM support an optional cache_control override. Route detection disables hints for Gemini and Bedrock Nova. Custom arguments consume the override before LiteLLM receives them. Tests cover default behavior, Bedrock Nova routes, forced hints, suppression, and argument removal.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the Bedrock Nova cache-hint fix and the per-model override added by the pull request.
Linked Issues check	✅ Passed	The changes prevent cache hints for Bedrock Nova models and add per-model cache-control configuration required by issue #2332.
Out of Scope Changes check	✅ Passed	The code and tests remain focused on route-aware prompt caching, Bedrock Nova support, and per-model cache-control overrides.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/core/test_llm_completion_cache_control.py`:
- Around line 123-126: Update the test docstring describing the per-model
cache_control override so it states that cache_control: true forces the cache
hint for models whose automatic defaults suppress it, including Bedrock Nova
models; remove the outdated “not covered by the default yet” wording.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 15740e31-950c-4e22-8cba-79dc13486632

📥 Commits

Reviewing files that changed from the base of the PR and between 8b478c0 and 802b80b.

📒 Files selected for processing (2)

• holmes/core/llm.py
• tests/core/test_llm_completion_cache_control.py

[hsmade]

Lgtm!