Add eval 283: CPU throttling caused by cache-key mismatch bug - #2310
Add eval 283: CPU throttling caused by cache-key mismatch bug#2310aantn wants to merge 2 commits into
Conversation
New ask-holmes eval testing a full metrics -> logs -> source-code root-cause chain: - quote-service (namespace app-283, CPU limit 200m) recomputes an expensive tariff matrix on every request because cache entries are written under the key format ORIGIN->DEST but looked up via _cache_key(), which builds ORIGIN:DEST — the cache never hits, CPU pegs at the CFS quota, and Prometheus fires CPUThrottlingHigh. - Loki (promtail sidecar) carries warnings that name only the slow function (compute_tariff_matrix took NNNNms), not the cause. - A GitLab-mimicking MCP server (FastMCP, streamable-http) exposes list_projects / get_repository_tree / get_file_contents / list_commits, serving the exact source files the pod runs (both mount the same Secret), plus a commit history whose latest entry is the refactor that introduced the bug. The expected root cause (the mismatched key formats in tariff_engine.py) can only be produced by reading the code, ruling out hallucination. Verified end-to-end in the sandbox: setup needles all pass and opus-4.6 (via OpenRouter) finds the exact bug and the offending commit, 1/1. Also document three new Claude Code sandbox limitations discovered while verifying (GitHub release downloads blocked, CFS quota not enforced, pip TLS MITM inside pods) in CLAUDE.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016LXH3dCxNQNVHbHQTqjBgj Signed-off-by: Claude <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
|
✅ Docker images ready for
Use these tags to pull the images for testing. 📋 Copy commandsgcloud auth configure-docker us-central1-docker.pkg.dev
docker pull us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:5b0334e05
docker tag us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes:5b0334e05 me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:5b0334e05
docker push me-west1-docker.pkg.dev/robusta-development/development/holmes-dev:5b0334e05
docker pull us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes-operator:5b0334e05
docker tag us-central1-docker.pkg.dev/robusta-development/temporary-builds/holmes-operator:5b0334e05 me-west1-docker.pkg.dev/robusta-development/development/holmes-operator-dev:5b0334e05
docker push me-west1-docker.pkg.dev/robusta-development/development/holmes-operator-dev:5b0334e05Patch Helm values in one line (choose the chart you use): HolmesGPT chart: helm upgrade --install holmesgpt ./helm/holmes \
--set registry=me-west1-docker.pkg.dev/robusta-development/development \
--set image=holmes-dev:5b0334e05 \
--set operator.registry=me-west1-docker.pkg.dev/robusta-development/development \
--set operator.image=holmes-operator-dev:5b0334e05Robusta wrapper chart: helm upgrade --install robusta robusta/robusta \
--reuse-values \
--set holmes.registry=me-west1-docker.pkg.dev/robusta-development/development \
--set holmes.image=holmes-dev:5b0334e05 \
--set holmes.operator.registry=me-west1-docker.pkg.dev/robusta-development/development \
--set holmes.operator.image=holmes-operator-dev:5b0334e05 |
✅ Deploy Preview for holmes-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
🚧 Files skipped from review as they are similar to previous changes (2)
WalkthroughAdds sandbox limitation workarounds and a Kubernetes eval fixture containing a quote service, tariff cache, request worker, Loki logging, GitLab MCP access, Prometheus checks, and cleanup orchestration. ChangesSandbox guidance
CPU throttling eval
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant RateSyncWorker
participant QuoteService
participant TariffEngine
participant Loki
RateSyncWorker->>QuoteService: Request quote
QuoteService->>TariffEngine: Compute or retrieve matrix
TariffEngine-->>QuoteService: Return matrix
QuoteService->>TariffEngine: Select cheapest quote
TariffEngine-->>QuoteService: Return quote
QuoteService->>Loki: Write JSON timing log
QuoteService-->>RateSyncWorker: Return quote response
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
✅ Results of HolmesGPT evalsAutomatically triggered by commit 205e008 on branch Results of HolmesGPT evals
Skills mechanism stats
Benchmark Comparison DetailsMaster baseline: latest master-* experiment (post-merge regression eval)
Benchmark baseline: latest ci-benchmark experiment on master
No baseline data available for comparison. Comparison indicators:
📖 Legend
🔄 Re-run evals manually
Option 1: Comment on this PR with Or with more options (one per line): Run evals on a different branch (e.g., master) for comparison:
Quick re-run: Use Option 2: Trigger via GitHub Actions UI → "Run workflow" Option 3: Add PR labels to include extra evals (applies to both automatic runs and
Examples: 🏷️ Valid tags
🤖 Valid models
Commands: CLI: |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@CLAUDE.md`:
- Around line 640-643: Update the `/tmp/jq` executable shim described in the
workaround so its `sed` transformation removes the entire `"oomScoreAdj"`
property, including its value and surrounding JSON syntax, regardless of whether
the value is positive, negative, or formatted differently. Preserve the existing
emulation of the runc wrapper’s jq invocation.
In
`@tests/llm/fixtures/test_ask_holmes/283_cpu_throttling_code_bug/deployment.yaml`:
- Around line 149-168: Move the inline rate-sync shell script from the
Deployment container args into a neighboring rate-sync.sh file, and reference
that script through a Secret volume and mount. Update before_test to create the
Secret from rate-sync.sh, following the existing quote-service-src and
gitlab-mcp-code Secret patterns while preserving the worker command and
resource-efficient behavior.
In
`@tests/llm/fixtures/test_ask_holmes/283_cpu_throttling_code_bug/test_case.yaml`:
- Around line 153-156: Update Needle 2’s grep check in the deployment validation
to match the specific un-spaced cache write-key format, such as the
`f"{origin.upper()}->{dest.upper()}"` expression, instead of the broad `->`
pattern. Keep the existing failure handling and success message unchanged so the
check only passes when the buggy write-key line is present.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 0bc5e12f-a583-4a8c-9abf-0759a401a2d3
📒 Files selected for processing (8)
CLAUDE.mdtests/llm/fixtures/test_ask_holmes/283_cpu_throttling_code_bug/app/README.mdtests/llm/fixtures/test_ask_holmes/283_cpu_throttling_code_bug/app/server.pytests/llm/fixtures/test_ask_holmes/283_cpu_throttling_code_bug/app/tariff_engine.pytests/llm/fixtures/test_ask_holmes/283_cpu_throttling_code_bug/deployment.yamltests/llm/fixtures/test_ask_holmes/283_cpu_throttling_code_bug/gitlab_mcp_server.pytests/llm/fixtures/test_ask_holmes/283_cpu_throttling_code_bug/test_case.yamltests/llm/fixtures/test_ask_holmes/283_cpu_throttling_code_bug/toolsets.yaml
- Tighten the needle-2 setup check: grep for the exact buggy write-key
expression ('}->{dest.upper()}') instead of a bare '->', which also
matched ordinary return-type annotations and could never fail.
- Move the rate-sync-worker loop out of inline Deployment args into
rate-sync.sh, mounted from a Secret, matching the repo convention and
the other scripts in this fixture.
- Drop the no-cicd tag: the labeled CI run executed this eval on the
KIND cluster and it passed 1/1 (opus-4.6), proving the prerequisites
(kube-prometheus, real CFS throttling, in-pod pip) all exist in CI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016LXH3dCxNQNVHbHQTqjBgj
Signed-off-by: Claude <noreply@anthropic.com>
2 participants
Summary
Adds a new LLM evaluation test (eval 283) that validates Holmes's ability to diagnose CPU throttling caused by an application-level caching bug. The eval tests the complete investigation chain: metrics → logs → source code analysis via MCP.
Key Changes
deployment.yaml): Sets up a quote-service pod with intentional CPU throttling, a rate-sync worker to generate load, Promtail for log shipping to Loki, and a GitLab-mimicking MCP server that serves the application source codeapp/server.py,app/tariff_engine.py,app/README.md): Implements a shipping quote service with a deliberate cache-key mismatch bug — tariff matrices are stored with key format"ORIGIN->DEST"but looked up via_cache_key()which builds"ORIGIN:DEST", causing cache misses and repeated expensive computationsgitlab_mcp_server.py): Provides read-only repository access tools (list_projects, get_repository_tree, get_file_contents, list_commits) that serve the exact source files the pod executes, enabling Holmes to discover the bug by reading the codetest_case.yaml): Defines the eval scenario, expected outputs, setup/teardown, and validation needles:compute_tariff_matrix took NNNms)->) is present in tariff_engine.py served by MCPcontainer_cpu_cfs_throttled_periods_total)toolsets.yaml): Enables Kubernetes, Loki, Prometheus, and GitLab MCP toolsets with appropriate port-forwards and API endpointsImplementation Details
_cache_key()returns"AMS:JFK"but the code stores under"AMS->JFK", so every request recomputes the expensive tariff matrixhttps://claude.ai/code/session_016LXH3dCxNQNVHbHQTqjBgj
Summary by CodeRabbit
New Features
/healthzand/api/v1/quoteendpoints, structured JSON logging, and tariff caching/cheapest-quote selection.Documentation