Skip to content

CLAUDE.md

Latest commit

 

History

History
892 lines (724 loc) · 45.1 KB

CLAUDE.md

File metadata and controls

892 lines (724 loc) · 45.1 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Repository Overview

HolmesGPT is an AI-powered troubleshooting agent that connects to observability platforms (Kubernetes, Prometheus, Grafana, etc.) to automatically diagnose and analyze infrastructure and application issues. It uses an agentic loop to investigate problems by calling tools to gather data from multiple sources.

Development Commands

Environment Setup

# Install dependencies with Poetry
poetry install

Testing

# Install test dependencies with Poetry
poetry install --with dev
# Run all non-LLM tests (unit and integration tests)
make test-without-llm
poetry run pytest tests -m "not llm"

# Run LLM evaluation tests (requires API keys)
make test-llm-ask-holmes          # Test single-question interactions
make test-llm-investigate         # Test AlertManager investigations
poetry run pytest tests/llm/ -n 6 -vv  # Run all LLM tests in parallel

# Run pre-commit checks (includes ruff, mypy, poetry validation)
# NOTE: Only run these when the user explicitly asks. They run in CI automatically.
make check
poetry run pre-commit run -a

Code Quality (only run when explicitly asked)

# Format code with ruff
poetry run ruff format

# Check code with ruff (auto-fix issues)
poetry run ruff check --fix

# Type checking with mypy
poetry run mypy

Architecture Overview

Core Components

CLI Entry Point (holmes/main.py):

  • Typer-based CLI with subcommands for ask, investigate, toolset
  • Handles configuration loading, logging setup, and command routing

** Interactive mode for CLI** (holmes/interactive.py):

  • Handles interactive mode for ask subcommand
  • Implements slash commands

Configuration System (holmes/config.py):

  • Loads settings from ~/.holmes/config.yaml or via CLI options
  • Manages API keys, model selection, and toolset configurations
  • Factory methods for creating sources (AlertManager, Jira, PagerDuty, etc.)

Core Investigation Engine (holmes/core/):

  • tool_calling_llm.py: Main LLM interaction with tool calling capabilities
  • investigation.py: Orchestrates multi-step investigations with runbooks
  • toolset_manager.py: Manages available tools and their configurations
  • tools.py: Tool definitions and execution logic

Plugin System (holmes/plugins/):

  • Sources: AlertManager, Jira, PagerDuty, OpsGenie integrations
  • Toolsets: Kubernetes, Prometheus, Grafana, AWS, Docker, etc.
  • Prompts: Jinja2 templates for different investigation scenarios
  • Destinations: Slack integration for sending results

Key Patterns

Toolset Architecture:

  • Each toolset is a YAML file defining available tools and their parameters
  • Tools can be Python functions or bash commands with safety validation
  • Toolsets are loaded dynamically and can be customized via config files
  • Important: All toolsets MUST return detailed error messages from underlying APIs to enable LLM self-correction
    • Include the exact query/command that was executed
    • Include time ranges, parameters, and filters used
    • Include the full API error response (status code and message)
    • For "no data" responses, specify what was searched and where

Thin API Wrapper Pattern for Python Toolsets:

  • Reference implementation: servicenow_tables/servicenow_tables.py
  • Use requests library for HTTP calls (not specialized client libraries like opensearchpy)
  • Simple config class with Pydantic validation
  • Health check in prerequisites_callable() method
  • Each tool is a thin wrapper around a single API endpoint

Server-Side Filtering is Critical:

  • Never return unbounded data from APIs - this causes token overflow
  • Always include filter parameters on tools that query collections (e.g., index parameter for Elasticsearch _cat APIs)
  • Example problem: opensearch_list_shards returned ALL shards → 25K+ tokens on large clusters
  • Example fix: elasticsearch_cat tool requires index parameter for shards/segments endpoints
  • When server-side filtering is not possible, use JsonFilterMixin (see json_filter_mixin.py) to add max_depth and jq parameters for client-side filtering

Toolset Config Backwards Compatibility: When renaming config fields in a toolset, maintain backwards compatibility using Pydantic's extra="allow":

# ✅ DO: Use extra="allow" to accept deprecated fields without polluting schema
class MyToolsetConfig(BaseModel):
    model_config = ConfigDict(extra="allow")

    # Only define current field names in schema
    new_field_name: int = 10

    @model_validator(mode="after")
    def handle_deprecated_fields(self):
        extra = self.model_extra or {}
        deprecated = []

        # Map old names to new names
        if "old_field_name" in extra:
            self.new_field_name = extra["old_field_name"]
            deprecated.append("old_field_name -> new_field_name")

        if deprecated:
            logging.warning(f"Deprecated config names: {', '.join(deprecated)}")
        return self

# ❌ DON'T: Define deprecated fields in schema with Optional[None]
class BadConfig(BaseModel):
    new_field_name: int = 10
    old_field_name: Optional[int] = None  # Pollutes schema, shows in model_dump()

Benefits of extra="allow" approach:

  • Schema only shows current field names
  • model_dump() returns clean output without deprecated fields
  • Old configs still work (backwards compatible)
  • Deprecation warnings guide users to update

See prometheus/prometheus.py PrometheusConfig for a complete example.

Class Hierarchy Placement:

  • When adding new config fields, methods, or behavior, always check the class hierarchy and place the change at the most general level that applies
  • Don't scope a fix to a specific subclass just because the issue/request mentions it by name — check if sibling classes share the same need
  • Example: timeout_seconds and max_retries belong on GrafanaConfig, not GrafanaTempoConfig, because all Grafana toolsets (Tempo, Loki, Dashboards) make HTTP requests

LLM Integration:

  • Uses LiteLLM for multi-provider support (OpenAI, Anthropic, Azure, etc.)
  • Structured tool calling with automatic retry and error handling
  • Context-aware prompting with system instructions and examples

Investigation Flow:

  1. Load user question/alert
  2. Select relevant toolsets based on context
  3. Execute LLM with available tools
  4. LLM calls tools to gather data
  5. LLM analyzes results and provides conclusions
  6. Optionally write results back to source system

Testing Framework

Three-tier testing approach:

  1. Unit Tests (tests/): Standard pytest tests for individual components
  2. Integration Tests: Test toolset integrations
  3. LLM Evaluation Tests (tests/llm/): End-to-end tests using fixtures

Running regular (non-LLM) tests:

poetry run pytest tests -m "not llm"
make test-without-llm

Running LLM eval tests:

# Run specific eval - IMPORTANT: Use -k flag, NOT full test path with brackets
poetry run pytest -k "09_crashpod" --no-cov

# Run all evals in parallel
poetry run pytest tests/llm/ -n 6 --no-cov

# Regression evals
poetry run pytest -m 'llm and easy' --no-cov

For the complete eval CLI reference (flags, env vars, model comparison, debugging), see the /create-eval skill which contains full documentation in its reference files.

Configuration

Config File Location: ~/.holmes/config.yaml

Key Configuration Sections:

  • model: LLM model to use (default: gpt-5.4)
  • api_key: LLM API key (or use environment variables)
  • custom_toolsets: Override or add toolsets
  • custom_runbooks: Add investigation runbooks
  • Platform-specific settings (alertmanager_url, jira_url, etc.)

Environment Variables:

  • OPENAI_API_KEY, ANTHROPIC_API_KEY: LLM API keys
  • OPENROUTER_API_KEY: Alternative LLM provider via OpenRouter (domain: api.openrouter.ai). When using OpenRouter, you must also set CLASSIFIER_MODEL to an OpenRouter model (e.g., CLASSIFIER_MODEL="openrouter/openai/gpt-4.1") because the default classifier model is not available via OpenRouter.
  • MODEL: Override default model(s) - supports comma-separated list
  • CLASSIFIER_MODEL: Override the classifier model used internally. Required when using OpenRouter (e.g., openrouter/openai/gpt-4.1)
  • RUN_LIVE: Enable live execution of tools in tests (default: true)
  • BRAINTRUST_API_KEY: For test result tracking and CI/CD report generation
  • BRAINTRUST_ORG: Braintrust organization name (default: "robustadev")
  • ELASTICSEARCH_URL, ELASTICSEARCH_API_KEY: For Elasticsearch/OpenSearch cloud testing

Development Guidelines

Code Quality:

  • Use Ruff for formatting and linting (configured in pyproject.toml)
  • Type hints required (mypy configuration in pyproject.toml)
  • Pre-commit hooks enforce quality checks in CI
  • ALWAYS place Python imports at the top of the file, not inside functions or methods
  • Use tenacity for retries, not hand-rolled retry loops — the @retry(...) decorator normally, or the Retrying(...) iterator form when the attempt budget/params must come from per-instance or dynamic values. Only hand-roll a retry when there's a real reason to, and document why.
  • NEVER run pre-commit, ruff, or mypy unless the user explicitly asks you to. These tools are triggered by commit hooks which are not installed on all machines, and running them causes widespread formatting/type changes to files unrelated to your task. Only lint/format files you are actively editing, and only if asked.

Documentation Examples:

  • Primary examples should use the latest Anthropic Claude models:
    • Recommended: anthropic/claude-sonnet-4-5-20250929 or anthropic/claude-opus-4-5-20251101
    • Use the latest Claude 4.5 family models (Sonnet or Opus) as the default/primary examples
  • You may include other providers (OpenAI, Gemini, etc.) where it would be useful for users, such as in model listing sections or provider-specific documentation
  • Avoid using deprecated or older model versions like claude-3.5-sonnet, gpt-4-vision-preview

Testing Requirements:

  • All new features require unit tests
  • New toolsets require integration tests
  • Complex investigations should have LLM evaluation tests
  • Maintain 40% minimum test coverage
  • Live execution is now enabled by default to ensure tests match real-world behavior
  • Use responses library for HTTP mocking, not @patch("requests.get"). The responses library intercepts at the transport/adapter level, giving more realistic test behavior. Use responses.RequestsMock() with rsps.add() for mock responses.

Pull Request Process:

  • PRs require maintainer approval
  • Pre-commit hooks are checked in CI (do NOT run them locally unless asked)
  • LLM evaluation tests run automatically in CI
  • Keep PRs focused and include tests
  • ALWAYS use git commit -s to sign off commits (required for DCO)
  • When committing, use git commit -s --no-verify to skip local pre-commit hooks (they are not installed consistently and will cause unrelated changes)

Git Workflow Guidelines:

  • ALWAYS create commits, NEVER amend
  • ALWAYS merge, NEVER rebase
  • ALWAYS push, NEVER force push
  • Maintain a history of your work to allow the user to revert back to a previous iteration

File Structure Conventions:

  • Toolsets: holmes/plugins/toolsets/{name}.yaml or {name}/
  • Prompts: holmes/plugins/prompts/{name}.jinja2
  • Tests: Match source structure under tests/

Adding a New Integration (Toolset): When adding a new toolset or integration, update all of the following pages to keep them in sync:

  1. README.md — Data Sources table (add a row with logo, link, status, and description)
  2. docs/walkthrough/why-holmesgpt.md — Categorized integration list under "Every Major Observability Platform"
  3. docs/data-sources/builtin-toolsets/index.md — Grid cards listing on the toolsets index page
  4. docs/data-sources/builtin-toolsets/{name}.md — Dedicated documentation page for the new toolset
  5. Add a logo image to images/integration_logos/ if one is available

Debugging CLI / Rich Live Display Issues

When troubleshooting terminal rendering bugs (ghost frames, flickering, misaligned output):

Capturing terminal output through a PTY:

# Use `script` to force a PTY and capture raw ANSI escape sequences
script -qec "poetry run python your_script.py" /dev/null > /tmp/raw_output.txt 2>&1

Without a PTY, Rich detects non-interactive mode and skips Live rendering entirely.

Analyzing ANSI escape sequences:

# Key escape codes for Rich Live:
# \x1b[1A  = cursor up 1 line
# \x1b[2K  = erase entire line
# Rich erases previous frame with: (erase + cursor-up) × height, then prints new frame

# Count cursor-ups per frame transition to detect drift:
import re
erase_pattern = r"\x1b\[2K(?:\x1b\[1A\x1b\[2K)*"
for match in re.finditer(erase_pattern, raw_output):
    ups = match.group(0).count("\x1b[1A")

Writing unit tests for Live display (no LLM required):

# Render to StringIO with force_terminal=True to get ANSI sequences
from io import StringIO
buf = StringIO()
console = Console(file=buf, force_terminal=True, width=120)
# ... render frames ...
raw = buf.getvalue()  # Contains full ANSI escape sequences
# Parse cursor-up counts vs frame heights to detect ghost frames

Key patterns:

  • Ghost frames = cumulative drift where each frame leaves 1+ orphaned lines
  • Verify by counting: cursor-ups per transition should equal rendered lines per frame
  • Known Rich 13.9.4 bug: Live.refresh() calls console.print(Control()) with default end="\\n", adding a trailing newline not counted in LiveRender._shape. When the terminal has room below the display, each frame leaks 1 ghost line. When the display is at the bottom (common case), the \\n causes scrolling and height-1 cursor-ups is correct.
  • Workaround: subclass Live and override refresh() to pass end="". Do NOT patch position_cursor — that over-erases when the display is at the terminal bottom (the common case).

Investigating Eval Regressions / Holmes Behavior Changes

Understand the behavior from trace data before designing a fix. Braintrust (or the local evals_report.md) holds the rendered prompts, per-LLM-call metrics, and tool-call sequences for each iter. Pull traces for one baseline run and one current run for the same (test, model) before reading any source diffs — file-level diffs (prompts, code, config) routinely mislead about what actually changed at runtime (e.g. a jinja2 template can grow in source lines and shrink in rendered output). Look at individual runs first; aggregate statistics over n iters can hide deterministic per-iter differences under variance.

When reverting or fixing a suspect PR, read its full diff (git show --stat <commit>) before deciding what to change — a PR often has multiple effects, and reverting only the file you noticed leaves the others still acting on the model.

Security Notes

  • All tools have read-only access by design
  • Bash toolset validates commands for safety
  • No secrets should be committed to repository
  • Use environment variables or config files for API keys
  • RBAC permissions are respected for Kubernetes access

Eval Tests (LLM Evaluations)

For creating, running, and debugging LLM eval tests, use the /create-eval skill. It contains the complete workflow, test_case.yaml field reference, anti-hallucination patterns, infrastructure setup guides, and CLI reference.

Test Structure:

  • Use sequential test numbers: check existing tests for next available number
  • Required files: test_case.yaml, infrastructure manifests, toolsets.yaml (if needed)
  • Use dedicated namespace per test: app-<testid> (e.g., app-177)
  • All resource names must be unique across tests to prevent conflicts

Tags:

  • CRITICAL: Only use valid tags from pyproject.toml - invalid tags cause test collection failures
  • Check existing tags before adding new ones, ask user permission for new tags

Cloud Service Evals (No Kubernetes Required):

  • Evals can test against cloud services (Elasticsearch, external APIs) directly via environment variables
  • Faster setup (<30 seconds vs minutes for K8s infrastructure)
  • before_test creates test data in the cloud service; after_test cleans up only if safe (see reentrancy below)
  • Use toolsets.yaml to configure the toolset with env var references: api_url: "{{ env.ELASTICSEARCH_URL }}"
  • CI/CD secrets: When adding evals for a new integration, you must add the required environment variables to .github/workflows/eval-regression.yaml in the "Run tests" step. Tell the user which secrets they need to add to their GitHub repository settings (e.g., ELASTICSEARCH_URL, ELASTICSEARCH_API_KEY).
  • HTTP request passthrough: The root conftest.py has a responses fixture with autouse=True that mocks ALL HTTP requests by default. When adding a new cloud integration, you MUST add the service's URL pattern to the passthrough list in conftest.py (search for rsps.add_passthru). Use re.compile() for pattern matching (e.g., rsps.add_passthru(re.compile(r"https://.*\.cloud\.es\.io"))).
  • Cloud Service Eval Reentrancy: The same eval can run on multiple PRs in parallel in CI. Cloud service evals that create resources with static names (e.g., Confluence spaces, Elasticsearch indices) must be reentrant:
    • before_test must be idempotent: create-or-reuse resources, never fail if they already exist
    • after_test must NOT delete shared resources that another parallel run may be using. Either omit after_test entirely, or limit cleanup to resources with a unique run-scoped identifier
    • Use test-ID-based resource names (e.g., HLMS233 for space keys) to avoid collisions with other evals, but accept that the same eval may overlap with itself across parallel PR runs
    • Kubernetes evals don't have this problem because each PR gets its own KIND cluster, so namespaces are already isolated. Cloud service evals share a single account/instance across all PR runs

User Prompts & Expected Outputs:

  • Be specific: Test exact values like "The dashboard title is 'Home'" not generic "Holmes retrieves dashboard"
  • Match prompt to test: User prompt must explicitly request what you're testing
    • BAD: "Get the dashboard"
    • GOOD: "Get the dashboard and tell me the title, panels, and time range"
  • Anti-cheat prompts: Don't use technical terms that give away solutions
    • BAD: "Find node_exporter metrics"
    • GOOD: "Find CPU pressure monitoring queries"
  • Test discovery, not recognition: Holmes should search/analyze, not guess from context
  • Ruling out hallucinations is paramount: When choosing between test approaches, prefer the one that rules out hallucinations:
    • Best: Check specific values that can only be discovered by querying (e.g., unique IDs, injected error codes, exact counts)
    • Acceptable: Use include_tool_calls: true to verify the tool was called when output values are too generic to rule out hallucinations
    • Bad: Check generic output patterns that an LLM could plausibly guess (e.g., "cluster status is green/yellow/red", "has N nodes")
  • expected_output is invisible to LLM: The expected_output field is only used by the evaluator - the LLM never sees it. This means:
    • You can safely put secrets/verification codes in expected_output that the LLM must discover
    • before_test can inject a unique verification code into test data, and expected_output can check for it
    • This is a powerful pattern for cloud service tests: create data with a unique code in before_test, ask LLM to find it, verify with expected_output
    # Example: before_test creates a page with verification code "HOLMES-EVAL-7x9k2m4p"
# The LLM must discover this code by querying the service
expected_output:
  - "Must report the verification code: HOLMES-EVAL-7x9k2m4p"
  • include_tool_calls: true: Use when expected output is too generic to be hallucination-proof. Prefer specific answer checking when possible, but verifying tool calls is better than a test that can't rule out hallucinations. 
    # Use when values are generic (cluster health could be guessed)
include_tool_calls: true
expected_output:
  - "Must call elasticsearch_cluster_health tool"
  - "Must report cluster status"

Infrastructure Setup:

  • Don't just test pod readiness - verify actual service functionality
  • Poll real API endpoints and check for expected content (e.g., "title":"Home", "type":"welcome")
  • CRITICAL: Use exit 1 when setup verification fails to fail the test early
  • Never use :latest container tags - use specific versions like grafana/grafana:12.3.1

Running and Testing Evals

🚨 CRITICAL: Always Test Your Changes

NEVER submit test changes without verification:

Required Testing Workflow:

  1. Setup Phase: poetry run pytest -k "test_name" --only-setup --no-cov
  2. Full Test: poetry run pytest -k "test_name" --no-cov
  3. Verify Results: Ensure 100% pass rate and expected behavior

When to Test:

  • ✅ After creating new tests
  • ✅ After modifying existing tests
  • ✅ After refactoring shared infrastructure
  • ✅ After performance optimizations
  • ✅ After adding/changing tags

Red Flags - Never Skip Testing:

  • ❌ "The changes look good" without running
  • ❌ "It's just a small change"
  • ❌ "I'll test it later"

Testing is Part of Development: Testing is not optional - it's an integral part of the development process. Untested code is broken code.

Testing Methodology:

  • Phase 1: Test setup with --only-setup flag first
  • Phase 2: Run full test after confirming setup works
  • Use background execution for long tests: nohup ... > logfile.log 2>&1 &
  • Handle port conflicts: clean up previous test port forwards before running

Common Flags:

  • --skip-cleanup: Keep resources after test (useful for debugging setup)
  • --skip-setup: Skip before_test commands (useful for iterative testing)

Shared Infrastructure Pattern

When to use shared infrastructure:

  • Multiple tests use the same service (Grafana, Loki, Prometheus)
  • Service configuration is standardized across tests

Implementation:

# Create shared manifest in tests/llm/fixtures/shared/servicename.yaml
# Use in tests:
kubectl apply -f ../../shared/servicename.yaml -n app-<testid>

Benefits:

  • Single place for version updates
  • Consistent configuration across tests
  • Reduced maintenance overhead
  • Follows established pattern (Loki, Prometheus, Grafana)

Setup Verification Best Practices

Prefer kubectl exec over port forwarding for setup verification:

# GOOD - kubectl exec pattern (no port conflicts)
kubectl exec -n namespace deployment/service -- wget -q -O- http://localhost:port/health

# AVOID - port forward for setup verification (causes conflicts)
kubectl port-forward svc/service port:port &
curl localhost:port/health
kill $PORTFWD_PID

Performance optimization guidelines:

  • Use sleep 1 instead of sleep 5 for most retry loops
  • Remove sleeps after straightforward operations (port forward start)
  • Reduce timeout values: 60s for pod readiness, 30s for API verification
  • Question every sleep - many are unnecessary

Race Condition Handling: Never use bare kubectl wait immediately after resource creation. Use retry loops:

# WRONG - fails if pod not scheduled yet
kubectl apply -f deployment.yaml
kubectl wait --for=condition=ready pod -l app=myapp --timeout=300s

# CORRECT - retry loop handles race condition
kubectl apply -f deployment.yaml
POD_READY=false
for i in {1..60}; do
  if kubectl wait --for=condition=ready pod -l app=myapp --timeout=5s 2>/dev/null; then
    echo "✅ Pod is ready!"
    POD_READY=true
    break
  fi
  sleep 1
done
if [ "$POD_READY" = false ]; then
  echo "❌ Pod failed to become ready after 60 seconds"
  kubectl logs -l app=myapp --tail=20  # Diagnostic info
  exit 1  # CRITICAL: Fail the test early
fi

Eval Best Practices

Realism:

  • No fake/obvious logs like "Memory usage stabilized at 800MB"
  • No hints in filenames like "disk_consumer.py" - use realistic names like "training_pipeline.py"
  • No error messages that give away it's simulated like "Simulated processing error"
  • Use real-world scenarios: ML pipelines with checkpoint issues, database connection pools
  • Resource naming should be neutral, not hint at the problem (avoid "broken-pod", "crashloop-app")

Architecture:

  • Implement full architecture even if complex (e.g., use Loki for log aggregation, not simplified alternatives)
  • Proper separation of concerns (app → file → Promtail → Loki → Holmes)
  • ALWAYS use Secrets for scripts, not inline manifests or ConfigMaps
  • Use minimal resource footprints (reduce memory/CPU for test services)

Anti-Cheat Testing Guidelines:

  • Prevent Domain Knowledge Cheats: Use neutral, application-specific names instead of obvious technical terms
    • Example: "E-Commerce Platform Monitoring" not "Node Exporter Full"
    • Example: "Payment Service Dashboard" not "MySQL Error Dashboard"
    • Add source comments: # Uses Node Exporter dashboard but renamed to prevent cheats
  • Resource Naming Rules: Avoid hint-giving names
    • Use realistic business context: "checkout-api", "user-service", "inventory-db"
    • Avoid obvious problem indicators: "broken-pod" → "payment-service-1"
    • Test discovery ability, not pattern recognition
  • Prompt Design: Don't give away solutions in prompts
    • BAD: "Find the node_pressure_cpu_waiting_seconds_total query"
    • GOOD: "Find the Prometheus query that monitors CPU pressure waiting time"
    • Test Holmes's search/analysis skills, not domain knowledge shortcuts

Configuration:

  • Custom runbooks: Add runbooks field in test_case.yaml (runbooks: {} for empty catalog)
  • Custom toolsets: Create separate toolsets.yaml file (never put in test_case.yaml)
  • Toolset config must go under config field:
toolsets:
  grafana/dashboards:
    enabled: true
    config:  # All toolset-specific config under 'config'
      api_url: http://localhost:10177

Always run evals before submitting when possible:

  1. poetry run pytest -k "test_name" --only-setup --no-cov — verify setup
  2. poetry run pytest -k "test_name" --no-cov — run full test
  3. Verify cleanup: kubectl get namespace app-NNN should return NotFound

Running Kubernetes Evals in the Claude Code Sandbox

The sandbox does not ship with a running Kubernetes cluster, kubectl, helm, or a running docker daemon. To run k8s-based evals (e.g. tests/llm/fixtures/test_ask_holmes/227_count_configmaps_per_namespace/test_case.yaml, which uses kubectl create namespace / kubectl create configmap in before_test), run ./scripts/setup-sandbox-k8s.sh. It brings up k3s in a container (KIND does NOT work — the inner node's systemd cannot mount /sys/fs/cgroup/systemd under the sandbox's cgroup v1 and restart-loops with "Failed to mount API filesystems"), installs kubectl / helm / a static jq, and patches two sandbox-specific issues that would otherwise prevent any pod from starting:

  • The sandbox MITMs HTTPS to public registries with an internal CA (egress-gateway-ca-*.crt, swp-ca-*.crt). k3s's containerd doesn't trust those, so the inner cluster can't pull rancher/mirrored-pause and every pod sandbox fails. Fix: bind-mount the host CA bundle into k3s and write a registries.yaml pointing containerd at it. The script also mirrors docker.io through mirror.gcr.io to dodge Docker Hub's 100/6h anonymous pull limit (the sandbox shares an outbound IP across sessions).
  • The sandbox strips CAP_SYS_RESOURCE from our user. K3s's pause container has oomScoreAdj: -998; setting a negative oom_score_adj requires CAP_SYS_RESOURCE. runc's nsexec fails with failed to update /proc/self/oom_score_adj: Permission denied, the child dies, the parent gets can't get final child's PID from pipe: EOF. Fix: wrap /bin/runc inside the k3s container with a shim that uses jq to strip .process.oomScoreAdj from the OCI config before runc create runs.

Verified end-to-end with the llm and regression test suite against a freshly-bootstrapped cluster under opus-4.6: 10 of 11 regression evals pass in ~256s wall at -n 4. The eleventh (176_network_policy_blocking_traffic_no_skills) cannot pass here — see the limitations section. Skip via -m "llm and regression and not network".

LLM provider env (OpenRouter — Anthropic / OpenAI keys are not in env):

Two non-obvious requirements for the classifier scoring step:

  • api_base: https://openrouter.ai/api/v1 is required on every model_list.yaml entry. The autoevals/braintrust correctness classifier bypasses litellm and calls the raw OpenAI SDK — without api_base it hits api.openai.com and 401s with the OpenRouter key.
  • Always use the openai/<provider>/<model> prefix in the model: field — including for Anthropic models routed through OpenRouter. Two problems get solved at once:
    • litellm's native Anthropic provider appends /v1/messages to api_base, which doubles the path to /api/v1/v1/messages and 404s. Forcing the openai/ prefix routes through litellm's OpenAI-compatible path (/chat/completions) instead, which OpenRouter speaks fine for all models.
    • The autoevals classifier passes the post-prefix string straight to OpenRouter as a model ID, and openrouter/openai/gpt-4.1 is rejected as "not a valid model ID"; openai/gpt-4.1 is accepted.
cat > /tmp/model_list.yaml << 'EOF'
opus-4.6:
  model: openai/anthropic/claude-opus-4.6
  api_key: "{{ env.OPENROUTER_API_KEY }}"
  api_base: https://openrouter.ai/api/v1
gpt-4.1:
  model: openai/gpt-4.1
  api_key: "{{ env.OPENROUTER_API_KEY }}"
  api_base: https://openrouter.ai/api/v1
gpt-4.1-mini:
  model: openai/gpt-4.1-mini
  api_key: "{{ env.OPENROUTER_API_KEY }}"
  api_base: https://openrouter.ai/api/v1
EOF

# BRAINTRUST_API_KEY / BRAINTRUST_SERVICE_TOKEN must be UNSET — the read-only
# service token in this env refuses to create experiments and crashes the test.
unset BRAINTRUST_API_KEY
unset BRAINTRUST_SERVICE_TOKEN

export MODEL=gpt-4.1-mini MODEL_LIST_FILE_LOCATION=/tmp/model_list.yaml
export CLASSIFIER_MODEL=gpt-4.1 RUN_LIVE=true OPENAI_API_KEY=dummy

Then run the regression suite using the invocation that setup-sandbox-k8s.sh prints on success. Wall time is 65s for a single k8s eval with gpt-4.1-mini ($0.03 OpenRouter spend); opus-4.6 is ~90s per run. The full 11-eval llm and regression suite under opus-4.6 runs in 256s wall at -n 4 ($3.30). Budget accordingly when looping.

What does NOT work in the sandbox:

  • kind create cluster — inner container restart-loops on cgroup v1 (systemd mount fails)
  • k3s without the oomScoreAdj runc wrapper — pod sandboxes never start; runc's nsexec fails with failed to update /proc/self/oom_score_adj: Permission denied because CAP_SYS_RESOURCE is stripped
  • k3s without the host-CA bind-mount + registries.yaml — image pulls fail because the sandbox MITMs HTTPS with a CA the k3s container doesn't trust
  • NetworkPolicy enforcement, period — both k3s's built-in NP controller (kube-router) and Calico's felix fail because ipset is restricted in this kernel (ipset list -name returns "Kernel error received: Invalid argument"). Calico's eBPF dataplane also doesn't help: /sys/fs isn't a shared mount, so the mount-bpffs init container can't run. The 176_network_policy_blocking_traffic_no_skills eval is the only regression test affected — it asserts a NetworkPolicy actually blocks traffic, which we cannot enforce here. Skip it via -m "llm and regression and not network" or --deselect
  • BRAINTRUST_API_KEY / BRAINTRUST_SERVICE_TOKEN set — unset both or the test crashes
  • model_list.yaml entries without api_base — classifier 401s against api.openai.com
  • openrouter/openai/<model> prefix in model: — OpenRouter rejects as invalid model ID
  • anthropic/<model> prefix in model: — litellm's Anthropic provider hits /api/v1/v1/messages (doubled path) and 404s; use openai/anthropic/<model> instead
  • Assuming Anthropic / OpenAI keys are present — only OPENROUTER_API_KEY is available

Caveats and gotchas (things that DO work but with strings attached):

  • oomScoreAdj rewrite is a behavioural fudge. The wrapper rewrites oomScoreAdj: -998 (set by k3s on the pause container and a few system pods like coredns/local-path-provisioner) to 0. User pods are unaffected — kubelet derives their oomScoreAdj from QoS class (0 for Guaranteed, ~999 Burstable, ~1000 BestEffort), all non-negative so the s/:-[0-9]+/0/ regex never matches them. In practice this means: under sustained memory pressure on the k3s container, the kernel OOM killer may pick a pause/coredns container before a user container that it would normally protect, which can tear the pod sandbox out from under a workload. Doesn't happen at the ~20-pod scale the regression suite hits here, but be aware if you write a memory-stress eval, or an eval that asserts directly on /proc/<pid>/oom_score_adj of a system pod.
  • The runc wrapper survives docker restart k3s-server but NOT docker rm. The wrapper lives on the container's writable layer at /bin/runc (with the original at /bin/runc.real). A fresh container starts with stock runc and pods get stuck again — re-run ./scripts/setup-sandbox-k8s.sh; it's idempotent and reinstalls the wrapper. To verify the wrapper is active: docker exec k3s-server head -3 /bin/runc should show the shebang + comment, not a binary.
  • The sandbox itself is reclaimed after a period of inactivity. dockerd, the k3s container, kubectl, and helm all disappear when the session container is reclaimed. Re-run ./scripts/setup-sandbox-k8s.sh. Anything you didn't commit and push is also gone — including local kubeconfigs and model_list.yaml.
  • CAP_SYS_RESOURCE removal blocks more than just oomScoreAdj. Anything that tries to raise rlimits (RLIMIT_NOFILE, RLIMIT_NPROC) also fails. The hard ulimit for open files is pinned at 4096 and not even root can raise it; prlimit --pid $$ --nofile=65536:65536 returns "Operation not permitted". In practice this is fine for the regression suite (small workloads), but a test that spins up a process requiring tens of thousands of fds (a DB under load, an HTTP load-generator, etc.) will hit it. Do NOT pass --ulimit nofile=65536:65536 to the k3s docker run line — it will fail the container start with error setting rlimit type 7: operation not permitted.
  • Parallel pytest workers cause image-pull contention. With -n 4 the first setup wave creates ~25 pods simultaneously and containerd pulls all the unique images in parallel through the sandbox proxy. The first parallel regression run takes ~80s of setup (mostly image pull) per worker; a second run on the same cluster reuses the image cache and is much faster. If you're iterating, do NOT delete the k3s container between runs.
  • Docker Hub anonymous pull rate limit can fail eval setup. The sandbox shares an outbound IP across sessions, so anonymous Docker Hub pulls are subject to the 100-pulls-per-6h limit (429 Too Many Requests: toomanyrequests: You have reached your unauthenticated pull rate limit). Symptoms: eval setup times out with pods stuck ImagePullBackOff. The setup script works around this by configuring containerd to pull docker.io images through mirror.gcr.io (Google's public pull-through cache for Docker Hub library/* and the k8s ecosystem) before falling back to registry-1.docker.io. The mirror has no rate limit and needs no credentials. If a workload pulls an image the mirror doesn't have and the fallback still 429s, additional mitigations:
    • Pre-pull the image on the host (docker pull python:3.9-slim) and side-load into k3s with docker save ... | docker exec -i k3s-server ctr -n k8s.io images import - (the same image-import gotcha applies — do them one at a time).
    • Add a Docker Hub auth block to /tmp/k3s-output/registries.yaml: 
      configs:
  "registry-1.docker.io":
    auth:
      username: <hub-user>
      password: <hub-pat>
    tls:
      ca_file: /etc/ssl/certs/ca-certificates.crt
    • Re-run after the 6h window expires.
  • Sandbox network policy controls outbound access. All recipe pulls (get.helm.sh, dl.k8s.io, kind.sigs.k8s.io, openrouter.ai, registry-1.docker.io, ghcr.io Calico images, etc.) succeed under the default policy that was active during verification, but if a future environment uses a stricter policy some installs may fail. The k3s registries.yaml only covers docker.io, registry.k8s.io, and quay.io; if a workload pulls from another registry (e.g. ghcr.io) extend registries.yaml with the same ca_file: block for it.
  • Eval before_test scripts can kubectl exec pods you just created. If your test_case.yaml does that, the inner pod has to actually run — make sure the wrapper is in place before launching pytest, not after.
  • docker save / ctr images import for side-loading images is unreliable in this combination. Some images import cleanly, others fail with content digest ...: not found (saw this with the multi-image tar bundle). If you need to pre-seed an image, save and import one at a time. Pulling through containerd with the registries.yaml fix is more reliable than side-loading.
  • kubeconfig regenerates if the k3s container is removed and recreated. Refresh ~/.kube/config from /tmp/k3s-output/kubeconfig.yaml (or via docker cp k3s-server:/output/kubeconfig.yaml ~/.kube/config) after any container recreation, or kubectl will fail with "x509: certificate signed by unknown authority".
  • NetworkPolicy enforcement isn't recoverable in this sandbox. Both approaches I tried bottom out on ipset: k3s's kube-router skips its NP controller at startup, and Calico's felix logs the same kernel error continuously. Calico's eBPF dataplane is blocked separately by /sys/fs not being a shared mount, so the mount-bpffs init container can't run. Don't waste cycles installing another CNI for NP support — it won't work until the sandbox kernel allows ipset/netlink-NFNL_SUBSYS_IPSET operations.

Reading CodeRabbit Review Comments

In the sandbox environment, gh CLI is not available and the GitHub REST API will quickly rate-limit unauthenticated requests. Use the following approach:

  1. Find the PR number via the GitHub API (unauthenticated, one call): 
    curl -s "https://api.github.com/repos/HolmesGPT/holmesgpt/pulls?head=HolmesGPT:BRANCH_NAME&state=open" \
  | python3 -c "import sys,json; [print(f'PR #{p[\"number\"]}') for p in json.load(sys.stdin)]"
  2. Fetch comments with WebFetch (not rate-limited): Use the WebFetch tool on https://github.com/HolmesGPT/holmesgpt/pull/<NUMBER> and ask it to extract all CodeRabbit comments, including file/line references, full text, and code suggestions.

What does NOT work:

  • gh CLI — not installed in the sandbox
  • Multiple curl calls to api.github.com — hits unauthenticated rate limits (60/hour) fast
  • The local git proxy (127.0.0.1) — only supports git protocol, not the GitHub REST API

Documentation Lookup

When asked about content from the HolmesGPT documentation website (https://holmesgpt.dev/), look in the local docs/ directory:

  • Python SDK examples: docs/installation/python-installation.md
  • CLI installation: docs/installation/cli-installation.md
  • Kubernetes deployment: docs/installation/kubernetes-installation.md
  • Toolset documentation: docs/data-sources/builtin-toolsets/
  • API reference: docs/reference/

MkDocs Navigation

The docs site uses the awesome-nav plugin. Navigation is controlled by .nav.yml files in each docs/ subdirectory, not by the nav: section in mkdocs.yml. When adding a new docs page, you must add it to the .nav.yml file in the corresponding directory (e.g., docs/reference/.nav.yml for reference pages).

Updating References When Docs Pages Change

When you rename or move a docs page, or change a section heading (which changes its anchor), the URL changes. Before committing, grep -rn across the entire repo for the old page path and old anchor slug and update every hit. References can appear in:

  • Other docs/*.md files (relative links like [text](../path/page.md#anchor))
  • Python source (user-facing error messages and hints, e.g. holmes/utils/memory_limit.py)
  • README.md and top-level markdown
  • Code comments

This applies to both https://holmesgpt.dev/... absolute URLs and repo-relative .md links.

MkDocs Formatting Notes

When writing documentation in the docs/ directory:

  • Lists after headers: Always add a blank line between a header/bold text and a list, otherwise MkDocs won't render the list properly

    **Good:**

- item 1
- item 2

**Bad:**
- item 1
- item 2

  • Headers inside tabs: Use bold text for section headings inside tabs, not markdown headers (##, ###, etc.)

    Why: MkDocs Material font sizes make H2 (~25px) and H3 (~20px) visually larger than tab titles (~14px). When a header inside a tab is bigger than the tab title itself, it looks like it belongs outside/above the tabs, breaking the visual hierarchy.

    <!-- GOOD: Bold text for sections inside tabs -->
=== "Tab Name"

    **Create the policy:**

    Instructions here...

    **Create the role:**

    More instructions...

<!-- BAD: Headers inside tabs look like they're outside -->
=== "Tab Name"

    ### Create the policy

    Instructions here...

  • Avoid excessive headers: Don't create a header for every small section. Headers should be used sparingly for major sections. For minor sections like test steps or examples, use bold text or combine content into a single code block with comments instead of separate headers.

    <!-- BAD: Header for every test step -->
## Testing
### Test 1: Check Status
### Test 2: Check Logs
### Test 3: Health Check

<!-- GOOD: Single section with combined content -->
## Testing the Connection

```bash
# Check pod status
kubectl get pods -n YOUR_NAMESPACE

# Check logs
kubectl logs -n YOUR_NAMESPACE

# Health check
curl http://localhost:8000/health

  • Don't describe Holmes's behavior: In "Common Use Cases" sections, show only the example prompts. Don't explain what Holmes will do or list steps like "Holmes will: 1. Query X, 2. Analyze Y, 3. Return Z". Users will see this when they run it.

  • Skip Capabilities sections: Don't list what a toolset/integration can do. Users discover capabilities by using Holmes. Feature lists become stale quickly.

  • Skip Security Best Practices sections: Assume users understand basics like rotating credentials, using least privilege, and deleting local secrets. These sections add little value.

  • Consolidate troubleshooting commands: Instead of separate headers for each troubleshooting scenario, use a single code block with comments:

    # Authentication errors - check if secret is mounted
kubectl exec ...

# Permission denied - verify roles
gcloud projects get-iam-policy ...

  • Common Use Cases format: Just example prompts, one per code block, no sub-headers, no explanations.

Robusta Platform/API URLs in Docs

Whenever you reference api.robusta.dev, platform.robusta.dev, or sp.robusta.dev (the Supabase host) in a docs page, use the robusta-region custom fence so readers can pick US/EU/AP. Never hardcode a single region — the Robusta platform is hosted in multiple regions and a bare api.robusta.dev link silently breaks for EU/AP users.

The fence (defined in docs/custom_fences.py, registered in mkdocs.yml) takes a US URL as input and emits a three-tab picker with the domain rewritten per region. Pick the input shape that matches your context:

  • Plain URL or text (renders as a code block per tab):

    ```robusta-region
https://api.robusta.dev/litellm/model_prices_and_context_window.json
```

  • Markdown link (renders as a clickable link per tab — use for "go to platform.robusta.dev" style prose):

    ```robusta-region
[platform.robusta.dev](https://platform.robusta.dev/)
```

  • YAML or other code with {lang=<name>} (applies the language-<name> class for syntax highlighting):

    ```robusta-region {lang=yaml}
holmes:
  additionalEnvVars:
    - name: ROBUSTA_API_ENDPOINT
      value: "https://api.robusta.dev"
```

Author the URL once using the US domain (api.robusta.dev / platform.robusta.dev / sp.robusta.dev); the fence handles the .eu / .ap rewrites. The set of rewritten hosts lives in ROBUSTA_DOMAIN_RE in docs/custom_fences.py — add a host there if you need another subdomain covered. If you add a new region or rename one, update ROBUSTA_REGIONS in the same file — that single change propagates to every page.

Existing usages (greppable starting points): docs/ai-providers/robusta-ai.md, docs/installation/ui-installation.md, docs/reference/environment-variables.md, docs/reference/troubleshooting.md, docs/data-sources/builtin-toolsets/coralogix-logs.md, docs/data-sources/builtin-toolsets/kubernetes-mcp.md.