|
| 1 | +# Deployment Example: ContextMatrix on Kubernetes with Remote Runner |
| 2 | + |
| 3 | +This document describes one way to deploy ContextMatrix — on a home lab |
| 4 | +Kubernetes cluster with a remote runner for autonomous agent tasks. It's a |
| 5 | +showcase of what a full-featured deployment *could* look like, not a |
| 6 | +requirement. ContextMatrix runs just as well as a single binary on your laptop |
| 7 | +with `./contextmatrix` and no containers, orchestration, or cloud services |
| 8 | +involved. |
| 9 | + |
| 10 | +If you just want to get started, see the Quick Start section in the README. |
| 11 | +Everything below is for when you want a persistent, multi-machine setup. |
| 12 | + |
| 13 | +## Architecture Overview |
| 14 | + |
| 15 | +```mermaid |
| 16 | +flowchart TB |
| 17 | + subgraph internet["Internet"] |
| 18 | + dev["Developer"] |
| 19 | + browser["Browser"] |
| 20 | + cf["Cloudflare Edge<br/>Access + WAF"] |
| 21 | + end |
| 22 | +
|
| 23 | + subgraph github["GitHub"] |
| 24 | + repo["contextmatrix<br/>(source repo)"] |
| 25 | + boards["contextmatrix-boards<br/>(boards repo)"] |
| 26 | + target["Project Repos<br/>(e.g., my-api, my-app)"] |
| 27 | + end |
| 28 | +
|
| 29 | + subgraph lan["Local Network"] |
| 30 | +
|
| 31 | + subgraph runner_host["Runner VM"] |
| 32 | + runner["contextmatrix-runner<br/>:19090"] |
| 33 | + gh_runner["GitHub Actions<br/>self-hosted runner"] |
| 34 | + docker["Docker Engine"] |
| 35 | + worker["Worker Container<br/>(Alpine + Claude Code)"] |
| 36 | + end |
| 37 | +
|
| 38 | + subgraph k8s["Kubernetes Cluster"] |
| 39 | +
|
| 40 | + subgraph cm_ns["contextmatrix namespace"] |
| 41 | + cm_pod["ContextMatrix Pod"] |
| 42 | + cm_svc["Service :8080"] |
| 43 | + pvc["Longhorn PVC<br/>(boards data)"] |
| 44 | + cfd["cloudflared"] |
| 45 | + end |
| 46 | +
|
| 47 | + subgraph cd_ns["contextmatrix-pipelines namespace"] |
| 48 | + kargo["Kargo<br/>(warehouse + stage)"] |
| 49 | + end |
| 50 | +
|
| 51 | + argocd["ArgoCD"] |
| 52 | + gateway["Cilium Gateway<br/>HTTPRoute + TLS"] |
| 53 | + harbor["Harbor Registry"] |
| 54 | + end |
| 55 | +
|
| 56 | + gitea["Gitea<br/>(CI repo)"] |
| 57 | + end |
| 58 | +
|
| 59 | + %% CI flow (blue) |
| 60 | + dev -- "push to main" --> repo |
| 61 | + repo -- "webhook" --> gh_runner |
| 62 | + gh_runner -- "docker build + push" --> harbor |
| 63 | +
|
| 64 | + %% CD flow (green) |
| 65 | + harbor -- "new image detected" --> kargo |
| 66 | + kargo -- "promote: render manifests" --> gitea |
| 67 | + gitea -- "sync rendered/" --> argocd |
| 68 | + argocd -- "deploy" --> cm_pod |
| 69 | +
|
| 70 | + %% Internet access (orange) |
| 71 | + browser --> cf |
| 72 | + cf -- "web UI only<br/>/mcp + /healthz blocked" --> cfd |
| 73 | + cfd --> cm_svc |
| 74 | +
|
| 75 | + %% LAN access |
| 76 | + gateway -- "cm.lan.example.com<br/>agents + web UI" --> cm_svc |
| 77 | +
|
| 78 | + %% Runner flow (purple) |
| 79 | + cm_pod -- "trigger webhook" --> runner |
| 80 | + runner -- "start container" --> docker |
| 81 | + docker --> worker |
| 82 | + worker -- "MCP tools" --> cm_svc |
| 83 | + worker -- "clone repo,<br/>code changes,<br/>create PR" --> target |
| 84 | + cm_pod -- "clone on startup" --> boards |
| 85 | + cm_pod --> pvc |
| 86 | +
|
| 87 | + %% Styling |
| 88 | + linkStyle 0,1,2 stroke:#1565c0,stroke-width:2px |
| 89 | + linkStyle 3,4,5 stroke:#2e7d32,stroke-width:2px |
| 90 | + linkStyle 6,7,8 stroke:#e65100,stroke-width:2px |
| 91 | + linkStyle 10,11,12,13 stroke:#6a1b9a,stroke-width:2px |
| 92 | +``` |
| 93 | + |
| 94 | +## Components |
| 95 | + |
| 96 | +| Component | Role | |
| 97 | +| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |
| 98 | +| **ContextMatrix** | Kanban server — REST API, web UI, MCP endpoint. Runs as a single-replica pod in k8s. | |
| 99 | +| **contextmatrix-runner** | Webhook receiver on a LAN VM. Spawns disposable Docker containers that run Claude Code autonomously — cloning project repos, making code changes, and creating pull requests. | |
| 100 | +| **GitHub Actions** | CI — builds Docker image on push to main, pushes to Harbor. Uses a self-hosted runner on the LAN (Harbor is not internet-accessible). | |
| 101 | +| **Kargo** | CD — detects new images in Harbor, renders Kustomize manifests, pushes to CI repo. | |
| 102 | +| **ArgoCD** | GitOps — syncs rendered manifests from CI repo to cluster. | |
| 103 | +| **Cloudflare Tunnel** | Exposes the web UI to the internet with Cloudflare Access authentication. MCP and health endpoints are blocked at the edge. | |
| 104 | +| **Cilium Gateway** | LAN ingress with TLS (cert-manager wildcard cert). Agents and local browsers connect here. | |
| 105 | +| **Longhorn** | Persistent storage for the boards git repository. | |
| 106 | +| **Harbor** | Private container registry on the LAN. | |
| 107 | +| **Gitea** | LAN git server hosting the CI/CD manifests repo. | |
| 108 | + |
| 109 | +## Prerequisites (for this example) |
| 110 | + |
| 111 | +This particular setup uses: |
| 112 | + |
| 113 | +- Kubernetes cluster with ArgoCD, Kargo, cert-manager, Longhorn, and a Gateway |
| 114 | + API implementation (e.g., Cilium) |
| 115 | +- Private container registry (e.g., Harbor) accessible from the LAN |
| 116 | +- Local git server (e.g., Gitea) for the CI repo |
| 117 | +- Cloudflare account with a domain and Zero Trust Access |
| 118 | +- A LAN machine (VM or bare metal) with Docker for the runner |
| 119 | +- GitHub account (source code + boards repo) |
| 120 | +- GitHub App for runner git operations (clone, push, PRs) |
| 121 | + |
| 122 | +None of these are required to use ContextMatrix itself — they're specific to |
| 123 | +this deployment pattern. |
| 124 | + |
| 125 | +## How This Example Is Set Up |
| 126 | + |
| 127 | +### 1. Container Image |
| 128 | + |
| 129 | +The repo includes a multi-stage Dockerfile: |
| 130 | + |
| 131 | +- **Stage 1**: Node.js — build the React frontend |
| 132 | +- **Stage 2**: Go — build the binary with embedded frontend |
| 133 | +- **Stage 3**: Alpine runtime with `git` and `openssh-client` |
| 134 | + |
| 135 | +Skills are baked into the image at build time. |
| 136 | + |
| 137 | +### 2. CI Pipeline (GitHub Actions) |
| 138 | + |
| 139 | +A workflow triggers on push to main: |
| 140 | + |
| 141 | +- Runs on a **self-hosted runner** on the LAN (required when the registry is |
| 142 | + LAN-only) |
| 143 | +- Builds the Docker image and tags with the short commit SHA |
| 144 | +- Pushes to the private registry |
| 145 | + |
| 146 | +The self-hosted runner is installed on the same VM as the contextmatrix-runner. |
| 147 | + |
| 148 | +### 3. CI/CD Repo (Gitea) |
| 149 | + |
| 150 | +A separate repo on the local git server holds infrastructure manifests: |
| 151 | + |
| 152 | +``` |
| 153 | +contextmatrix-ci/ |
| 154 | +├── base/runtime/ # Deployment, Service, PVC, HTTPRoute, ConfigMap |
| 155 | +├── stages/production/ # Kustomize overlay (image tag, managed by Kargo) |
| 156 | +├── rendered/production/# Kargo output (ArgoCD reads this) |
| 157 | +├── kargo/ # Warehouse, Stage, Project |
| 158 | +├── cloudflared/ # Tunnel deployment + encrypted token |
| 159 | +├── bootstrap/argocd/ # AppProject + Application resources |
| 160 | +└── bootstrap.sh # One-time setup script |
| 161 | +``` |
| 162 | + |
| 163 | +### 4. Kubernetes Manifests |
| 164 | + |
| 165 | +**Deployment** — single replica with `Recreate` strategy (one writer to the |
| 166 | +boards repo): |
| 167 | + |
| 168 | +- Boards data on a Longhorn PVC |
| 169 | +- SSH deploy key mounted for boards repo git operations |
| 170 | +- All configuration via environment variables (no config file in the container) |
| 171 | +- Clone-on-empty: if the PVC is fresh, CM clones the boards repo on startup |
| 172 | +- `readOnlyRootFilesystem: true` with emptyDir mounts for `/tmp` and home |
| 173 | + |
| 174 | +**ConfigMap** — for settings that can't be set via env vars (e.g., |
| 175 | +`token_costs`). Mounted at `/config/config.yaml`, passed via `--config` flag. |
| 176 | + |
| 177 | +**PVC** — Longhorn, 1Gi, ReadWriteOnce. The boards git repo lives here. |
| 178 | + |
| 179 | +**HTTPRoute** — routes `cm.lan.example.com` through the Cilium Gateway for LAN |
| 180 | +access. |
| 181 | + |
| 182 | +### 5. Kargo Pipeline |
| 183 | + |
| 184 | +- **One warehouse** watching Harbor for new image tags and the CI repo for |
| 185 | + config changes |
| 186 | +- **One stage** (`production`) with auto-promote |
| 187 | +- Promotion steps: clone CI repo → kustomize-set-image → kustomize-build → |
| 188 | + commit → push → ArgoCD update |
| 189 | + |
| 190 | +### 6. Cloudflare Tunnel |
| 191 | + |
| 192 | +A separate cloudflared deployment in the CM namespace connects to Cloudflare's |
| 193 | +edge. |
| 194 | + |
| 195 | +**Security layers:** |
| 196 | + |
| 197 | +- **Cloudflare Access** — requires authentication for all requests to the |
| 198 | + hostname |
| 199 | +- **WAF rules** — block `/mcp*` and `/healthz` at the edge (these endpoints |
| 200 | + should only be reachable from the LAN) |
| 201 | + |
| 202 | +The tunnel is configured in the Cloudflare dashboard (token-based). The token is |
| 203 | +stored as a SOPS-encrypted Kubernetes secret. |
| 204 | + |
| 205 | +### 7. TLS Certificate |
| 206 | + |
| 207 | +This example uses cert-manager with a wildcard certificate for |
| 208 | +`*.lan.example.com` on the gateway, covering all LAN services without |
| 209 | +per-service certificates. |
| 210 | + |
| 211 | +### 8. Bootstrap |
| 212 | + |
| 213 | +A one-time bootstrap script creates: |
| 214 | + |
| 215 | +- Namespaces (application + pipelines) |
| 216 | +- Kargo git credentials (SSH key for CI repo) |
| 217 | +- Harbor registry credentials (for Kargo image discovery) |
| 218 | +- Boards repo SSH deploy key |
| 219 | +- MCP API key (randomly generated) |
| 220 | +- Runner API key (randomly generated) |
| 221 | +- ArgoCD applications |
| 222 | + |
| 223 | +### 9. Runner Setup |
| 224 | + |
| 225 | +On the runner VM: |
| 226 | + |
| 227 | +- Build the worker Docker image (Alpine-based with Claude Code, Go, Node.js, |
| 228 | + GitHub CLI) |
| 229 | +- Configure `config.yaml` with the CM URL, shared API key, Claude auth |
| 230 | + directory, and GitHub App credentials |
| 231 | +- Run as a systemd user service |
| 232 | + |
| 233 | +The runner resolves the CM hostname on the host (where `/etc/hosts` entries |
| 234 | +work) and injects it into container `/etc/hosts` entries so containers can reach |
| 235 | +LAN services. |
| 236 | + |
| 237 | +## Security Model |
| 238 | + |
| 239 | +| Layer | Protection | |
| 240 | +| ----------------------- | ---------------------------------------------------------------------------- | |
| 241 | +| **Internet → Web UI** | Cloudflare Access (SSO/email auth) | |
| 242 | +| **Internet → MCP** | Blocked at Cloudflare edge (WAF rule) | |
| 243 | +| **Internet → /healthz** | Blocked at Cloudflare edge (WAF rule) | |
| 244 | +| **LAN → MCP** | Bearer token authentication (`mcp_api_key`) | |
| 245 | +| **CM ↔ Runner** | HMAC-SHA256 signed webhooks (shared secret, never transmitted) | |
| 246 | +| **Runner containers** | All capabilities dropped, `no-new-privileges`, memory/PID limits, disposable | |
| 247 | +| **Git credentials** | Short-lived GitHub App tokens (1-hour expiry), SSH deploy keys | |
| 248 | +| **Secrets in k8s** | SOPS encryption (age) for cloudflared token, k8s Secrets for API keys | |
| 249 | +| **Container images** | Private Harbor registry, image allowlist in runner config | |
| 250 | + |
| 251 | +## Key Design Decisions |
| 252 | + |
| 253 | +**Single stage, no progressive delivery** — ContextMatrix is tested locally |
| 254 | +before pushing. A single auto-promoting stage keeps the pipeline simple. |
| 255 | + |
| 256 | +**GitHub Actions with self-hosted runner** — The source code is on GitHub but |
| 257 | +the registry is LAN-only. A self-hosted runner bridges the gap without exposing |
| 258 | +the registry to the internet. |
| 259 | + |
| 260 | +**CI repo on local git server** — Keeps infrastructure configuration close to |
| 261 | +the infrastructure. ArgoCD and Kargo access it directly on the LAN with no |
| 262 | +external dependency. |
| 263 | + |
| 264 | +**Separate cloudflared tunnel** — Isolates CM's internet exposure from other |
| 265 | +services. Can be torn down independently. |
| 266 | + |
| 267 | +**Clone-on-empty** — When the PVC is fresh (first deployment or data loss), CM |
| 268 | +automatically clones the boards repo from the configured remote URL. No manual |
| 269 | +initialization needed. |
| 270 | + |
| 271 | +**Alpine worker image** — Smaller and faster to pull than Ubuntu-based |
| 272 | +alternatives. Clean user namespace with no pre-existing users at common UIDs. |
0 commit comments