chart: FlarePodMemoryHigh/Critical alerts on the POD cgroup slice

The pod slice (container="") is what the OOM killer enforces and the only
series that includes the tmpfs data pages: container-level working_set
misses most of them, and kubectl top sums containers and loses tmpfs
charges reparented on container restarts (all observed live). On a tmpfs
cluster an OOM is data loss on that node — warn at 85%/10m, page at
95%/2m.

Author: junjihashimoto

helm/flare-operator/templates/prometheusrule.yaml

@@ -55,6 +55,35 @@ spec:
             summary: "Flare cluster {{`{{ $labels.cluster }}`}}: draining master has NO successor — data-bearing node dies at grace expiry"
             runbook: docs/RUNBOOK.md#drain-no-successor
 
+        # Pod memory pressure vs the limit. The POD cgroup slice
+        # (container="") is what the OOM killer enforces and is the only
+        # series that includes the tmpfs data pages (container-level
+        # working_set misses most of them; kubectl top sums containers and
+        # misses reparented tmpfs charges after container restarts). On a
+        # tmpfs cluster OOM = data loss on that node, so page before it.
+        - alert: FlarePodMemoryHigh
+          expr: >-
+            100 * container_memory_working_set_bytes{container="", pod=~".+-nodes-.+"}
+              / on(namespace, pod) group_left()
+                cluster:namespace:pod_memory:active:kube_pod_container_resource_limits{container="flared"}
+            > 85
+          for: 10m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Flare pod {{`{{ $labels.pod }}`}}: memory (incl. tmpfs data) above 85% of the limit"
+        - alert: FlarePodMemoryCritical
+          expr: >-
+            100 * container_memory_working_set_bytes{container="", pod=~".+-nodes-.+"}
+              / on(namespace, pod) group_left()
+                cluster:namespace:pod_memory:active:kube_pod_container_resource_limits{container="flared"}
+            > 95
+          for: 2m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Flare pod {{`{{ $labels.pod }}`}}: memory above 95% of the limit — OOM (= tmpfs data loss) imminent"
+
         # A node stuck in Prepare is reconstructing forever — it will never
         # serve and never be failed over (Prepare is exempt from dead
         # detection by design). Runbook: docs/RUNBOOK.md#prepare-stuck