Bug Report

Which version of the demo are you using?

Helm chart opentelemetry-demo 0.40.10 (app 2.2.0).
The same limits are still on main today.

Which service(s) are you using?

product-catalog and checkout.

Symptom

These two services have a memory limit of 20Mi. Under the load generator they sit at 87-91% of that limit. They never go over it. They are never OOMKilled. Page cache counts inside the cgroup memory limit. So when the container gets close to 20Mi, the kernel drops cached pages instead of killing the container. The next file access has to read from disk again. This repeats forever.

The result is a constant disk read loop. On our qual cluster it kept one node's disk 97% busy for weeks.

What we measured

Per container, 15 minute average:

container          working set      limit    disk reads
product-catalog    17.4-18.2 MiB    20Mi     3.42 MB/s
checkout           15.1-15.9 MiB    20Mi     1.99 MB/s

On that node:

disk busy time     0.97      (disk busy 97% of the time)
read IOPS          ~100
CPU iowait         3.48 of 4 cores
load average       9 to 15 on 4 vCPU

We scaled these services to 0. Nothing else changed. The node disk went from 0.97 to 0.002 within ten minutes. The load average went back to normal.

Environment

• Kubernetes 1.32
• Helm chart 0.40.10, app 2.2.0
• Nodes: 4 vCPU / 6 GB RAM, gp2 root volume