[opentelemetry-demo]: raise memory limit on product-catalog and checkout - #2383
Open
pkriko wants to merge 2 commits into
Open
[opentelemetry-demo]: raise memory limit on product-catalog and checkout#2383pkriko wants to merge 2 commits into
pkriko wants to merge 2 commits into
Conversation
Both services ship a 20Mi memory limit. Under the load generator their working set sits at 87-91% and 75-79% of that ceiling. Page cache is charged to the cgroup, so as a container approaches its limit the kernel evicts cached pages instead of OOM-killing it, and the next file access goes back to disk. The cycle repeats for as long as the demo runs. The containers never die and never restart, so nothing surfaces in kubectl get pods. The only visible symptom is a slow node. Measured on a 4 vCPU / 6 GB node, 15 minute averages: container working set limit disk reads product-catalog 17.4-18.2 MiB 20Mi 3.42 MB/s checkout 15.1-15.9 MiB 20Mi 1.99 MB/s That kept the node disk 97% busy with 3.48 of 4 cores in iowait. Raising both to 128Mi, then re-running the same load: node disk busy time went from 0.97 to 0.002, product-catalog reads from 3.42 MB/s to 0.0012 MB/s, and checkout from 1.99 MB/s to 0.0007 MB/s. Working set barely moved (16.8-21.2 MiB), so the extra headroom is not consumed, it just leaves room for the page cache. Fixes open-telemetry#2376
|
|
JaredTan95
approved these changes
Sep 1, 2026
Member
|
@pkriko Thank you. Please follow our contributing guidelines for agent: https://github.com/open-telemetry/opentelemetry-helm-charts/blob/main/AGENTS.md. |
Author
|
Because I don't commit anything, i said in the PR description that I used claude to help me. |
julianocosta89
left a comment
julianocosta89
left a comment
Member
There was a problem hiding this comment.
Those 2 are significant bumps.
Do you have a reasoning behind those numbers?
Author
|
Hello @julianocosta89 Honestly no specific reason behind those numbers (just based on other services). I could give a try to 64Mio or 32Mio to see If I get the same high I/O utilization. |
marcschaeferger
added
chart:demo
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The memory limits of some services were too low, which caused disk latency on our k8s cluster and even nodes that could no longer schedule new pods. To fix this, we increased the memory limit of the services.
Link to tracking issue
Fixes #2376
Authorship
Assited-by: Claude