This document explains what each panel reports, what the numbers mean, and how to use them to make tuning decisions.
OBSERVER README RESOURCE MANAGEMENT HOST MEMORY
This is the panel you should look at first after a fresh lab deployment. It answers: is KSM saving more memory than it costs in CPU?
The horizontal gauge shows three zones:
| Zone | Color | Meaning |
|---|---|---|
| EFFECTIVE | green | KSM is earning its keep |
| MARGINAL | amber | borderline -- savings are modest or CPU cost is creeping up |
| WASTE | red | KSM is spending more CPU than the memory it recovers is worth |
The verdict is computed from an efficiency ratio:
efficiency = memory_saved_GiB / ksmd_cpu_host_percent
- EFFECTIVE: savings > 5 GiB and CPU < 2%, or efficiency > 10
- MARGINAL: efficiency between 2 and 10, or savings < 2 GiB
- WASTE: efficiency < 2, or CPU > 5% with savings < 1 GiB
An INACTIVE verdict means ksmd is consuming no measurable CPU and reporting no savings. This is normal before guests are started or if KSM is disabled.
Memory saved is (pages_sharing + ksm_zero_pages) * page_size. This is how
much RAM would be consumed if every shared page had its own physical frame.
pages_sharing counts content-identical pages merged into copy-on-write
references. ksm_zero_pages counts all-zero pages that KSM has collapsed
(requires use_zero_pages=1 in the kernel).
ksmd CPU is the delta of cumulative CPU time consumed by the ksmd kernel
thread between poll intervals, expressed as both percent-of-one-core and
percent-of-host. On m5.metal with 96 logical CPUs, 1% of one core is roughly
0.01% of the host. The host percentage is what matters for capacity planning.
Merge ratio is pages_sharing / pages_shared. A ratio of 8:1 means each
unique shared page has 8 other pages referencing it. For 9 RHCOS guests
running the same base OS image, expect 6-10x once the scanner converges.
Volatility is the fraction of scanned pages that changed before KSM could merge them. High volatility (> 30%) means the workload is churning memory faster than the scanner can process it. This is normal during cluster install, rolling updates, or etcd compaction. It should drop during steady state.
Scan throughput is pages_to_scan * (1000 / sleep_millisecs), the
theoretical maximum pages per second the scanner can examine. With the default
settings (1000 pages, 20ms sleep), this is 50,000 pages/second.
Full scans and scans/min show how quickly the scanner is cycling through all of guest memory. A full scan rate below 1/min on a fully deployed lab means the scanner is working through a large memory footprint. This is normal during initial convergence.
- KSM Savings (green): memory saved in GiB over the last 10 minutes. Should climb and stabilize after guest deployment.
- ksmd CPU (amber): host CPU percentage over the last 10 minutes. Should spike during initial scanning and then flatten to near-zero during steady state.
The panel generates contextual text based on the current state:
- "KSM is performing its initial scan" -- first full scan hasn't completed. Wait before tuning.
- "KSM is highly effective. No tuning needed." -- savings > 10 GiB, CPU < 1%. Leave it alone.
- "Consider: increase sleep_millisecs..." -- CPU cost outweighs savings. Follow the recommendation.
- "The workload may not benefit from KSM." -- multiple full scans completed with minimal sharing. The guests may not have enough shared content, or the content is changing too fast.
On a fully deployed lab with 9 RHCOS guests and 3 support guests:
- Savings: 10-30 GiB (depends on guest OS convergence)
- ksmd CPU: < 0.05% of host during steady state
- Merge ratio: 6-10x
- Volatility: < 10% during steady state
- Verdict: EFFECTIVE
- Verdict stays MARGINAL or WASTE after guests have been stable for 30+ minutes
- ksmd CPU stays above 1% of host during steady state
- Merge ratio stays below 2x after multiple full scans
- Savings stay below 1 GiB with a fully deployed lab
This panel shows whether the Gold/Silver/Bronze tier model is distributing CPU as intended.
The top line shows aggregate host CPU utilization and a sparkline. On
m5.metal with 96 logical CPUs, 10% host utilization means roughly 10 cores
are busy. This is the same metric you would see from top or sar.
Three metrics help you gauge CPU pressure:
vCPU oversubscription ratio is total_guest_vCPUs / guest_domain_pCPUs.
With the current default layout (106 vCPUs, 72 pCPUs), this is about 1.47:1.
The ratio is color-coded:
| Color | Range | Meaning |
|---|---|---|
| green | <= 1.0 | no oversubscription -- every vCPU could run simultaneously |
| amber | 1.0 - 2.0 | moderate -- normal for a lab, contention is managed by tier weights |
| red | > 2.0 | aggressive -- watch for %steal and tier throttling |
Total guest vCPUs shows the aggregate and the guest domain pool size for context.
Host steal time is the percentage of CPU time the hypervisor reported as stolen from guest execution. On bare metal (not nested inside another hypervisor), this should be 0%. On AWS metal instances it is occasionally nonzero during host-level maintenance events but should not persist.
The horizontal bar shows how much of the host CPU each tier is consuming, color-coded Gold/Silver/Bronze. The bar represents 100% of the host. During steady state, expect Gold (masters) to be modest, Silver (infra) moderate, and Bronze (workers) variable depending on workload.
Below the bar, each tier shows:
- Weight: the configured
CPUWeightvalue (512/333/167) - Host %: actual CPU consumption as a percentage of the host
- Throttled badge: appears when
nr_throttledincrements between samples
Throttling means a tier's processes were delayed because the tier exceeded its proportional share while sibling tiers were competing. Occasional throttling during bursts is normal. Sustained throttling on Gold indicates the masters are being starved, which is a problem.
Each running VM shows:
| Column | Source |
|---|---|
| VM | domain name from libvirt |
| Tier | Gold/Silver/Bronze badge from libvirt partition |
| vCPU | configured vCPU count from domain XML |
| CPU (cores) | actual cores consumed (from cgroup usage_usec delta) |
| CPU % | same value as a percentage of one core |
| Trend | 10-minute sparkline |
CPU (cores) is the most useful column. A VM with 8 vCPUs consuming 2.5 cores is using about 31% of its allocated capacity. A VM consuming more cores than its vCPU count means the emulator thread and IOThreads are adding overhead beyond the vCPU execution.
- Gold tier (masters): low steady-state CPU (< 5 cores total for 3 masters), occasional spikes during API activity, no throttling
- Silver tier (infra + IdM): moderate CPU depending on ingress traffic and monitoring workload, occasional throttling during spikes is acceptable
- Bronze tier (workers): variable, depends entirely on scheduled workload
- No tier shows sustained throttling
- Gold tier is throttled while Silver or Bronze are consuming significant CPU
- A single VM is consuming more CPU cores than its vCPU count by a wide margin
- Steal time persists above 1%
This is the per-CPU heatmap. It shows all 96 logical CPUs organized by NUMA node, with SMT thread pairs aligned vertically.
Each cell represents one logical CPU. The color encodes two things:
-
Hue identifies the pool assignment:
- Blue = Host Housekeeping (CPUs 0-1, 24-25, 48-49, 72-73)
- Purple = Host Emulator (CPUs 2-5, 26-29, 50-53, 74-77)
- Green = Guest Domain (CPUs 6-23, 30-47, 54-71, 78-95)
-
Opacity encodes utilization: dim = idle, bright = busy. A fully saturated cell means that CPU is near 100% utilized.
Cells at 80%+ utilization show their utilization number inside the cell.
Hover over any cell to see: CPU number, pool assignment, exact utilization percentage, and current clock frequency.
The heatmap is split into two blocks: NUMA Node 0 (Socket 0) and NUMA Node 1 (Socket 1). Each block has two rows:
- T0: primary threads (socket 0: CPUs 0-23, socket 1: CPUs 24-47)
- T1: SMT siblings (socket 0: CPUs 48-71, socket 1: CPUs 72-95)
The pool assignments are symmetric across sockets: each socket contributes 6 physical cores to host reserved and 18 to the guest domain.
Below each NUMA block, the average clock frequency per pool is shown. This is useful for spotting frequency scaling issues. Under load, guest domain CPUs should be running at or near maximum turbo frequency. Idle housekeeping CPUs will typically show lower frequencies due to power management.
- Guest domain CPUs (green) show variable utilization, with brightness correlated to guest workload
- Host housekeeping CPUs (blue) are mostly dim -- busy only during admin activity
- Host emulator CPUs (purple) are mostly dim with brief spikes during guest disk I/O
- Utilization is reasonably balanced across both NUMA nodes
- Host housekeeping CPUs are persistently bright -- the host control plane is under pressure, consider expanding the reserved pool
- Guest domain CPUs are uniformly saturated -- the guest pool may be undersized, or a workload is consuming more than expected
- One NUMA node is significantly hotter than the other -- possible NUMA affinity imbalance in guest placement
This panel shows the host memory picture and how much of it is committed to guests.
The top line shows:
- Total: physical RAM on the host
- Committed: sum of all guest memory allocations (from
virsh dumpxml) - Overcommit ratio: committed / total. Color-coded:
- Green (< 1.0): guests fit entirely in RAM
- Amber (1.0-1.5): moderate overcommit, memory efficiency features are load-bearing
- Red (> 1.5): aggressive overcommit, KSM and zram must deliver savings
- Available: free memory as reported by the kernel (includes reclaimable cache)
A horizontal stacked bar breaking the host RAM into:
- Used (dark blue): anonymous pages and non-reclaimable kernel memory
- Cached (light blue): page cache and reclaimable slab
- Available (grey): memory the kernel considers allocatable without swapping
The bar represents MemTotal. The KSM and zram savings shown in the legend are virtual -- they represent memory that would be consumed without those features but is not directly visible in the bar segments. Think of them as explaining why Available is larger than it would otherwise be.
Four metrics quantify what the memory efficiency features are delivering:
- KSM dedup: bytes saved by page merging (same as the KSM panel)
- KSM zero pages: bytes saved by collapsing all-zero pages
- Total reclaim: KSM + zram savings combined
- THP mode: current Transparent Huge Pages setting (should show
madvise)
Shows each tier's committed memory, number of VMs, and percentage of host RAM. This is the memory analog of the CPU tier bars -- it tells you where the memory commitments are concentrated.
- Overcommit ratio between 0.8 and 1.3
- Available memory > 20 GiB on a 384 GiB host
- KSM dedup contributing 10-30 GiB
- THP mode is
madvise - Gold and Silver tiers have stable memory commitments
- Available memory drops below 10 GiB persistently
- Overcommit ratio > 1.5 without substantial KSM savings
- Total reclaim is negligible -- the memory efficiency features are not compensating for the overcommit
This panel answers: is zram providing useful memory pressure relief, or is it adding overhead with poor compression?
Five zones, left to right:
| Zone | Color | Meaning |
|---|---|---|
| THRASHING | red | high swap I/O rate -- the host is under severe memory pressure |
| POOR | dark amber | data is being swapped but compression ratio is bad (< 1.5:1) |
| MODERATE | amber | compression ratio is decent (1.5-2.5:1) |
| EFFECTIVE | green | good compression (> 2.5:1) with real savings |
| STANDBY | grey | zram device exists but no pages have been swapped into it |
STANDBY is the expected verdict during normal operation. It means the host has enough free memory that the kernel never needs to push pages into swap. This is the healthy state -- zram is configured and ready but not needed.
EFFECTIVE means the host entered memory pressure at some point and zram absorbed it efficiently. Check whether the pressure is transient (burst during cluster install or rolling update) or sustained.
THRASHING (swap I/O > 5000 pages/sec) means the kernel is actively swapping pages in and out. This is a sign of sustained memory pressure that zram alone cannot solve. Consider reducing guest memory commitments.
- zram Savings: memory saved by compression over time. Should be near zero during STANDBY and climb only during pressure events.
- Swap I/O: combined page-in and page-out rate. Spikes are normal during transient pressure. Sustained high rates indicate a capacity problem.
- Swap Utilization: percentage of the zram device capacity in use.
Compression ratio is data_bytes / mem_used_bytes. How much RAM zram needs
to store the data that was swapped into it. With zstd on typical guest
memory:
- 3:1 or better is excellent (guest kernel pages compress well)
- 2:1 is good
- < 1.5:1 is poor (incompressible data, possibly encrypted pages)
Algorithm / streams: the compression algorithm (zstd) and parallelism
level (typically matches CPU count).
Original data vs Compressed to vs RAM consumed: the raw numbers behind the compression ratio. Original data is the uncompressed size of everything stored in zram. Compressed is the compressed payload. RAM consumed includes compression metadata overhead.
Same pages: pages that were entirely duplicate content and stored without compression (just a reference). Similar to KSM but at the zram level.
kswapd CPU: the CPU cost of the kernel reclaim thread that feeds pages into swap. This is the zram analog of ksmd CPU in the KSM panel.
Swap in/out rates: pages per second being read from and written to swap. A high out rate means the kernel is actively pushing pages into zram. A high in rate means applications are faulting on pages that were previously swapped out.
kswapd scan rate: pages per second being scanned by kswapd looking for reclaim candidates. This is upstream of actual reclaim -- high scan rates with low steal rates mean the kernel is having difficulty finding pages to reclaim.
Direct reclaim: pages per second reclaimed synchronously in the application's allocation path. This is the expensive kind of reclaim -- the application stalls until memory is freed. Any sustained direct reclaim rate above zero indicates memory pressure that is directly impacting application performance. This metric is highlighted in amber when nonzero.
Shows the zram device capacity breakdown. When a writeback backing store is active, the bar splits into four segments. When no backing store is configured, it shows the original three-segment view.
Four-segment mode (writeback active):
- RAM cost (dark teal): physical RAM consumed by compressed pages still in memory
- Compression avoided (light teal): RAM saved by compression alone -- these pages are in zram but stored smaller than their original size
- Writeback offloaded (medium teal): pages that did not compress well and were pushed to the backing store by the writeback policy timer. These consume zero RAM.
- Free (grey): unused logical capacity in the zram device
Three-segment mode (no writeback):
- RAM cost (dark teal): physical RAM consumed by compressed data
- Avoided RAM (light teal): the difference between original data and compressed data
- Free (grey): unused device capacity
When a writeback backing store is active, the single "Swap Occupancy" gauge is replaced by a two-tier "Swap Capacity" card showing:
- RAM tier: occupancy of the zram device itself (the 16 GiB disksize cap). This is the percentage of uncompressed data the device is holding across both tiers. When this reaches 100%, the kernel cannot push more pages into swap regardless of backing store headroom.
- Backing tier: occupancy of the writeback backing device. Shows how much of the backing store has been consumed by offloaded pages. Low utilization here means the writeback policy has room to continue freeing RAM.
When no backing store is configured, the card shows a single occupancy gauge as before.
Contextual text similar to the KSM panel:
- "zram is configured and standing by" -- STANDBY, no action needed
- "Compression ratio is excellent" -- EFFECTIVE with good ratio
- "High swap I/O rate indicates significant memory pressure" -- THRASHING, reduce guest memory or add RAM
- "Direct reclaim is active" -- applications are stalling on memory allocation
- Verdict: STANDBY during normal operation
- Compression ratio: 2.5:1 or better when pages are present
- Swap I/O: zero during steady state, brief spikes during transients
- Direct reclaim: zero
- Verdict stays at THRASHING or POOR for more than a few minutes
- Direct reclaim is persistently nonzero
- Swap utilization climbs above 50% of device capacity and stays there
- kswapd CPU exceeds 0.5% of host during steady state
This panel shows the CPU cost of the kernel threads responsible for memory reclaim, page merging, and memory compaction.
| Thread | Role |
|---|---|
ksmd |
KSM scanner -- finds and merges identical pages |
kswapd0 |
NUMA node 0 background reclaim (feeds pages to swap) |
kswapd1 |
NUMA node 1 background reclaim |
kcompactd0 |
NUMA node 0 memory compaction (defragments physical memory for huge pages) |
kcompactd1 |
NUMA node 1 memory compaction |
The top line shows each thread's CPU as a percentage of one core, plus the combined total as both core-percent and host-percent. For example:
ksmd: 0.12% kswapd0: 0.01% kswapd1: 0.00% kcompactd0: 0.00% kcompactd1: 0.00%
total: 0.13% core (0.00% host)
This means the five threads combined are consuming 0.13% of one core, which is negligible on a 96-core host.
Each thread has its own 10-minute sparkline. The shape tells you whether overhead is steady, bursty, or trending upward.
- ksmd: < 0.5% core during steady state, spikes during guest deployments
- kswapd0/1: near zero unless memory pressure is present
- kcompactd0/1: near zero (with THP in
madvisemode, compaction is rare) - Total: < 1% core, essentially invisible at the host level
- ksmd stays above 2% core during steady state -- consider reducing
pages_to_scanor increasingsleep_millisecs - kswapd shows sustained activity -- the host is under memory pressure, check the Memory Overview and zram panels
- kcompactd shows sustained activity -- THP compaction is running, which is
unusual with
madvisemode. Check if a guest is requesting huge pages aggressively - Total overhead exceeds 5% core -- the memory management subsystem is consuming meaningful CPU that could be serving guests
The panels are designed to be read together. Here are common patterns:
- KSM: EFFECTIVE, 15+ GiB saved, near-zero CPU
- CPU: low utilization, no throttling, Gold tier idle
- Memory: overcommit ratio ~1.2, plenty of available memory
- zram: STANDBY
- Overhead: all threads near zero
- KSM: climbing savings, elevated ksmd CPU (0.5-2% core), first-scan-pending or low full scan count
- CPU: elevated as guests are booting and pulling images
- Memory: available memory dropping as guests consume their allocations
- zram: may briefly enter MODERATE as transient pressure occurs
- Overhead: ksmd elevated, others near zero
Action: wait. KSM needs time to build its scan tree. Savings will appear after the first full scan completes.
- KSM: savings may increase (good) or decrease (pages being unshared)
- CPU: may show elevated utilization and throttling
- Memory: available memory low, overcommit ratio climbing
- zram: MODERATE or EFFECTIVE, swap I/O increasing
- Overhead: kswapd active, possibly kcompactd
Action: check which guests or tiers are driving the pressure. The CPU panel's per-domain table shows who is consuming the most. The Memory panel's per-tier table shows where memory is committed. Consider reducing worker count or size (Bronze is the elastic tier).
- KSM: MARGINAL or WASTE, CPU cost visible but savings modest
- CPU: ksmd visible in overhead panel
- Memory: overcommit ratio low, plenty of available memory
- zram: STANDBY
Action: if the host has plenty of free memory and KSM isn't finding much
to merge, consider increasing sleep_millisecs to reduce scan frequency. KSM
is most valuable when memory is tight; on an undercommitted host it is doing
work for minimal benefit.