Root filesystem had been changed to composefs and it's 100% full after 4.19.0-okd-scos.19 upgrade

[stanbog]

Describe the bug

After the upgrade of OKD from version 4.18.0-okd-scos.10 to version 4.19.0-okd-scos.19, the root filesystem got changed as follow:

From:

sh-5.1# df -h / 
Filesystem      Size  Used Avail Use% Mounted on
overlay         200G   36G  164G  18% /

To:

sh-5.1# df -h /
Filesystem      Size  Used Avail Use% Mounted on
composefs       5.8M  5.8M     0 100% /

Which is causing issues because all our monitoring tools are seeing the nodes as being 'full'.

We are also not sure of this is expected behavior or an actual issue.

Version

4.19.0-okd-scos.19

Reproducibility

100%

Log Bundle

No response

[sarthakpurohit]

Thanks for the report.

This appears to be an expected behavior change in 4.19 due to the switch to composefs. The composefs mount shown at / is a small, read-only metadata filesystem, which explains the fixed size (~5.8M) and 100% usage. The underlying host filesystem is not actually full.

If this df -h / output is being collected from a containerized context (monitoring pods, debug containers, etc.), the host root filesystem may no longer be exposed the same way as in 4.18. Monitoring configurations may need to be updated accordingly.

To confirm, can you clarify:

Where is this command being run from (container vs node)?
Do you see the same result when running df -h directly on the node?