Proposal for support for runtime composefs validation by alexlarsson · Pull Request #28658 · containers/podman

alexlarsson · 2026-05-06T13:40:10Z

In the automotive sphere, we're interested in having some level of runtime validation. For the rootfs we already get this from bootc using composefs. However, if we're also using containers in /var/lib/containers, those are not protected by this. However, containers/storage already (optionally) supports composefs, so we should be able to do something similar for podman.

Here is what this MR, and the related changes in container-libs does:

Add option --security-opt signature=[check,require]. If this is set, then the signature for the manifest is validated at podman run time, which allows us to trust the manifest data, like the annotations.
Add support in the overlay driver to pass in a set of expected composefs fs-verity digests for the layers. When these are used, it will validate at mount time that all the layers have an expected digest, and ensures the overlayfs mount is mounted with verity=require (which requires all files in the mount to be backed by a file with a fs-verity digest from the composefs blob).
If --security-opt verity=enforce is passed to podman run, then podman looks at the per-layer annotations in the image manifest (which we ideally trust due to a signature) for io.containers.composefs.digest keys, where you can give a list of allowable fs-verity digests. These are then forwarded to the overlayfs driver which validates this at mount time.

With the above, we can have a pretty robust validation of the container at runtime. There are some weak points:

An attacker could replace the podman binary or some dependency to not do the checks
An attacker could change the signature policy
You have to start podman with the right arguments

All of these are fixable in a controlled environment. For example, if you have a read-only /usr and /etc like bootc, and you ship a quadlet file in /usr that has the right arguments, then you can have some trust in that the right code is running, and you can do "podman pull" to get a new image version, keeping this trust.

I have an example signed image with annotations at https://quay.io/repository/alexl42/centos-verity. See the description there for the public key used. With I can run a validated image:

# podman run --security-opt signature=require --security-opt verity=enforce -ti quay.io/alexl42/centos-verity:10 echo foo

You can also see how it works with an unsigned and no-verity image:

# podman run --security-opt signature=require --security-opt verity=enforce -ti quay.io/centos/centos:9 echo foo
Error: manifest signature verification failed: No signature verification policy found for image docker:quay.io/centos/centos:9
# podman run --security-opt verity=enforce -ti quay.io/centos/centos:9 echo foo
Error: verity enforcement: layer 0 missing io.containers.composefs.digest annotation

Or if i tweak the composefs blob:

# podman run --security-opt signature=require --security-opt verity=enforce -ti quay.io/alexl42/centos-verity:10 echo foo
Error: mounting storage for container 45b43778da976b9997ace9eda98b17ac4c87c4e2f820c59a07f9c4f7429e884d: composefs blob /mnt/data/containers/storage/overlay/433ca53ed3a7208fc5834c1ee92839b01ed5ff7fccef4e80c0437992c4806267/composefs-data/composefs.blob has fs-verity digest "sha256:f86904b89e9d4b5c5436a657647c3d0309be742c5f9460b786027fb2968ea167", not in allowed list

I don't expect this PR to just be necessarily merged as is, but I'd like to bring this up for discussion. We'd like a feature like this in automotive, is that reasonable? Is the approach reasonable? Is the interface reasonable?

Discussion points:

Do we maybe also want config options for making these checks default?
Is using layer annotations right? They are sort of a pain to set.

Some minor notes:

We need to support multiple for forward compatibility if we support e.g. other fs-verity hashes or composefs versions.
This depends on composefs: Properly add verity info to generated dump file container-libs#816
In the work on OCI sealing in the next gen composefs-rs there is some similar annotations for this, although those are in a separate artifact. We should ensure that these don't conflict, and are properly versioned for future use of this in podman.

If this is set to `check`, then we validate the image manifest against the policy every time we run the container. If instead it is set to `require` then we do the same, but we also fail if there is no policy for the image. Also, we take care to validate the actual manifest data that we decode in image.Inspect(), so that we can trust the ImageData, like the annotations.

TODO: Docs

This is a small script that rewrites a manifest to add the fs-verity annotations, converts to zstd::chunked, signs it with cosign and pushes it to a repo.

pypingou · 2026-05-07T11:53:50Z

Do we maybe also want config options for making these checks default?

I think we would like this yes, especially as we know that not everyone will use quadlet files in /usr

mheon · 2026-05-08T17:54:32Z


 		options = append(options, libpod.WithRootFSFromImage(newImage.ID(), resolvedImageName, s.RawImageName))
+
+		if s.SignaturePolicy != "" {


I feel like there's a TOCTOU risk doing this in Specgen. The container isn't created yet, if the image tag we want to use is replaced before it is this check is subverted

To add to this anything in specgen is by design not doing runtime validation, specgen runs once when the container is created. For podman stop/start it will not be called again.

Yes that does not matter for the quadlet use case but still if such cli options exists it must work with all of podman not just quadlet.

ok nvm I fully read the code now, I think the TOCTOU does not matter because you pass in the verify digests and if the image was replaced in between then at mount time the digests will be invald and cause failure as they should.

Yes, that main requirement is that we validate the manifest json at some point, and then make any security relative decisions based on the manifest content (such as the fs-verity deltas) only from exactly the same data that we validated.

mheon · 2026-05-08T17:56:59Z

+
+const verityDigestAnnotation = "io.containers.composefs.digest"
+
+func extractVerityDigests(imageData *libimage.ImageData) ([][]string, error) {


A further reason not to do this in Specgen: You don't know what storage driver is in use by Libpod, so you could try to run this check on a system using a btrfs or zfs store. From my initial read of the code it'll probably work but you definitely aren't getting the benefits you expect if composefs isn't the backing store.

Luap99 · 2026-05-11T15:43:45Z

Can you expand more on your security/thread model please?

Right now the digest is looked up once then put into the container config (db) before it is passed to the storage mount, you say you use a read only fs but the db of course must be writeable so the attacker could try to write to the sqlite db to unset these container config fields.

For policy verification we also support loading that via env var that points to any file so that could also be used as attacker if they can set this for the podman process.

Regarding quadlets are only in ro /usr/etc I am not sure that alone matters. We also lookup /run/containers/systemd/ which I guess would be writeable or even systemd itself uses run/systemd/system which could be used to overwrite the service with another unit.

So I am not sure the controlled env will help you that much. If the attacker can get root with write access on the system there is nothing we could do.

alexlarsson · 2026-05-18T08:45:22Z

Its been a while since I looked at the podman codebase, so I marked this Draft for precisely the reason that I'd like some highlevel review on where this should go, and you're right that there is a potential issue with the config being stored in a db post-validation. Doing the validation later seems better from this PoV, but then it may be harder to do the actual validation.

Anyway, you correctly ask for a threat model, and I can at least give you what the automotive version of this is:

Suppose you have a sealed system, that verifies (secure-boot style) at boot and runtime:

The kernel
The kernel command line
The initrd
The composefs rootfs (against per-build key in initrd)
Uses a transient /etc (i.e. a per-boot tmpfs backed on /usr/etc from the composefs)
Ships podman in the read-only root, configured to use a transient store.
/var is writable in a persistent way

In such a setup we can trust everything except /var, at least from a clean boot. This is good, because the goal are:

Protect against accidental changes to the system (hw issue, damage, fs bugs, whatever)
Protect against a deliberate attacker gaining root access to persist this across boot. (Ideally we also protect against root attacks in the first place, but this is a separate defense-in-depth concern).
Disallow end users to run modified software in their cars.

We already support embedding containers in the system image (as separate image store directory), which gives use some of the above features for containers. However, we would like to also extend this to being able to install and update containers in /var, separate from the bootc image. This will allow faster and partial OTA updates with less risk. And, we would like to keep the above features and goals for apps using such containers.

So, lets assume we can "trust" /usr, /etc, $PATH, etc. What can we do to ensure that we can also install container images in /var and when we run them, we will run the code that was intended. And, this model should include hostile root-running code trying to persist root-rights across reboot.

As the most basic example, lets assume we have a quadlet in the trusted read-only bootc image. This means we have a trusted podman binary, a trusted podman config, and a trusted podman run commandline, but an untrusted /var/lib/containers. The storage config enables the overlayfs backend and the features to allow composefs images and makes the container root read-only by default (and this is not overridden in the run command). Can we, in this world, give the above guarantees? The hope is that we can run only trusted code and config up to the container being started, and have the container image content be validated using composefs digests that are trusted due to being referenced in a signed json manifest that map to a key that is part of the read-only trusted rootfs.

The main weakness I see is that an individual container may use a volume to persist data, and an attacker could modify it to attack the container, and then further use some container escape to get root access on the host. But, ignoring that vector, I think this is doable, although my initial draft may be naive in some aspects.

Luap99 · 2026-05-18T11:03:33Z

As the most basic example, lets assume we have a quadlet in the trusted read-only bootc image. This means we have a trusted podman binary, a trusted podman config, and a trusted podman run commandline, but an untrusted /var/lib/containers. The storage config enables the overlayfs backend and the features to allow composefs images and makes the container root read-only by default (and this is not overridden in the run command). Can we, in this world, give the above guarantees? The hope is that we can run only trusted code and config up to the container being started, and have the container image content be validated using composefs digests that are trusted due to being referenced in a signed json manifest that map to a key that is part of the read-only trusted rootfs.

Thanks, given that I think the current approach sounds reasonable if we move more of the validation into libpod (container start time) and then do not store the annotations as part of the container config, once validated with policy.json they need to passed along the call stack in memory to the storage mount code IMO. I have no real opinion on the image design questions, i.e. layer annotations for the hashes.

Of course once an attacker gains root on the running system they could turn of the policy.json verification and/or overmount /usr/bin/podman or the quadlet file.

alexlarsson added 4 commits May 6, 2026 14:40

WIP: Vendor in the containers-libs from composefs-verity branch

88628dd

Add support for --security-opt verity=enforce

022c992

TODO: Docs

Add contrib/add-verity-annotations helper script

01f4e91

This is a small script that rewrites a manifest to add the fs-verity annotations, converts to zstd::chunked, signs it with cosign and pushes it to a repo.

alexlarsson marked this pull request as draft May 6, 2026 13:40

mheon reviewed May 8, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proposal for support for runtime composefs validation#28658

Proposal for support for runtime composefs validation#28658
alexlarsson wants to merge 4 commits into
containers:mainfrom
alexlarsson:composefs-verity

alexlarsson commented May 6, 2026

Uh oh!

pypingou commented May 7, 2026

Uh oh!

mheon May 8, 2026

Uh oh!

Luap99 May 11, 2026

Uh oh!

Luap99 May 11, 2026

Uh oh!

alexlarsson May 18, 2026

Uh oh!

mheon May 8, 2026

Uh oh!

Luap99 commented May 11, 2026

Uh oh!

alexlarsson commented May 18, 2026

Uh oh!

Luap99 commented May 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants


		options = append(options, libpod.WithRootFSFromImage(newImage.ID(), resolvedImageName, s.RawImageName))

		if s.SignaturePolicy != "" {


		const verityDigestAnnotation = "io.containers.composefs.digest"

		func extractVerityDigests(imageData *libimage.ImageData) ([][]string, error) {

Conversation

alexlarsson commented May 6, 2026

Uh oh!

pypingou commented May 7, 2026

Uh oh!

mheon May 8, 2026

Choose a reason for hiding this comment

Uh oh!

Luap99 May 11, 2026

Choose a reason for hiding this comment

Uh oh!

Luap99 May 11, 2026

Choose a reason for hiding this comment

Uh oh!

alexlarsson May 18, 2026

Choose a reason for hiding this comment

Uh oh!

mheon May 8, 2026

Choose a reason for hiding this comment

Uh oh!

Luap99 commented May 11, 2026

Uh oh!

alexlarsson commented May 18, 2026

Uh oh!

Luap99 commented May 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants