Skip to content

flatpak: use container resolver to resolve flatpak refs#2334

Open
lzap wants to merge 3 commits into
osbuild:mainfrom
lzap:mem-ostree1
Open

flatpak: use container resolver to resolve flatpak refs#2334
lzap wants to merge 3 commits into
osbuild:mainfrom
lzap:mem-ostree1

Conversation

@lzap
Copy link
Copy Markdown
Contributor

@lzap lzap commented May 13, 2026

We need to optimize memory consumption and we ideally want to do this just once, for container refs. Flatpak did have its own resolver, let's reuse what we have.


Initially I wanted to solve the memory problem described here: osbuild/image-builder-cli#514

Only to figure out that fedora registry does not support server-side filtering, at least not the one we need here. I think we really need to optimize the parsing. I believe @achilleas-k has some ideas already.


Update: I did have one idea which was confirmed - aren't we querying the same image over and over again just with different inputs? And this was confirmed. A simple patch (2nd commit) that implements a trivial "cache" shaves the processing time significantly. It just remembers the last tag and parsed data so it can be immediately reused, no need for fancy LRU caching or something like that.

Before:

time go run ./cmd/image-builder -v manifest --distro fedora-44 silverblue-installer
real	2m44.135s
user	0m21.812s
sys	0m9.427s

After:

time go run ./cmd/image-builder -v manifest --distro fedora-44 silverblue-installer
real	0m49.459s
user	0m10.343s
sys	0m4.078s

Now, I really hope this is not just quay.io being slow for me, please test this too :-)


Reminder for myself: rebase after osbuild/osbuild-composer#5136 and osbuild/osbuild-composer#5138

@lzap lzap requested a review from a team as a code owner May 13, 2026 12:45
supakeen
supakeen previously approved these changes May 13, 2026
Copy link
Copy Markdown
Member

@supakeen supakeen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me to re-use the existing code for this instead of the one I duplicated; needs linter appeasement.

lzap added 3 commits May 13, 2026 17:58
We need to optimize memory consumption and we ideally want to do this
just once, for container refs. Flatpak did have its own resolver, let's
reuse what we have.
We want to avoid downloading the registry index for every flatpak ref.
Copy link
Copy Markdown
Member

@achilleas-k achilleas-k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good stuff, thank you.

LGTM

Comment on lines +63 to +65
echo "git status:"
git status -vv
echo "-----------------------------------------------------------------------------"
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!!

@lzap lzap requested a review from supakeen May 13, 2026 16:49
@lzap
Copy link
Copy Markdown
Contributor Author

lzap commented May 13, 2026

Also tested memory, is down from 413 MiB to 285 MiB. Before:

	Command being timed: "go run ./cmd/image-builder -v manifest --distro fedora-44 silverblue-installer"
	User time (seconds): 15.90
	System time (seconds): 6.92
	Percent of CPU this job got: 20%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:52.87
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 413000
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 2
	Minor (reclaiming a frame) page faults: 529761
	Voluntary context switches: 268150
	Involuntary context switches: 488
	Swaps: 0
	File system inputs: 0
	File system outputs: 104
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

After:

	Command being timed: "go run ./cmd/image-builder -v manifest --distro fedora-44 silverblue-installer"
	User time (seconds): 4.91
	System time (seconds): 2.09
	Percent of CPU this job got: 15%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:43.81
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 285636
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 189780
	Voluntary context switches: 31028
	Involuntary context switches: 294
	Swaps: 0
	File system inputs: 0
	File system outputs: 104
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants