Skip to content

Commit 74680f9

Browse files
feat: build operator images hermetically in upstream repo [RHIDP-15044] (#3359)
* feat: build operator images hermetically in upstream repo [RHIDP-15044] Consolidate upstream and downstream Dockerfiles into a single unified Dockerfile that works with both Hermeto (upstream) and Cachi2 (Konflux) hermetic builds. This catches build issues at PR time instead of waiting for downstream Konflux builds. - Remove dnf/microdnf update commands and downstream comment toggles from Dockerfile (RPMs managed via rpms.lock.yaml) - Delete .rhdh/docker/Dockerfile (consolidated into root Dockerfile) - Add .github/actions/docker-build composite action for hermetic builds using Hermeto (pre-fetch deps, transform Dockerfile, build with --network none) - Add scripts/local-hermeto-build.sh for local hermetic build testing - Update PR validation workflow to use hermetic builds - Update main CI and PR image build workflows to build operator image hermetically - Add multi-arch (x86_64 + aarch64) support to rpms.in.yaml and regenerate rpms.lock.yaml - Update all references from .rhdh/docker/Dockerfile to root Dockerfile - Add hermetic-build Makefile target and developer documentation Signed-off-by: fndlovu@redhat.com * feat: cache hermeto dependencies in CI to speed up PR builds Cache the hermeto dependency cache (Go modules + RPMs) in GitHub Actions keyed on go.sum + rpms.lock.yaml. When deps haven't changed, only generate-env and inject-files run (~30s) instead of the full fetch-deps (~3-5 min). The hermetic build with --network none still runs every time. Signed-off-by: fndlovu@redhat.com * fix: address QODO review findings for hermetic build - Ensure podman is available in composite action (self-sufficient) - Quote $GITHUB_ENV and $GITHUB_OUTPUT variable expansions - Use readarray for safe tag handling in podman save - Combine set -e and set -uo pipefail into single set -euo pipefail - Guard EXIT trap rm -rf with || true to prevent cleanup failures - Pass resolved component dir to build_cache() instead of using $PWD - Remove unnecessary -ti flags from podman run invocations Signed-off-by: fndlovu <fndlovu@redhat.com> Signed-off-by: Fortune-Ndlovu <fndlovu@redhat.com> * fix: pin hermeto image to 0.60.1 with renovate tracking Pin quay.io/konflux-ci/hermeto from :latest to :0.60.1 in both the CI composite action and local build script to avoid unexpected updates. Added renovate comment hint so version bumps are automated. Signed-off-by: fndlovu <fndlovu@redhat.com> Signed-off-by: Fortune-Ndlovu <fndlovu@redhat.com> * fix: add go.mod to hermeto cache key to avoid stale deps A go.mod-only change (new module path, replace directive, Go version bump) without a go.sum change could hit stale cache and skip fetch-deps, leaving stale output for a --network=none build. Signed-off-by: fndlovu <fndlovu@redhat.com> Signed-off-by: Fortune-Ndlovu <fndlovu@redhat.com> * chore: clarify Dockerfile supports both hermetic and non-hermetic builds * fix: restore original job name to match required status check Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Signed-off-by: fndlovu@redhat.com Signed-off-by: fndlovu <fndlovu@redhat.com> Signed-off-by: Fortune-Ndlovu <fndlovu@redhat.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 5d6bf7b commit 74680f9

16 files changed

Lines changed: 588 additions & 127 deletions

.dockerignore

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,3 +2,7 @@
22
# Ignore build and test binaries.
33
bin/
44
testbin/
5+
6+
# Hermetic build artifacts
7+
hermeto-cache/
8+
Dockerfile.hermeto
Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
# Copyright Red Hat, Inc.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
name: Docker Build (Hermetic)
16+
description: Build operator image hermetically using Hermeto (offline/reproducible build)
17+
inputs:
18+
imageName:
19+
description: The full image name including registry (e.g., quay.io/rhdh-community/operator)
20+
required: true
21+
imageTags:
22+
description: The tags to apply to the image
23+
required: true
24+
imageLabels:
25+
description: The labels for the Docker image
26+
required: false
27+
platform:
28+
description: "Target given CPU platform architecture (default: linux/amd64)"
29+
required: false
30+
default: linux/amd64
31+
containerfilePath:
32+
description: Path to the Dockerfile to use
33+
required: false
34+
default: 'Dockerfile'
35+
skipArtifactUpload:
36+
description: Skip uploading the built image as a GitHub artifact
37+
required: false
38+
default: 'false'
39+
40+
runs:
41+
using: composite
42+
steps:
43+
- name: Extract metadata (tags, labels) for Docker
44+
id: meta
45+
uses: docker/metadata-action@c299e40c65443455700f0fdfc63efafe5b349051 # v5.10.0
46+
with:
47+
images: ${{ inputs.imageName }}
48+
tags: |
49+
${{ inputs.imageTags }}
50+
labels: |
51+
${{ inputs.imageLabels }}
52+
53+
- name: Ensure podman is available
54+
shell: bash
55+
run: |
56+
if ! command -v podman &>/dev/null; then
57+
echo "podman not found, installing..."
58+
sudo apt-get -y update && sudo apt-get -y install podman
59+
fi
60+
podman --version
61+
62+
- name: Set up hermetic build variables
63+
shell: bash
64+
run: |
65+
# renovate: datasource=docker depName=quay.io/konflux-ci/hermeto
66+
echo "HERMETO_IMAGE=quay.io/konflux-ci/hermeto:0.60.1" >> "$GITHUB_ENV"
67+
echo "LOCAL_CACHE_DIR=./hermeto-cache/operator" >> "$GITHUB_ENV"
68+
69+
- name: Restore hermeto dependency cache
70+
id: cache-deps
71+
uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
72+
with:
73+
path: ./hermeto-cache/operator
74+
key: hermeto-deps-${{ inputs.platform }}-${{ hashFiles('go.mod', 'go.sum', 'rpms.lock.yaml') }}
75+
76+
- name: Fetch dependencies with hermeto
77+
if: steps.cache-deps.outputs.cache-hit != 'true'
78+
shell: bash
79+
run: |
80+
set -ex
81+
82+
mkdir -p "$LOCAL_CACHE_DIR"
83+
84+
echo "::group::Fetching dependencies with hermeto"
85+
podman run --rm -v "$PWD:/source:z" -v "$LOCAL_CACHE_DIR:/cachi2:z" -w /source "$HERMETO_IMAGE" \
86+
--log-level DEBUG \
87+
fetch-deps \
88+
--source . \
89+
--output /cachi2/output \
90+
'[{"type": "rpm", "path": "."}, {"type": "gomod", "path": "."}]'
91+
echo "::endgroup::"
92+
93+
if [ ! -d "$LOCAL_CACHE_DIR/output" ]; then
94+
echo "No output directory found after fetch-deps"
95+
exit 1
96+
fi
97+
98+
echo "::group::Generating environment file"
99+
podman run --rm -v "$PWD:/source:z" -v "$LOCAL_CACHE_DIR:/cachi2:z" -w /source "$HERMETO_IMAGE" \
100+
--log-level DEBUG \
101+
generate-env --format env \
102+
--output /cachi2/cachi2.env /cachi2/output
103+
echo "::endgroup::"
104+
105+
echo "::group::Injecting files"
106+
podman run --rm -v "$PWD:/source:z" -v "$LOCAL_CACHE_DIR:/cachi2:z" -w /source "$HERMETO_IMAGE" \
107+
--log-level DEBUG \
108+
inject-files /cachi2/output
109+
echo "::endgroup::"
110+
111+
- name: Generate env and inject files (cache hit)
112+
if: steps.cache-deps.outputs.cache-hit == 'true'
113+
shell: bash
114+
run: |
115+
set -ex
116+
echo "::group::Generating environment file from cached deps"
117+
podman run --rm -v "$PWD:/source:z" -v "$LOCAL_CACHE_DIR:/cachi2:z" -w /source "$HERMETO_IMAGE" \
118+
--log-level DEBUG \
119+
generate-env --format env \
120+
--output /cachi2/cachi2.env /cachi2/output
121+
echo "::endgroup::"
122+
123+
echo "::group::Injecting files from cached deps"
124+
podman run --rm -v "$PWD:/source:z" -v "$LOCAL_CACHE_DIR:/cachi2:z" -w /source "$HERMETO_IMAGE" \
125+
--log-level DEBUG \
126+
inject-files /cachi2/output
127+
echo "::endgroup::"
128+
129+
- name: Fix cache ownership for non-root buildah
130+
shell: bash
131+
run: |
132+
set -ex
133+
echo LOCAL_CACHE_DIR_REALPATH=$(realpath "$LOCAL_CACHE_DIR") >> "$GITHUB_ENV"
134+
sudo chown -R runner "$(realpath "$LOCAL_CACHE_DIR")"
135+
136+
- name: Transform Dockerfile for hermetic build
137+
shell: bash
138+
id: transform-containerfile
139+
env:
140+
CONTAINERFILE_PATH: ${{ inputs.containerfilePath }}
141+
TRANSFORMED_CONTAINERFILE: ${{ inputs.containerfilePath }}.hermeto
142+
run: |
143+
set -x
144+
145+
cp "$CONTAINERFILE_PATH" "$TRANSFORMED_CONTAINERFILE"
146+
147+
# Insert RPM repo replacement before every dnf/microdnf install
148+
sed -i '/RUN *\(dnf\|microdnf\) install/i RUN rm -r /etc/yum.repos.d/* && cp /cachi2/output/deps/rpm/$(uname -m)/repos.d/hermeto.repo /etc/yum.repos.d/' \
149+
"$TRANSFORMED_CONTAINERFILE"
150+
151+
# Prepend cachi2 env sourcing to every RUN command
152+
sed -i 's/^\s*RUN /RUN . \/cachi2\/cachi2.env \&\& /' "$TRANSFORMED_CONTAINERFILE"
153+
154+
echo "transformed_containerfile=$TRANSFORMED_CONTAINERFILE" >> "$GITHUB_OUTPUT"
155+
156+
- name: Build Docker Image
157+
id: build
158+
uses: redhat-actions/buildah-build@7a95fa7ee0f02d552a32753e7414641a04307056 # v2.13
159+
with:
160+
containerfiles: ${{ steps.transform-containerfile.outputs.transformed_containerfile }}
161+
context: .
162+
platform: ${{ inputs.platform }}
163+
tags: ${{ steps.meta.outputs.tags }}
164+
labels: ${{ steps.meta.outputs.labels }}
165+
extra-args: |
166+
--network=none
167+
--volume ${{ env.LOCAL_CACHE_DIR_REALPATH }}:/cachi2:z
168+
169+
- name: Save image as artifact
170+
if: ${{ inputs.skipArtifactUpload != 'true' }}
171+
shell: bash
172+
env:
173+
TAGS_LIST: ${{ steps.meta.outputs.tags }}
174+
run: |
175+
mkdir -p ./operator-podman-artifacts
176+
echo "Saving images with tags:"
177+
echo "$TAGS_LIST"
178+
readarray -t tags <<< "$TAGS_LIST"
179+
podman save "${tags[@]}" -o ./operator-podman-artifacts/image.tar
180+
echo "$TAGS_LIST" > ./operator-podman-artifacts/tags.txt
181+
182+
- name: Upload image artifact
183+
if: ${{ inputs.skipArtifactUpload != 'true' }}
184+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
185+
with:
186+
name: podman-image-${{ github.event.number || github.ref_name }}-${{ env.SHORT_SHA }}
187+
path: ./operator-podman-artifacts/
188+
retention-days: 1
189+
if-no-files-found: error

.github/workflows/next-container-build.yaml

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -191,7 +191,15 @@ jobs:
191191
username: ${{ vars.QUAY_USERNAME }}
192192
password: ${{ secrets.QUAY_TOKEN }}
193193

194-
- name: Build and push per-arch operator and catalog images
194+
- name: Build operator image (hermetic)
195+
uses: ./.github/actions/docker-build
196+
with:
197+
imageName: ${{ env.REGISTRY }}/${{ vars.REGISTRY_ORG }}/${{ vars.OPERATOR_IMAGE_NAME || 'operator' }}
198+
imageTags: type=raw,value=${{ env.BASE_VERSION }}
199+
platform: ${{ env.PLATFORM }}
200+
skipArtifactUpload: 'true'
201+
202+
- name: Build catalog image and push per-arch images
195203
run: |
196204
set -euo pipefail
197205
sudo apt-get -y update; sudo apt-get -y install podman
@@ -200,22 +208,15 @@ jobs:
200208
export VERSION="${BASE_VERSION}"
201209
export REGISTRY_WITH_ORG="${REGISTRY}/${REGISTRY_ORG}"
202210
export IMAGE_TAG_BASE="${REGISTRY_WITH_ORG}/${OPERATOR_IMAGE_NAME}"
203-
# Point catalog-build's bundle reference at the already-published, stable bundle
204-
# tag from the dedicated `bundle` job instead of rebuilding/pushing it again here.
205211
export BUNDLE_IMGS="${REGISTRY_WITH_ORG}/${OPERATOR_IMAGE_NAME}-bundle:${VERSION}"
206212
207213
: "${PLATFORM:?PLATFORM must be set}"
208214
: "${SHORT_SHA:?SHORT_SHA must be set}"
209215
: "${LATEST_NEXT:?LATEST_NEXT must be set}"
210216
: "${OPERATOR_IMAGE_NAME:?OPERATOR_IMAGE_NAME must be set}"
211217
212-
# Build the operator image, then the catalog image. catalog-build's only other
213-
# prerequisite (besides the bundle-push we're skipping) is `opm`, which downloads
214-
# the opm CLI if needed -- that still runs normally. `-o bundle-push` tells make to
215-
# treat that phony prerequisite as already satisfied, so `opm index add` runs
216-
# directly against BUNDLE_IMGS above (a safe concurrent *read* of an already-stable
217-
# tag) instead of each matrix leg re-pushing its own bundle to a shared tag first.
218-
CONTAINER_TOOL="${CONTAINER_TOOL}" VERSION="${VERSION}" PLATFORM="${PLATFORM}" BUNDLE_IMGS="${BUNDLE_IMGS}" make -o bundle-push image-build catalog-build
218+
# Build the catalog image only (operator was built hermetically above)
219+
CONTAINER_TOOL="${CONTAINER_TOOL}" VERSION="${VERSION}" PLATFORM="${PLATFORM}" BUNDLE_IMGS="${BUNDLE_IMGS}" make -o bundle-push catalog-build
219220
220221
# Push per-arch tagged images
221222
for image in "${OPERATOR_IMAGE_NAME}" "${OPERATOR_IMAGE_NAME}-catalog"; do

.github/workflows/pr-container-build.yaml

Lines changed: 13 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -88,15 +88,20 @@ jobs:
8888
username: ${{ vars.QUAY_USERNAME }}
8989
password: ${{ secrets.QUAY_TOKEN }}
9090

91-
- name: Build and push images
91+
- name: Build operator image (hermetic)
92+
uses: ./.github/actions/docker-build
93+
with:
94+
imageName: ${{ env.REGISTRY }}/${{ vars.REGISTRY_ORG }}/${{ vars.OPERATOR_IMAGE_NAME }}
95+
imageTags: type=raw,value=${{ env.BASE_VERSION }}-pr-${{ steps.pr.outputs.number }}-${{ env.SHORT_SHA }}
96+
platform: linux/amd64
97+
skipArtifactUpload: 'true'
98+
99+
- name: Build bundle and catalog images, then push all
92100
# We explicitly do NOT pass GH_TOKEN or RHDH_BOT_TOKEN here.
93-
# This makes running 'make' safe, even if the Makefile is malicious,
94-
# because there are no secrets in the env to steal.
95101
env:
96102
REGISTRY_ORG: ${{ vars.REGISTRY_ORG }}
97103
OPERATOR_IMAGE_NAME: ${{ vars.OPERATOR_IMAGE_NAME }}
98104
CONTAINER_TOOL: podman
99-
# Construct version safely
100105
VERSION: ${{ env.BASE_VERSION }}-pr-${{ steps.pr.outputs.number }}-${{ env.SHORT_SHA }}
101106
run: |
102107
sudo apt-get -y update; sudo apt-get -y install skopeo podman
@@ -106,11 +111,11 @@ jobs:
106111
107112
set -ex
108113
109-
# Run ONLY release-build (Lint removed)
110-
# We use 'make' here for convenience, but it is sandboxed from secrets.
111-
make release-build
114+
# Build bundle and catalog (operator was built hermetically above)
115+
make bundle bundle-build
116+
BUNDLE_IMGS="${REGISTRY_WITH_ORG}/${OPERATOR_IMAGE_NAME}-bundle:${VERSION}" make -o bundle-push catalog-build
112117
113-
# Push logic
118+
# Push all images
114119
for image in ${OPERATOR_IMAGE_NAME} ${OPERATOR_IMAGE_NAME}-bundle ${OPERATOR_IMAGE_NAME}-catalog; do
115120
podman push -q ${REGISTRY_WITH_ORG}/${image}:${VERSION} docker://${REGISTRY_WITH_ORG}/${image}:${VERSION}
116121
skopeo --insecure-policy copy --all docker://${REGISTRY_WITH_ORG}/${image}:${VERSION} docker://${REGISTRY_WITH_ORG}/${image}:${VERSION}

.github/workflows/pr-dockerbuild-validation.yaml

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ jobs:
1515
validate-dockerfile:
1616
name: Validate Dockerfile
1717
runs-on: ubuntu-latest
18-
timeout-minutes: 20
18+
timeout-minutes: 30
1919
permissions:
2020
contents: read
2121

@@ -31,6 +31,7 @@ jobs:
3131
with:
3232
files: |
3333
.github/workflows/pr-dockerbuild-validation.yaml
34+
.github/actions/docker-build/**
3435
Makefile
3536
**/*.go
3637
bundle/**
@@ -42,19 +43,20 @@ jobs:
4243
**/Containerfile
4344
**/*.Dockerfile
4445
**/.dockerignore
46+
rpms.in.yaml
47+
rpms.lock.yaml
48+
scripts/**
4549
files_ignore: |
4650
**/*.md
4751
**/*.adoc
4852
.rhdh/**
4953
tests/**
5054
51-
- name: Setup Go
55+
- name: Build operator image (hermetic)
5256
if: steps.changed-files.outputs.any_changed == 'true'
53-
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7
57+
uses: ./.github/actions/docker-build
5458
with:
55-
go-version-file: 'go.mod'
56-
57-
- name: Test build Dockerfile (no push)
58-
if: steps.changed-files.outputs.any_changed == 'true'
59-
run: |
60-
make image-build IMG=localhost/operator:validate
59+
imageName: localhost/operator
60+
imageTags: type=raw,value=validate
61+
platform: linux/amd64
62+
skipArtifactUpload: 'true'

.github/workflows/update-rpm-lockfile.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,14 +11,14 @@ on:
1111
- 'release-[0-9]+.[0-9]+'
1212
paths:
1313
- 'rpms.in.yaml'
14-
- '.rhdh/docker/Dockerfile'
14+
- 'Dockerfile'
1515
- '.github/workflows/update-rpm-lockfile.yaml'
1616

1717
permissions:
1818
contents: write
1919

2020
env:
21-
DOCKERFILE_PATH: .rhdh/docker/Dockerfile
21+
DOCKERFILE_PATH: Dockerfile
2222

2323
jobs:
2424
update-lockfile:

.github/workflows/validate-image-digests.yaml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,6 @@ on:
88
- 'hack/validate-image-digests.sh'
99
- 'Dockerfile'
1010
- '**/*bundle.Dockerfile'
11-
- '.rhdh/docker/Dockerfile'
1211

1312
jobs:
1413
validate-image-digests:

.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@ bin/*
88
bundle_tmp*
99
testbin/*
1010
Dockerfile.cross
11+
Dockerfile.hermeto
12+
hermeto-cache/
1113
__debug_bin*
1214
tmp/*
1315

0 commit comments

Comments
 (0)