rolling-upgrade-testing #38

Workflow file for this run

.github/workflows/rolling-upgrade-testing.yml at 83a9d9b

	##############################################################################
	# Rolling Upgrade Testing Workflow
	#
	# Purpose
	# -------
	# Validates partial rolling upgrades of KMS enclave nodes (thresholdWithEnclave)
	# on the aws-perf cluster (namespace kms-ci). The cluster must behave correctly
	# while some parties run the old image and others the new image.
	#
	# What mixed-version state exercises
	# ----------------------------------
	# - Per-party AWS KMS key policy (recipientAttestationImageSHA384)
	# - Dual trustedReleases PCR entries for TLS peer attestation
	# - Per-party image tag upgrades (via ci/scripts/rolling_upgrade.sh)
	#
	# How it runs (high level)
	# ------------------------
	# 1. Optional: if workflow input build=true, call docker-build.yml to build the
	# new image from the selected branch; its image_tag becomes the new tag.
	# 3. Deploy all 13 parties with OLD_IMAGE_TAG and old_kms_chart_version
	# (ci/scripts/deploy.sh --target aws-perf).
	# 4. Baseline Argo test-suite (thresholdWithEnclave-test-suite-workflow):
	# insecure key-gen + CRS + public/user-decrypt correctness smoke, all-old.
	# Capture the generated key-id for the mixed-state decrypt runs.
	# 5. First rolling upgrade: upgrade first_batch_parties to NEW_IMAGE_TAG
	# (default 1-5); core-client moves to new tag on all parties as required.
	# 6. Argo rolling-upgrade workflow: decrypt the baseline key-id on the mixed
	# cluster (public + user decrypt, correctness-only; label mixed-5-of-13).
	# 7. Second rolling upgrade: upgrade second_batch_parties (default 6-9);
	# cumulative upgraded set is first_batch + second_batch (9/13).
	# 8. Argo rolling-upgrade workflow again (label mixed-9-of-13).
	# 9. Upload Argo and kms-core logs; helm/kubectl cleanup; stop runner.
	#
	# Correctness signal: each decrypt encrypts a known value client-side (-e) and
	# fails on mismatch; runs use -a (expect a response from ALL 13 cores) so the
	# upgraded nodes can't be masked by an old-node quorum. A step fails the job if
	# any test pod does not reach phase Succeeded, excluding warmup tasks and the
	# prss-threshold profile's `reqid-above` probes (whose failure is expected at
	# some mixed states — read their per-request outcome from the summary instead).
	#
	# Triggers
	# --------
	# workflow_dispatch only (manual run in GitHub Actions).
	#
	# Workflow inputs (workflow_dispatch)
	# -------------------------------------
	# old_image_tag (string, required)
	# Baseline KMS Core image tag for initial deploy.
	# new_image_tag (string, required)
	# Target KMS Core image tag after upgrades (ignored if
	# build=true; then docker-build output is used).
	# core_client_image_tag (string, optional, default = new_image_tag)
	# Core-client (test harness) image tag, decoupled from
	# the core versions under test. Same across all stages.
	# build (boolean, default false)
	# If true, build new images via docker-build.yml and use
	# that tag as NEW; kms_branch applies to the build.
	# kms_branch (string, optional)
	# Branch to build from when build=true; also used with
	# new_kms_chart_version=repository for chart checkout.
	# fhe_params (choice Default\|Test, default Test)
	# FHE parameters for preprocessing and keygen in Argo.
	# old_kms_chart_version (string, default 1.5.1)
	# KMS Helm chart version for the all-old deploy (step 3).
	# new_kms_chart_version (string, default repository)
	# KMS Helm chart for upgraded parties; use "repository"
	# to use charts from the checked-out branch.
	# tkms_infra_chart_version (string, default 0.3.2)
	# TKMS Infra Helm chart version.
	# first_batch_parties (string, default 1,2,3,4,5)
	# Comma-separated party IDs for the first upgrade wave.
	# second_batch_parties (string, default 6,7,8,9)
	# Comma-separated party IDs for the second upgrade wave.
	#
	# Jobs
	# ----
	# docker-build Runs only when build=true; reusable docker-build.yml.
	# start-runner EC2 runner for the long-running test job.
	# rolling-upgrade-testing Main steps above; timeout 2400 minutes.
	# stop-runner Stops the EC2 runner (always).
	##############################################################################
	name: rolling-upgrade-testing

	on:
	workflow_dispatch:
	inputs:
	old_image_tag:
	description: 'Old (baseline) KMS Core image tag'
	required: true
	type: string
	new_image_tag:
	description: 'New KMS Core image tag to roll out'
	required: true
	type: string
	core_client_image_tag:
	description: 'Core-client (test harness) image tag; independent of the core versions under test. Defaults to new_image_tag if empty.'
	required: false
	type: string
	client_logs:
	description: 'Enable core-client tracing logs (default off, matching perf runs; turn on to capture decrypt/reconstruction errors)'
	required: true
	default: false
	type: boolean
	build:
	description: 'Build new Docker image from branch (overrides new_image_tag)'
	required: true
	default: false
	type: boolean
	kms_branch:
	description: 'KMS branch to build from (only used when build=true)'
	required: false
	type: string
	fhe_params:
	description: 'FHE parameters for preprocessing and keygen'
	required: true
	default: 'Test'
	type: choice
	options:
	- 'Default'
	- 'Test'
	old_kms_chart_version:
	description: 'KMS Helm chart version for old (baseline) deployment'
	required: true
	type: string
	default: '1.5.1'
	new_kms_chart_version:
	description: 'KMS Helm chart version for new (upgraded) deployment'
	required: true
	type: string
	default: 'repository'
	tkms_infra_chart_version:
	description: 'TKMS Infra chart version'
	required: true
	default: '0.3.2'
	type: string
	first_batch_parties:
	description: 'Comma-separated party IDs for 1st upgrade batch (e.g. 1,2,3,4,5)'
	required: true
	default: '1,2,3,4,5'
	type: string
	second_batch_parties:
	description: 'Comma-separated party IDs for 2nd upgrade batch (e.g. 6,7,8,9)'
	required: true
	default: '6,7,8,9'
	type: string
	test_profile:
	description: 'decrypt = the standard pre-upgrade check for any n->n+1 pair (plain pdec/udec correctness on the mixed cluster). prss-threshold = the exceptional legacy-PRSS-mask request-ID boundary probe (needs a threshold-aware new image + core-client).'
	required: true
	default: 'decrypt'
	type: choice
	options:
	- decrypt
	- prss-threshold

	permissions: {}

	jobs:
	############################################################################
	# Docker Build Job (Conditional)
	# Only runs when build=true to create the "new" image
	############################################################################
	docker-build:
	if: inputs.build == true
	name: rolling-upgrade/docker-build
	permissions:
	actions: read # Required to read workflow run information
	contents: write # Required to checkout repository code
	id-token: write # Required for OIDC authentication
	pull-requests: read # Required to read pull requests information
	packages: write # Required to publish Docker images
	attestations: write # Required to create build attestations
	uses: ./.github/workflows/docker-build.yml
	secrets:
	BLOCKCHAIN_ACTIONS_TOKEN: ${{ secrets.BLOCKCHAIN_ACTIONS_TOKEN }}
	AWS_ACCESS_KEY_S3_USER: ${{ secrets.AWS_ACCESS_KEY_S3_USER }}
	AWS_SECRET_KEY_S3_USER: ${{ secrets.AWS_SECRET_KEY_S3_USER }}
	CGR_USERNAME: ${{ secrets.CGR_USERNAME }}
	CGR_PASSWORD: ${{ secrets.CGR_PASSWORD }}

	############################################################################
	# Rolling Upgrade Testing Job
	############################################################################
	rolling-upgrade-testing:
	needs: [docker-build]
	if: always() && !cancelled()
	name: rolling-upgrade-testing
	runs-on: "runs-on=${{ github.run_id }}/runner=8cpu-linux-x64/spot=false/extras=s3-cache"
	timeout-minutes: 2400
	env:
	DEPLOYMENT_TYPE: 'thresholdWithEnclave'
	TLS: 'true'
	NAMESPACE: 'kms-ci'
	PATH_SUFFIX: 'kms-enclave-ci'
	steps:
	- name: Setup Runs-on
	uses: runs-on/action@cd2b598b0515d39d78c38a02d529db87d2196d1e # v2.0.3

	- name: Checkout Project KMS
	uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
	with:
	token: ${{ secrets.BLOCKCHAIN_ACTIONS_TOKEN }}
	persist-credentials: false
	# ====================================================================
	# DETERMINE IMAGE TAGS
	# ====================================================================
	- name: Determine image tags
	env:
	DOCKER_BUILD_TAG: ${{ needs.docker-build.outputs.image_tag }}
	INPUT_OLD_TAG: ${{ inputs.old_image_tag }}
	INPUT_NEW_TAG: ${{ inputs.new_image_tag }}
	INPUT_CLIENT_TAG: ${{ inputs.core_client_image_tag }}
	run: \|
	OLD_TAG="${INPUT_OLD_TAG}"

	if [[ -n "${DOCKER_BUILD_TAG}" ]]; then
	echo "Using freshly built image as new tag"
	NEW_TAG="${DOCKER_BUILD_TAG}"
	else
	NEW_TAG="${INPUT_NEW_TAG}"
	fi

	# The core-client is the test harness, deliberately decoupled from the
	# old/new core images under test. Falls back to the new tag if unset.
	CLIENT_TAG="${INPUT_CLIENT_TAG:-${NEW_TAG}}"

	if [[ -z "${OLD_TAG}" ]]; then
	echo "ERROR: old_image_tag is required"
	exit 1
	fi
	if [[ -z "${NEW_TAG}" ]]; then
	echo "ERROR: No new image tag available. Either build or provide new_image_tag"
	exit 1
	fi

	echo "Old image tag: ${OLD_TAG}"
	echo "New image tag: ${NEW_TAG}"
	echo "Core-client (harness): ${CLIENT_TAG}"

	{
	echo "OLD_IMAGE_TAG=${OLD_TAG}"
	echo "NEW_IMAGE_TAG=${NEW_TAG}"
	echo "KMS_CORE_IMAGE_TAG=${OLD_TAG}"
	# Test-harness image, independent of the core versions under test
	# (see core_client_image_tag input); consistent across all stages.
	echo "KMS_CORE_CLIENT_IMAGE_TAG=${CLIENT_TAG}"
	} >> "$GITHUB_ENV"

	- name: Configure deployment parameters
	env:
	FHE_PARAMS: ${{ inputs.fhe_params \|\| 'Test' }}
	KMS_BRANCH: ${{ inputs.kms_branch \|\| github.ref }}
	OLD_KMS_CHART_VERSION: ${{ inputs.old_kms_chart_version }}
	NEW_KMS_CHART_VERSION: ${{ inputs.new_kms_chart_version }}
	TKMS_INFRA_CHART_VERSION: ${{ inputs.tkms_infra_chart_version }}
	FIRST_BATCH: ${{ inputs.first_batch_parties }}
	SECOND_BATCH: ${{ inputs.second_batch_parties }}
	SYNC_SECRETS_CHART_VERSION: '0.2.1'
	CLIENT_LOGS: ${{ inputs.client_logs }}
	TEST_PROFILE: ${{ inputs.test_profile }}
	run: \|
	# Map the boolean client_logs input to the argo enabled/disabled value
	# (same convention as performance-testing.yml).
	if [[ "${CLIENT_LOGS}" == "true" ]]; then
	CLIENT_LOGS_STATUS="enabled"
	else
	CLIENT_LOGS_STATUS="disabled"
	fi
	# Resolve the test profile into the mixed-state Argo workflow file and
	# whether the legacy-PRSS-mask threshold rollout runs in phase 2.
	# decrypt -> standard pdec/udec correctness, no reqid, no threshold
	# prss-threshold -> request-ID boundary probe + legacy-mask rollout
	case "${TEST_PROFILE}" in
	prss-threshold)
	MIXED_WORKFLOW_FILE="${DEPLOYMENT_TYPE}-rolling-upgrade-workflow-${PATH_SUFFIX}.yaml"
	ENABLE_PRSS_THRESHOLD="true"
	;;
	*)
	MIXED_WORKFLOW_FILE="decrypt-test-during-upgrades.yaml"
	ENABLE_PRSS_THRESHOLD="false"
	;;
	esac
	{
	echo "FHE_PARAMS=${FHE_PARAMS}"
	echo "KMS_BRANCH=${KMS_BRANCH}"
	echo "OLD_KMS_CHART_VERSION=${OLD_KMS_CHART_VERSION}"
	echo "NEW_KMS_CHART_VERSION=${NEW_KMS_CHART_VERSION}"
	echo "TKMS_INFRA_CHART_VERSION=${TKMS_INFRA_CHART_VERSION}"
	echo "SYNC_SECRETS_CHART_VERSION=${SYNC_SECRETS_CHART_VERSION}"
	echo "FIRST_BATCH=${FIRST_BATCH}"
	echo "SECOND_BATCH=${SECOND_BATCH}"
	echo "CLIENT_LOGS_STATUS=${CLIENT_LOGS_STATUS}"
	echo "TEST_PROFILE=${TEST_PROFILE}"
	echo "MIXED_WORKFLOW_FILE=${MIXED_WORKFLOW_FILE}"
	echo "ENABLE_PRSS_THRESHOLD=${ENABLE_PRSS_THRESHOLD}"
	} >> "$GITHUB_ENV"

	# ====================================================================
	# TOOLING SETUP
	# ====================================================================
	- name: Login to hub.zama.org Container Registry
	uses: ./.github/actions/docker-login
	with:
	registry: ${{ secrets.HARBOR_URL }}
	username: ${{ secrets.HARBOR_READ_LOGIN }}
	password: ${{ secrets.HARBOR_READ_TOKEN }}

	- name: Setup tailscale
	uses: tailscale/github-action@84a3f23bb4d843bcf4da6cf824ec1be473daf4de # v3.2.3
	with:
	oauth-client-id: ${{ secrets.TS_OAUTH_CLIENT_ID }}
	oauth-secret: ${{ secrets.TS_OAUTH_SECRET }}
	tags: tag:${NAMESPACE}

	- name: Setup helm
	run: \|
	curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 \| bash
	export PATH=$PATH:/usr/local/bin

	- name: Setup argo workflow CLI
	env:
	ARGO_OS: '${{ runner.os }}'
	ARGO_VERSION: 'v3.7.2'
	run: \|
	if [[ "$(uname -s)" != "Darwin" ]]; then
	ARGO_OS="linux"
	fi
	curl -sLO "https://github.com/argoproj/argo-workflows/releases/download/${ARGO_VERSION}/argo-${ARGO_OS}-amd64.gz"
	gunzip "argo-${ARGO_OS}-amd64.gz"
	chmod +x "argo-${ARGO_OS}-amd64"
	mv "./argo-${ARGO_OS}-amd64" /usr/local/bin/argo
	argo version

	- name: Setup kubectl
	uses: azure/setup-kubectl@776406bce94f63e41d621b960d78ee25c8b76ede

	# ====================================================================
	# KUBERNETES SETUP
	# ====================================================================
	- name: Setup kubeconfig
	run: \|
	echo "Configuring kubeconfig for Tailscale cluster..."
	tailscale configure kubeconfig tailscale-operator-zws-dev.diplodocus-boa.ts.net

	- name: Checkout Project
	if: ${{ env.NEW_KMS_CHART_VERSION != 'repository' }}
	uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
	with:
	token: ${{ secrets.BLOCKCHAIN_ACTIONS_TOKEN }}
	persist-credentials: false

	- name: Checkout Project (from branch)
	if: ${{ env.NEW_KMS_CHART_VERSION == 'repository' }}
	uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
	with:
	ref: "${{ env.KMS_BRANCH }}"
	token: ${{ secrets.BLOCKCHAIN_ACTIONS_TOKEN }}
	persist-credentials: false

	- name: Set kubeconfig context
	run: \|
	kubectl config get-contexts
	kubectl config use-context tailscale-operator-zws-dev.diplodocus-boa.ts.net

	- name: Install jq
	env:
	JQ_VERSION: 1.8.1
	run: \|
	wget https://github.com/jqlang/jq/releases/download/jq-"${JQ_VERSION}"/jq-linux-amd64 -P /tmp
	sudo mv /tmp/jq-linux-amd64 /usr/local/bin/jq
	sudo chmod +x /usr/local/bin/jq
	echo "JQ version: $(jq --version)"

	# ====================================================================
	# STEP 1: Deploy all 13 nodes with OLD version
	# ====================================================================
	- name: 'Step 1: Deploy all 13 nodes with old version'
	run: \|
	echo "Deploying all 13 nodes with old tag: ${OLD_IMAGE_TAG}"
	chmod +x ci/scripts/deploy.sh
	./ci/scripts/deploy.sh \
	--target aws-perf \
	--namespace "${NAMESPACE}" \
	--deployment-type "${DEPLOYMENT_TYPE}" \
	--core-tag "${OLD_IMAGE_TAG}" \
	--client-tag "${OLD_IMAGE_TAG}" \
	--num-parties 13 \
	--kms-chart-version "${OLD_KMS_CHART_VERSION}" \
	--tkms-infra-version "${TKMS_INFRA_CHART_VERSION}"

	# ====================================================================
	# STEP 2: Run baseline performance tests
	# ====================================================================
	- name: 'Step 2: Run baseline performance tests (all old version)'
	env:
	SERVER_URL: ${{ github.server_url }}
	REPOSITORY: ${{ github.repository }}
	RUN_ID: ${{ github.run_id }}
	run: \|
	echo "Running baseline performance tests with all nodes on old version..."

	if [[ "${TLS}" == "true" ]]; then
	TLS_STATUS="enabled"
	else
	TLS_STATUS="disabled"
	fi

	# Baseline runs the correctness test-suite (insecure key-gen + CRS +
	# public/user-decrypt smoke) while all 13 nodes are on OLD_IMAGE_TAG.
	# The generated key-id is reused by the mixed-state decrypt runs below.
	cp ./ci/perf-testing/argo-workflow/"${DEPLOYMENT_TYPE}"-test-suite-workflow-"${PATH_SUFFIX}".yaml \
	/tmp/baseline-workflow.yaml
	# <client-version> = the test harness image (modern/NEW); <version> is the
	# displayed server version, which is OLD while the whole cluster is old.
	sed -i "s/<client-version>/${KMS_CORE_CLIENT_IMAGE_TAG}/g" /tmp/baseline-workflow.yaml
	sed -i "s/<version>/${OLD_IMAGE_TAG}/g" /tmp/baseline-workflow.yaml

	RUN_URL="${SERVER_URL}/${REPOSITORY}/actions/runs/${RUN_ID}"

	argo submit -n argo /tmp/baseline-workflow.yaml \
	-p tls="${TLS_STATUS}" \
	-p client-logs="${CLIENT_LOGS_STATUS}" \
	-p fhe-params="${FHE_PARAMS}" \
	-p run_url="${RUN_URL}"

	echo "Streaming baseline workflow logs..."
	argo logs @latest -f -n "${NAMESPACE}" \| tee argo-baseline-logs.txt
	echo "Baseline tests completed."

	# Capture the workflow JSON once for the correctness gate + key-id.
	BASELINE_JSON=$(argo get @latest -n "${NAMESPACE}" -o json)

	# Correctness gate: fail the job if any non-warmup test pod did not
	# succeed (a decrypt mismatch makes the core-client exit non-zero, and
	# continueOn keeps the workflow phase "Succeeded", so we inspect nodes).
	FAILED_NODES=$(echo "${BASELINE_JSON}" \| jq -r '
	.status.nodes[]?
	\| select(.type == "Pod" and .phase != "Succeeded"
	and (.displayName \| test("warmup\|reqid-above") \| not))
	\| .displayName')
	if [[ -n "${FAILED_NODES}" ]]; then
	echo "ERROR: baseline test pods did not succeed:"
	echo "${FAILED_NODES}"
	exit 1
	fi

	echo "Extracting key-id from baseline workflow..."
	BASELINE_KEY_ID=$(echo "${BASELINE_JSON}" \
	\| jq -r '.status.nodes \| to_entries[]
	\| select(.value.displayName == "insecure-key-gen")
	\| .value.outputs.parameters[]
	\| select(.name == "request-id") \| .value')

	if [[ -z "${BASELINE_KEY_ID}" \|\| "${BASELINE_KEY_ID}" == "null" ]]; then
	echo "ERROR: Failed to extract key-id from baseline workflow"
	exit 1
	fi
	echo "Extracted BASELINE_KEY_ID=${BASELINE_KEY_ID}"
	echo "BASELINE_KEY_ID=${BASELINE_KEY_ID}" >> "$GITHUB_ENV"

	# ====================================================================
	# STEP 3: Rolling upgrade 5/13 (first batch)
	# ====================================================================
	- name: 'Step 3: Rolling upgrade - first batch'
	run: \|
	echo "Upgrading first batch [${FIRST_BATCH}] to new tag: ${NEW_IMAGE_TAG}"
	echo "Core-client will be upgraded to ${NEW_IMAGE_TAG} on all parties"
	chmod +x ci/scripts/rolling_upgrade.sh
	./ci/scripts/rolling_upgrade.sh \
	--old-tag "${OLD_IMAGE_TAG}" \
	--new-tag "${NEW_IMAGE_TAG}" \
	--parties-to-upgrade "${FIRST_BATCH}" \
	--all-upgraded-parties "${FIRST_BATCH}" \
	--namespace "${NAMESPACE}" \
	--num-parties 13 \
	--deployment-type "${DEPLOYMENT_TYPE}" \
	--old-kms-chart-version "${OLD_KMS_CHART_VERSION}" \
	--new-kms-chart-version "${NEW_KMS_CHART_VERSION}" \
	--tkms-infra-version "${TKMS_INFRA_CHART_VERSION}"

	# ====================================================================
	# STEP 4: Run performance tests with 5/13 upgraded
	# ====================================================================
	- name: 'Step 4: Run performance tests (5/13 mixed state)'
	env:
	SERVER_URL: ${{ github.server_url }}
	REPOSITORY: ${{ github.repository }}
	RUN_ID: ${{ github.run_id }}
	run: \|
	echo "Running decrypt correctness tests (public + user) with 5/13 nodes upgraded..."
	echo "Reusing pre-upgrade BASELINE_KEY_ID=${BASELINE_KEY_ID}"

	if [[ "${TLS}" == "true" ]]; then
	TLS_STATUS="enabled"
	else
	TLS_STATUS="disabled"
	fi

	# Mixed-state run: decrypt the pre-upgrade BASELINE_KEY_ID on the mixed
	# cluster. core-client runs the NEW tag; core images stay mixed.
	# The Argo workflow file depends on the profile; MIXED_WORKFLOW_FILE is
	# set in "Configure deployment parameters".
	cp ./ci/perf-testing/argo-workflow/"${MIXED_WORKFLOW_FILE}" \
	/tmp/mixed-5-workflow.yaml
	sed -i "s/<client-version>/${KMS_CORE_CLIENT_IMAGE_TAG}/g" /tmp/mixed-5-workflow.yaml
	sed -i "s/<version>/${NEW_IMAGE_TAG}/g" /tmp/mixed-5-workflow.yaml

	# Count upgraded nodes from FIRST_BATCH
	NUM_UPGRADED=$(echo "${FIRST_BATCH}" \| tr ',' '\n' \| wc -l \| tr -d ' ')
	RUN_URL="${SERVER_URL}/${REPOSITORY}/actions/runs/${RUN_ID}"

	# The prss-threshold profile pins unique request-IDs straddling the
	# legacy-mask boundary; the decrypt profile uses random IDs. IDs must be
	# globally unique across pub/user AND across the 5/13 and 9/13 runs.
	REQID_ARGS=()
	if [[ "${ENABLE_PRSS_THRESHOLD}" == "true" ]]; then
	REQID_ARGS=(
	-p pdec-below-id=000000000000000000000000000000000000000000000000000000000000005a
	-p pdec-above-id=000000000000000000000000000000000000000000000000000000000000006e
	-p udec-below-id=000000000000000000000000000000000000000000000000000000000000005b
	-p udec-above-id=000000000000000000000000000000000000000000000000000000000000006f
	)
	fi

	argo submit -n argo /tmp/mixed-5-workflow.yaml \
	-p tls="${TLS_STATUS}" \
	-p client-logs="${CLIENT_LOGS_STATUS}" \
	-p fhe-params="${FHE_PARAMS}" \
	-p run_url="${RUN_URL}" \
	-p key-id="${BASELINE_KEY_ID}" \
	-p test_label="mixed-${NUM_UPGRADED}-of-13" \
	-p old_version="${OLD_IMAGE_TAG}" \
	-p new_version="${NEW_IMAGE_TAG}" \
	-p num_upgraded="${NUM_UPGRADED}" \
	-p num_total="13" \
	"${REQID_ARGS[@]}"

	echo "Streaming mixed-5/13 workflow logs..."
	argo logs @latest -f -n "${NAMESPACE}" \| tee argo-mixed-5-logs.txt
	echo "Mixed 5/13 tests completed."

	# Correctness gate: fail if any non-warmup decrypt pod did not succeed.
	FAILED_NODES=$(argo get @latest -n "${NAMESPACE}" -o json \| jq -r '
	.status.nodes[]?
	\| select(.type == "Pod" and .phase != "Succeeded"
	and (.displayName \| test("warmup\|reqid-above") \| not))
	\| .displayName')
	if [[ -n "${FAILED_NODES}" ]]; then
	echo "ERROR: mixed-5/13 test pods did not succeed:"
	echo "${FAILED_NODES}"
	exit 1
	fi

	# ====================================================================
	# STEP 5: Rolling upgrade 9/13 (second batch)
	# ====================================================================
	- name: 'Step 5: Rolling upgrade - second batch'
	run: \|
	ALL_UPGRADED="${FIRST_BATCH},${SECOND_BATCH}"
	echo "Upgrading second batch [${SECOND_BATCH}] to new tag: ${NEW_IMAGE_TAG}"
	echo "Cumulative upgraded parties: ${ALL_UPGRADED}"

	./ci/scripts/rolling_upgrade.sh \
	--old-tag "${OLD_IMAGE_TAG}" \
	--new-tag "${NEW_IMAGE_TAG}" \
	--parties-to-upgrade "${SECOND_BATCH}" \
	--all-upgraded-parties "${ALL_UPGRADED}" \
	--namespace "${NAMESPACE}" \
	--num-parties 13 \
	--deployment-type "${DEPLOYMENT_TYPE}" \
	--old-kms-chart-version "${OLD_KMS_CHART_VERSION}" \
	--new-kms-chart-version "${NEW_KMS_CHART_VERSION}" \
	--tkms-infra-version "${TKMS_INFRA_CHART_VERSION}"

	# ====================================================================
	# STEP 6: Run performance tests with 9/13 upgraded
	# ====================================================================
	- name: 'Step 6: Run performance tests (9/13 mixed state)'
	env:
	SERVER_URL: ${{ github.server_url }}
	REPOSITORY: ${{ github.repository }}
	RUN_ID: ${{ github.run_id }}
	run: \|
	echo "Running decrypt correctness tests (public + user) with 9/13 nodes upgraded..."
	echo "Reusing pre-upgrade BASELINE_KEY_ID=${BASELINE_KEY_ID}"

	if [[ "${TLS}" == "true" ]]; then
	TLS_STATUS="enabled"
	else
	TLS_STATUS="disabled"
	fi

	# Mixed-state run: decrypt the pre-upgrade BASELINE_KEY_ID on the mixed
	# cluster. core-client runs the NEW tag; core images stay mixed.
	# The Argo workflow file depends on the profile; MIXED_WORKFLOW_FILE is
	# set in "Configure deployment parameters".
	cp ./ci/perf-testing/argo-workflow/"${MIXED_WORKFLOW_FILE}" \
	/tmp/mixed-9-workflow.yaml
	sed -i "s/<client-version>/${KMS_CORE_CLIENT_IMAGE_TAG}/g" /tmp/mixed-9-workflow.yaml
	sed -i "s/<version>/${NEW_IMAGE_TAG}/g" /tmp/mixed-9-workflow.yaml

	# Count all upgraded nodes (first + second batch)
	ALL_UPGRADED="${FIRST_BATCH},${SECOND_BATCH}"
	NUM_UPGRADED=$(echo "${ALL_UPGRADED}" \| tr ',' '\n' \| wc -l \| tr -d ' ')
	RUN_URL="${SERVER_URL}/${REPOSITORY}/actions/runs/${RUN_ID}"

	# prss-threshold pins unique request-IDs (distinct from the 5/13 run's
	# ids to avoid "already exists" collisions); decrypt uses random IDs.
	REQID_ARGS=()
	if [[ "${ENABLE_PRSS_THRESHOLD}" == "true" ]]; then
	REQID_ARGS=(
	-p pdec-below-id=000000000000000000000000000000000000000000000000000000000000005c
	-p pdec-above-id=0000000000000000000000000000000000000000000000000000000000000070
	-p udec-below-id=000000000000000000000000000000000000000000000000000000000000005d
	-p udec-above-id=0000000000000000000000000000000000000000000000000000000000000071
	)
	fi

	argo submit -n argo /tmp/mixed-9-workflow.yaml \
	-p tls="${TLS_STATUS}" \
	-p client-logs="${CLIENT_LOGS_STATUS}" \
	-p fhe-params="${FHE_PARAMS}" \
	-p run_url="${RUN_URL}" \
	-p key-id="${BASELINE_KEY_ID}" \
	-p test_label="mixed-${NUM_UPGRADED}-of-13" \
	-p old_version="${OLD_IMAGE_TAG}" \
	-p new_version="${NEW_IMAGE_TAG}" \
	-p num_upgraded="${NUM_UPGRADED}" \
	-p num_total="13" \
	"${REQID_ARGS[@]}"

	echo "Streaming mixed-9/13 workflow logs..."
	argo logs @latest -f -n "${NAMESPACE}" \| tee argo-mixed-9-logs.txt
	echo "Mixed 9/13 tests completed."

	# Correctness gate: fail if any non-warmup decrypt pod did not succeed.
	FAILED_NODES=$(argo get @latest -n "${NAMESPACE}" -o json \| jq -r '
	.status.nodes[]?
	\| select(.type == "Pod" and .phase != "Succeeded"
	and (.displayName \| test("warmup\|reqid-above") \| not))
	\| .displayName')
	if [[ -n "${FAILED_NODES}" ]]; then
	echo "ERROR: mixed-9/13 test pods did not succeed:"
	echo "${FAILED_NODES}"
	exit 1
	fi

	# ====================================================================
	# UPLOAD TEST LOGS
	# ====================================================================
	- name: Upload Argo workflow logs
	if: always()
	uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
	with:
	name: argo-rolling-upgrade-logs
	path: argo-*-logs.txt
	retention-days: 30

	# ====================================================================
	# LOG COLLECTION
	# ====================================================================
	- name: Collect logs from kms-core pods
	if: always()
	run: \|
	echo "Collecting logs from all KMS Core pods..."
	for i in {1..13}; do
	POD_NAME="kms-core-${i}-core-${i}"
	LOG_FILE="kms-core-${i}-logs.txt"

	if kubectl get pod "${POD_NAME}" -n "${NAMESPACE}" &> /dev/null; then
	# The enclave's stdout is tailed by the logger sidecar; fall back to
	# the enclave container itself if the sidecar isn't present.
	kubectl logs "${POD_NAME}" -c kms-core-enclave-logger -n "${NAMESPACE}" > "${LOG_FILE}" 2>/dev/null \|\| \
	kubectl logs "${POD_NAME}" -c kms-core-enclave -n "${NAMESPACE}" > "${LOG_FILE}" 2>/dev/null \|\| \
	echo "Failed to get logs" > "${LOG_FILE}"
	echo "Logs saved for pod ${POD_NAME}"
	else
	echo "Pod ${POD_NAME} not found" > "${LOG_FILE}"
	fi
	done

	- name: Upload kms-core logs
	if: always()
	uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
	with:
	name: kms-core-rolling-upgrade-logs
	path: kms-core-*.txt
	retention-days: 30
	if-no-files-found: warn

	# ====================================================================
	# CLEANUP
	# ====================================================================
	- name: Cleanup
	if: always()
	run: \|
	echo "Starting cleanup of KMS resources..."
	helm list -n "${NAMESPACE}" -a -q \| xargs -r -n1 helm uninstall -n "${NAMESPACE}" \|\| true
	kubectl delete cm -l app=kms-core -n "${NAMESPACE}" \|\| true
	kubectl delete cm -l app=kms-core-client -n "${NAMESPACE}" \|\| true
	kubectl delete sts -l app=kms-core-client -n "${NAMESPACE}" \|\| true
	kubectl delete job -l app=kms-core -n "${NAMESPACE}" \|\| true
	kubectl delete job -l app=kms-threshold-init-job -n "${NAMESPACE}" \|\| true
	argo delete @latest -n kms-ci -n "${NAMESPACE}" \|\| true
	echo "Cleanup completed"

Navigation Menu

Uh oh!

rolling-upgrade-testing #38

Workflow file

rolling-upgrade-testing #38

Uh oh!

Workflow file for this run