Update stuck during update from 4.12.0-0.okd-2023-04-16-041331 to 4.12.0-0.okd-2023-11-20-220231

[peterroth]

My test cluster's update stuck during the procedure, the machine-config reports the following error:

machine-config                             4.12.0-0.okd-2023-04-16-041331   True        True          True       5d      Unable to apply 4.12.0-0.okd-2023-11-20-220231: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: machineconfig.machineconfiguration.openshift.io "rendered-master-687cb43d802834a531078f46d4e05ad1" not found, retrying]

The above-mentioned machine config certainly exists:

~/OKD: oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-687cb43d802834a531078f46d4e05ad1   False     True       False      3              0                   0                     0                      5d
worker   rendered-worker-3b0a8138a1965629d05ab1da3150b484   False     True       False      2              0                   0                     0                      5d

I had to edit the master's taints (with oc edit schedulers.config.openshift.io cluster) to be schedulable, as at installation the prometheus-k8s-0 couldn't start up due to insufficient memory in the worker nodes (GH#68182, hence the installation didn't finish.

Must-gather is available on this OneDrive link

[vrutkovs]

Seems rendered-master-687cb43d802834a531078f46d4e05ad1 (previous master MC) is not available, but there is uncordon-rendered-master-687cb43d802834a531078f46d4e05ad1 - was it recreated by you?

Check if creating rendered-master-687cb43d802834a531078f46d4e05ad1 with the same content as uncordon-... helps

[peterroth]

I checked if there is such a machineconfig, but can't find it. The command oc get mc doesn't list it, and if I run oc get mc uncordon-rendered-master-687cb43d802834a531078f46d4e05ad1 -o yaml I get an error message saying it isn't found.

[vrutkovs]

Oh, sorry, I see it being referred to in nodes: machineconfiguration.openshift.io/desiredDrain: uncordon-rendered-master-687cb43d802834a531078f46d4e05ad1, but not seeing it in the must-gather.

If you can ssh on the nodes afaik there should be a JSON of last applied MC at /etc/ignition-machine-config-encapsulated.json - is it there?

[peterroth]

I ran a quick check and none of the machines carry the /etc/ignition-machine-config-encapsulated.json file.

edit: I checked and a .bak file exist: ignition-machine-config-encapsulated.json.bak
Based on its content I think I could use it to regenerate the rendered-master-687cb43d802834a531078f46d4e05ad1 MC.

[peterroth]

Short update (just for the record): I re-created the rendered-master-687cb43d802834a531078f46d4e05ad1 machine config based on the above-mentioned .json.bak file.
A few minutes later the machine-config clusteroperator reported the following:

Unable to apply 4.12.0-0.okd-2023-11-20-220231: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-687cb43d802834a531078f46d4e05ad1 expected ef25c81205a65d5361cfc464e16fd5d47c0c6f17 has 87fedee690ae487f8ae044ac416000172c9576a5: 0 (ready 0) out of 3 nodes are updating to latest configuration rendered-master-e8f844e6a530a0bdb052f4bf418bd4e8, retrying]

It seems the machine-config expects something newer than what was actually on the cluster.
So, oc edit mc rendered-master-687cb43d802834a531078f46d4e05ad1 and edited the machineconfiguration.openshift.io/generated-by-controller-version to the expected value.

Again a few minutes and got this:

Unable to apply 4.12.0-0.okd-2023-11-20-220231: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: osImageURL mismatch for master in rendered-master-687cb43d802834a531078f46d4e05ad1 expected: registry.ci.openshift.org/origin/4.12-2023-11-20-220231@sha256:49a24e89421324f8259f91611861bf000d7946e71e50dc53e3fac6452a7f835f got: quay.io/openshift/okd-content@sha256:53abd41811f53c940ac41dc8b9d5475813d9c96e20ce893c8664ba951044d6a3, retrying]

The machine config surely expects the nightly build's (2023-11-20) rendered-master mc, however I was updating to that version (from the initial 2023-04-16).

[vrutkovs]

Its odd that MCO (?) has removed rendered- machine, but we'd need audit log to find out more. Also, to escalate this, we'd need to reproduce this on 4.15, which is not trivial.

Lets see if this happens again and see if any simple workaround exists

[peterroth]

After the trials and failures yesterday, the master machine config pool reported my 3 master nodes as available, 0 degraded machine and one mysterious node as unavailable (out of the 3 master I had).
Due to the goal of the cluster (testing updates) I deleted the setup.

thank you for your assistance, @vrutkovs