OpenshiftSDN to OVN-Kubernetes migration failed

[balait4]

Hi All,

I'm doing network migration from OpenshiftSDN to OVN-Kubernetes in my 4.16.0-okd-scos.1 cluster. Got into this issue, any recommendation how to proceed further?

# oc patch Network.config.openshift.io cluster --type='merge' --patch '{"metadata":{"annotations":{"network.openshift.io/network-type-migration":""}},"spec":{"networkType":"OVNKubernetes"}}'

E0503 14:41:56.346020       1 controller.go:329] "Reconciler error" err="invalid configuration: [can only change default network type to the target migration network type]" controller="operconfig-controller" object="cluster" namespace="" name="cluster" reconcileID="0b4fb39f-49cc-46a3-99d9-6798d1b25e1d"

$ oc get co network
NAME      VERSION             AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
network   4.16.0-okd-scos.1   True        False         True       3y46d   Not applying unsafe configuration change: invalid configuration: [can only change default network type to the target migration network type]. Use 'oc edit network.operator.openshift.io cluster' to undo the change.

As it is stuck and not progressing further I tried below to rollback to OpenshiftSDN, all node got restarted but still the NetworkTypeMigrationStarted: status: "True" waiting for more than 12 hours still the same.. So I'm not sure what to do next

# oc get  Network.config.openshift.io cluster -o yaml

apiVersion: config.openshift.io/v1
kind: Network
metadata:
  name: cluster
spec:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  externalIP:
    policy: {}
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16
status:
  clusterNetwork:
  - cidr: 10.128.0.0/14
    hostPrefix: 23
  clusterNetworkMTU: 1450
  conditions:
  - lastTransitionTime: "2025-05-04T09:31:36Z"
    message: ""
    reason: AsExpected
    status: "True"
    type: NetworkDiagnosticsAvailable
  - lastTransitionTime: "2025-05-03T14:09:58Z"
    message: The desired MachineConfig is applied to cluster
    reason: MachineConfigApplied
    status: "True"
    type: NetworkTypeMigrationMTUReady
  - lastTransitionTime: "2025-05-03T12:36:57Z"
    message: OpenShiftSDN is deployed
    reason: TargetCNIDeployed
    status: "True"
    type: NetworkTypeMigrationTargetCNIAvailable
  - lastTransitionTime: "2025-05-03T10:32:14Z"
    message: The desired MachineConfig is not applied to cluster
    reason: MachineConfigNotApplied
    status: "False"
    type: NetworkTypeMigrationTargetCNIInUse
  - lastTransitionTime: "2025-05-03T10:32:14Z"
    message: The original CNI plugin is running in the cluster
    reason: OriginalCNINotPurged
    status: "False"
    type: NetworkTypeMigrationOriginalCNIPurged
  - lastTransitionTime: "2025-05-03T10:32:14Z"
    message: Network type migration is started
    reason: NetworkTypeMigrationStarted
    status: "True"
    type: NetworkTypeMigrationInProgress
  networkType: OpenShiftSDN
  serviceNetwork:
  - 172.30.0.0/16

oc patch Network.config.openshift.io cluster --type='merge' --patch '{"metadata":{"annotations":{"network.openshift.io/network-type-migration":""}},"spec":{"networkType":"OpenShiftSDN"}}'
network.config.openshift.io/cluster patched

[Network_migration.txt](https://github.com/user-attachments/files/20029068/Network_migration.txt)

[balait4]

@vrutkovs any thought on this please?

[balait4]

Hi All, any help on this please? How to clear this NetworkTypeMigrationStarted: status: "True" and start the migration again?

[twilson724]

As noted in https://docs.okd.io/4.16/networking/ovn_kubernetes_network_provider/migrate-from-openshift-sdn.html#considerations-live-migrating-ovn-kubernetes-network-provider_migrate-from-openshift-sdn .

The migration is intended to be a one-way process. However, for users that want to rollback to OpenShift-SDN, migration from OpenShift-SDN to OVN-Kubernetes must have succeeded.

When your cluster was stuck what was the state of the cluster operators, were nodes failing to drain and reboot to apply the updates? To progress you will need to go forward with the migration. If you continue the migration please provide events from degraded operators

[balait4]

Thanks for the reply, please find the logs attached
Network_migration.txt

It didnt progress further with this error, yes nodes failed to drain, then I reverted the changes with patch it back to OpenShift-SDN, where it reboots all the node and then the status shows its currently with OpenShift-SDN. But NetworkTypeMigrationStarted: status: "True" was still there.

It went smooth in my other cluster, in this cluster I created multiple MCP before this migration to control the reboot of my worker node. May be its not able to patch my newly created MCP? But I don't see any error related to that.. Its been a month now which i reverted, but due to this I'm not able to upgrade the cluster. So again I want to try the network migration. So what will be your suggestion ? should I clean up the current migration state with below command and restart the migration only with one mcp worker and master which is default one?

oc patch Network.config.openshift.io cluster --type='merge' --patch '{"metadata":{"annotations":{"network.openshift.io/network-type-migration":null}}}'
oc patch Network.config.openshift.io cluster --type='merge' --patch '{"status":{"conditions":null}}'

[balait4]

Should this be safe to retry the migration without clearing the NetworkTypeMigrationInProgress: True ?

[balait4]

with the oc get Network.config.openshift.io cluster -o yaml its

OpenShiftSDN is deployed (TargetCNIAvailable: True)

OVN-Kubernetes is not fully purged (OriginalCNINotPurged: False)

Migration is still marked as in progress (NetworkTypeMigrationInProgress: True)

The MCP is DEGRADED is FALSE. The network-operator reports no error and shows successful reconciliation.

[Tyrone-W]

I wouldn't be able to decide if it is safe to retry the migration, but if I had a cluster in this state I would try the migration back to OVN-Kubernetes and look at the operator errors generated there. Or use a test cluster to try to replicate this state and test solutions.

[balait4]

Thanks for the reply! The issue is I already upgraded my test and dev cluster and didn't face this issue. And also I don't know if at all I can reproduce this. My Only concern is if I retry it should not break the cluster. Now everything seems to be fine and only this NetworkTypeMigrationInProgress: True makes confusion. So need to clear this status before retry the migration or directly can try the migration to OVN-kubernetes?

[balait4]

@twilson724 @Tyrone-W I tired to patch to OVNKubernetes though still NetworkTypeMigrationInProgress: True was there.. But this time it says network operator degraded.

Network Operator - Not applying unsafe configuration change: invalid configuration: [cannot change default network type when not doing migration]. Use 'oc edit network.operator.openshift.io cluster' to undo the change.

so had to edit oc edit network.config.openshift.io cluster to put back the OpenShiftSDN.

May be the NetworkTypeMigrationInProgress:True is the issue here? How can I cleanup this one? Any idea on this? Thanks

[balait4]

All good able to clear the status and then migrated the network successfully. Thanks