Friday, June 04, 2021

Migrating From OpenShiftSDN to OVNKubernetes

 


In previous version of OpenShift if one wanted to switch from OpenShiftSDN to OVNKubernetes there was no migration path and hence the cluster needed to be reinstalled.   That burden is about to become a thing of the past with OpenShift 4.8 because in this version and beyond the ability to migrate the cluster without re-installation is now possible.   In the following details below I will outline and show an example of how this process looks on a working OpenShift cluster.

First lets cover some basics about the environment setup.   I have a five node OpenShift 4.8.0-fc.7 pre-release cluster.   Three of those nodes are masters and the other two are worker nodes in a baremetal IPI deployment.   My current network type is configured as OpenShiftSDN.

First lets validate that the current cluster is running without issues.  We can start by confirming all the nodes are in a ready state:

$ oc get nodes
NAME                                 STATUS   ROLES    AGE     VERSION
master-0.n6s2d.dynamic.opentlc.com   Ready    master   31m     v1.21.0-rc.0+4b2b6ff
master-1.n6s2d.dynamic.opentlc.com   Ready    master   31m     v1.21.0-rc.0+4b2b6ff
master-2.n6s2d.dynamic.opentlc.com   Ready    master   31m     v1.21.0-rc.0+4b2b6ff
worker-0.n6s2d.dynamic.opentlc.com   Ready    worker   13m     v1.21.0-rc.0+4b2b6ff
worker-1.n6s2d.dynamic.opentlc.com   Ready    worker   8m53s   v1.21.0-rc.0+4b2b6ff

Next lets confirm the cluster operators are all functioning properly and are available:

$ oc get co
NAME                                       VERSION      AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.8.0-fc.7   True        False         False      51s
baremetal                                  4.8.0-fc.7   True        True          False      29m
cloud-credential                           4.8.0-fc.7   True        False         False      43m
cluster-autoscaler                         4.8.0-fc.7   True        False         False      29m
config-operator                            4.8.0-fc.7   True        False         False      30m
console                                    4.8.0-fc.7   True        False         False      7m30s
csi-snapshot-controller                    4.8.0-fc.7   True        False         False      15m
dns                                        4.8.0-fc.7   True        False         False      29m
etcd                                       4.8.0-fc.7   True        False         False      28m
image-registry                             4.8.0-fc.7   True        False         False      25m
ingress                                    4.8.0-fc.7   True        False         False      12m
insights                                   4.8.0-fc.7   True        False         False      23m
kube-apiserver                             4.8.0-fc.7   True        False         False      27m
kube-controller-manager                    4.8.0-fc.7   True        False         False      27m
kube-scheduler                             4.8.0-fc.7   True        False         False      28m
kube-storage-version-migrator              4.8.0-fc.7   True        False         False      17m
machine-api                                4.8.0-fc.7   True        False         False      24m
machine-approver                           4.8.0-fc.7   True        False         False      30m
machine-config                             4.8.0-fc.7   True        False         False      29m
marketplace                                4.8.0-fc.7   True        False         False      29m
monitoring                                 4.8.0-fc.7   True        False         False      8m9s
network                                    4.8.0-fc.7   True        False         False      30m
node-tuning                                4.8.0-fc.7   True        False         False      29m
openshift-apiserver                        4.8.0-fc.7   True        False         False      15m
openshift-controller-manager               4.8.0-fc.7   True        False         False      28m
openshift-samples                          4.8.0-fc.7   True        False         False      25m
operator-lifecycle-manager                 4.8.0-fc.7   True        False         False      29m
operator-lifecycle-manager-catalog         4.8.0-fc.7   True        False         False      29m
operator-lifecycle-manager-packageserver   4.8.0-fc.7   True        False         False      26m
service-ca                                 4.8.0-fc.7   True        False         False      30m
storage                                    4.8.0-fc.7   True        False         False      30m

And for fun lets show that the OpenShiftSDN pods are running as that is our current network type on this cluster:

$ oc get pods -n openshift-sdn 
NAME                   READY   STATUS    RESTARTS   AGE
sdn-controller-chcvr   1/1     Running   3          31m
sdn-controller-wzhdz   1/1     Running   0          31m
sdn-controller-zp4qr   1/1     Running   0          31m
sdn-czhf6              2/2     Running   0          9m14s
sdn-fvnt6              2/2     Running   0          31m
sdn-njcfn              2/2     Running   0          14m
sdn-nmpt2              2/2     Running   0          31m
sdn-zj2fz              2/2     Running   0          31m

At this point I feel comfortable moving onto the migration process.  If anything above looked suspicious or not functioning properly then I would pause and address that first.   For now though we can move on and the first this we will do is backup the current configuration for the cluster network:

$ oc get Network.config.openshift.io cluster -o yaml > cluster-openshift-sdn.yaml
$

Once we have our backup we can now set the migration field on the cluster network operator.  Note that this will not trigger a migration but rather tell the machine config operator (MCO) to apply new machine configs to all the nodes in the cluster in preparation for OVN-Kubernetes:

$ oc patch Network.operator.openshift.io cluster --type='merge' --patch '{ "spec": { "migration": {"networkType": "OVNKubernetes" } } }'
network.operator.openshift.io/cluster patched

At this point if we look at the mcp and describe our nodes and grep for macine config we can see things are starting to update from an MCO perspective:

$ oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-1309c87d08601c58ddf8abd9bb08a7ea   False     True       False      3              0                   0                     0                      34m
worker   rendered-worker-67d75dc2ab02f295344703f157da14d7   False     True       False      2              0                   0                     0                      34m

$ oc describe node | egrep "hostname|machineconfig"
                    kubernetes.io/hostname=master-0.n6s2d.dynamic.opentlc.com
                    machineconfiguration.openshift.io/controlPlaneTopology: HighlyAvailable
                    machineconfiguration.openshift.io/currentConfig: rendered-master-1309c87d08601c58ddf8abd9bb08a7ea
                    machineconfiguration.openshift.io/desiredConfig: rendered-master-1309c87d08601c58ddf8abd9bb08a7ea
                    machineconfiguration.openshift.io/reason: 
                    machineconfiguration.openshift.io/state: Done
                    kubernetes.io/hostname=master-1.n6s2d.dynamic.opentlc.com
                    machineconfiguration.openshift.io/controlPlaneTopology: HighlyAvailable
                    machineconfiguration.openshift.io/currentConfig: rendered-master-1309c87d08601c58ddf8abd9bb08a7ea
                    machineconfiguration.openshift.io/desiredConfig: rendered-master-9b49007a4d027c9d7f40cfd3c485e31f
                    machineconfiguration.openshift.io/reason: 
                    machineconfiguration.openshift.io/state: Working
                    kubernetes.io/hostname=master-2.n6s2d.dynamic.opentlc.com
                    machineconfiguration.openshift.io/controlPlaneTopology: HighlyAvailable
                    machineconfiguration.openshift.io/currentConfig: rendered-master-1309c87d08601c58ddf8abd9bb08a7ea
                    machineconfiguration.openshift.io/desiredConfig: rendered-master-1309c87d08601c58ddf8abd9bb08a7ea
                    machineconfiguration.openshift.io/reason: 
                    machineconfiguration.openshift.io/state: Done
                    kubernetes.io/hostname=worker-0.n6s2d.dynamic.opentlc.com
                    machineconfiguration.openshift.io/controlPlaneTopology: HighlyAvailable
                    machineconfiguration.openshift.io/currentConfig: rendered-worker-67d75dc2ab02f295344703f157da14d7
                    machineconfiguration.openshift.io/desiredConfig: rendered-worker-e17471318b8b3a61fc931cedeca303e1
                    machineconfiguration.openshift.io/reason: 
                    machineconfiguration.openshift.io/state: Working
                    kubernetes.io/hostname=worker-1.n6s2d.dynamic.opentlc.com
                    machineconfiguration.openshift.io/controlPlaneTopology: HighlyAvailable
                    machineconfiguration.openshift.io/currentConfig: rendered-worker-67d75dc2ab02f295344703f157da14d7
                    machineconfiguration.openshift.io/desiredConfig: rendered-worker-67d75dc2ab02f295344703f157da14d7
                    machineconfiguration.openshift.io/reason: 
                    machineconfiguration.openshift.io/state: Done

We will know the updates are done when we see that the updated field for the mcp is set to true for both the worker and master nodes.  We should also see that the status of the machine configuration state is set to done for every node in the cluster.  Do not preceed until the output looks similar to the below:

$ oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-9b49007a4d027c9d7f40cfd3c485e31f   True      False      False      3              3                   3                     0                      46m
worker   rendered-worker-e17471318b8b3a61fc931cedeca303e1   True      False      False      2              2                   2                     0                      46m

$ oc describe node | egrep "hostname|machineconfig"
                    kubernetes.io/hostname=master-0.n6s2d.dynamic.opentlc.com
                    machineconfiguration.openshift.io/controlPlaneTopology: HighlyAvailable
                    machineconfiguration.openshift.io/currentConfig: rendered-master-9b49007a4d027c9d7f40cfd3c485e31f
                    machineconfiguration.openshift.io/desiredConfig: rendered-master-9b49007a4d027c9d7f40cfd3c485e31f
                    machineconfiguration.openshift.io/reason: 
                    machineconfiguration.openshift.io/state: Done
                    kubernetes.io/hostname=master-1.n6s2d.dynamic.opentlc.com
                    machineconfiguration.openshift.io/controlPlaneTopology: HighlyAvailable
                    machineconfiguration.openshift.io/currentConfig: rendered-master-9b49007a4d027c9d7f40cfd3c485e31f
                    machineconfiguration.openshift.io/desiredConfig: rendered-master-9b49007a4d027c9d7f40cfd3c485e31f
                    machineconfiguration.openshift.io/reason: 
                    machineconfiguration.openshift.io/state: Done
                    kubernetes.io/hostname=master-2.n6s2d.dynamic.opentlc.com
                    machineconfiguration.openshift.io/controlPlaneTopology: HighlyAvailable
                    machineconfiguration.openshift.io/currentConfig: rendered-master-9b49007a4d027c9d7f40cfd3c485e31f
                    machineconfiguration.openshift.io/desiredConfig: rendered-master-9b49007a4d027c9d7f40cfd3c485e31f
                    machineconfiguration.openshift.io/reason: 
                    machineconfiguration.openshift.io/state: Done
                    kubernetes.io/hostname=worker-0.n6s2d.dynamic.opentlc.com
                    machineconfiguration.openshift.io/controlPlaneTopology: HighlyAvailable
                    machineconfiguration.openshift.io/currentConfig: rendered-worker-e17471318b8b3a61fc931cedeca303e1
                    machineconfiguration.openshift.io/desiredConfig: rendered-worker-e17471318b8b3a61fc931cedeca303e1
                    machineconfiguration.openshift.io/reason: 
                    machineconfiguration.openshift.io/state: Done
                    kubernetes.io/hostname=worker-1.n6s2d.dynamic.opentlc.com
                    machineconfiguration.openshift.io/controlPlaneTopology: HighlyAvailable
                    machineconfiguration.openshift.io/currentConfig: rendered-worker-e17471318b8b3a61fc931cedeca303e1
                    machineconfiguration.openshift.io/desiredConfig: rendered-worker-e17471318b8b3a61fc931cedeca303e1
                    machineconfiguration.openshift.io/reason: 
                    machineconfiguration.openshift.io/state: Done

To confirm that OVN-Kubernetes has been appropriately staged for the migration we can look at the machine config for both workers and masters and see that the configure-ovs.sh script has been set to start OVNKubernetes:

$ oc get machineconfig rendered-master-9b49007a4d027c9d7f40cfd3c485e31f -o yaml | grep ExecStart | grep OVNKubernetes
ExecStart=/usr/local/bin/configure-ovs.sh OVNKubernetes $ oc get machineconfig rendered-worker-e17471318b8b3a61fc931cedeca303e1 -o yaml | grep ExecStart | grep OVNKubernetes ExecStart=/usr/local/bin/configure-ovs.sh OVNKubernetes

At this point we are ready to start the migration process and we do that by patching the network type and setting it to OVNKubernetes:

$ oc patch Network.config.openshift.io cluster --type='merge' --patch '{ "spec": { "networkType": "OVNKubernetes" } }'
network.config.openshift.io/cluster patched

Once we have executed the patch command the multus daemon set will begin to be rolled out.  We can observe that by watching the rollout status:

$ oc -n openshift-multus rollout status daemonset/multus
Waiting for daemon set "multus" rollout to finish: 1 out of 5 new pods have been updated...
Waiting for daemon set "multus" rollout to finish: 1 out of 5 new pods have been updated...
Waiting for daemon set "multus" rollout to finish: 1 out of 5 new pods have been updated...
Waiting for daemon set "multus" rollout to finish: 2 out of 5 new pods have been updated...
Waiting for daemon set "multus" rollout to finish: 2 out of 5 new pods have been updated...
Waiting for daemon set "multus" rollout to finish: 2 out of 5 new pods have been updated...
Waiting for daemon set "multus" rollout to finish: 3 out of 5 new pods have been updated...
Waiting for daemon set "multus" rollout to finish: 3 out of 5 new pods have been updated...
Waiting for daemon set "multus" rollout to finish: 3 out of 5 new pods have been updated...
Waiting for daemon set "multus" rollout to finish: 4 out of 5 new pods have been updated...
Waiting for daemon set "multus" rollout to finish: 4 out of 5 new pods have been updated...
Waiting for daemon set "multus" rollout to finish: 4 out of 5 new pods have been updated...
Waiting for daemon set "multus" rollout to finish: 4 of 5 updated pods are available...
daemon set "multus" successfully rolled out

One the rollout status is complete we next need to reboot all the nodes in the cluster.  The following script below makes it easy to do this process assuming ssh keys are configured properly:

$ cat << EOF > ~/reboot-nodes.sh
#!/bin/bash
 
for ip in $(oc get nodes  -o jsonpath='{.items[*].status.addresses[?(@.type=="InternalIP")].address}')
do
  echo "reboot node $ip"
  ssh -o StrictHostKeyChecking=no core@\$ip sudo shutdown -r -t 3
done
EOF

Once we have created the script we can go ahead and execute it to start the reboot process:

$ ~/reboot-nodes.sh
reboot node 10.20.0.100
Warning: Permanently added '10.20.0.100' (ECDSA) to the list of known hosts.
Shutdown scheduled for Fri 2021-06-04 13:44:07 UTC, use 'shutdown -c' to cancel.
reboot node 10.20.0.101
Warning: Permanently added '10.20.0.101' (ECDSA) to the list of known hosts.
Shutdown scheduled for Fri 2021-06-04 13:44:08 UTC, use 'shutdown -c' to cancel.
reboot node 10.20.0.102
Warning: Permanently added '10.20.0.102' (ECDSA) to the list of known hosts.
Shutdown scheduled for Fri 2021-06-04 13:44:08 UTC, use 'shutdown -c' to cancel.
reboot node 10.20.0.200
Warning: Permanently added '10.20.0.200' (ECDSA) to the list of known hosts.
Shutdown scheduled for Fri 2021-06-04 13:44:09 UTC, use 'shutdown -c' to cancel.
reboot node 10.20.0.201
Warning: Permanently added '10.20.0.201' (ECDSA) to the list of known hosts.
Shutdown scheduled for Fri 2021-06-04 13:44:10 UTC, use 'shutdown -c' to cancel.


Depending on the number of nodes and how quickly they reboot this process could take a little bit.    After some time has passed though we should be cable to confirm that the nodes are back up and in a ready state:

$ oc get nodes
NAME                                 STATUS   ROLES    AGE   VERSION
master-0.n6s2d.dynamic.opentlc.com   Ready    master   64m   v1.21.0-rc.0+4b2b6ff
master-1.n6s2d.dynamic.opentlc.com   Ready    master   64m   v1.21.0-rc.0+4b2b6ff
master-2.n6s2d.dynamic.opentlc.com   Ready    master   64m   v1.21.0-rc.0+4b2b6ff
worker-0.n6s2d.dynamic.opentlc.com   Ready    worker   46m   v1.21.0-rc.0+4b2b6ff
worker-1.n6s2d.dynamic.opentlc.com   Ready    worker   41m   v1.21.0-rc.0+4b2b6ff

We can also confirm that the network type is now configured as OVNKubernetes:

$ oc get network.config/cluster -o jsonpath='{.status.networkType}{"\n"}'
OVNKubernetes

We should also confirm that there are no pods that are in a pending or crashloop state:

$ oc get pods --all-namespaces -o wide --sort-by='{.spec.nodeName}' | egrep -v "Running|Completed"
NAMESPACE          NAME       READY   STATUS      RESTARTS   AGE    IP            NODE               NOMINATED NODE   READINESS GATES

And we can also validate that all the cluster operators are running appropriately:

$ oc get co
NAME                                       VERSION      AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.8.0-fc.7   True        False         False      4m36s
baremetal                                  4.8.0-fc.7   True        False         False      63m
cloud-credential                           4.8.0-fc.7   True        False         False      76m
cluster-autoscaler                         4.8.0-fc.7   True        False         False      63m
config-operator                            4.8.0-fc.7   True        False         False      64m
console                                    4.8.0-fc.7   True        False         False      4m21s
csi-snapshot-controller                    4.8.0-fc.7   True        False         False      21m
dns                                        4.8.0-fc.7   True        False         False      63m
etcd                                       4.8.0-fc.7   True        False         False      62m
image-registry                             4.8.0-fc.7   True        False         False      59m
ingress                                    4.8.0-fc.7   True        False         False      4m5s
insights                                   4.8.0-fc.7   True        False         False      57m
kube-apiserver                             4.8.0-fc.7   True        False         False      61m
kube-controller-manager                    4.8.0-fc.7   True        False         False      60m
kube-scheduler                             4.8.0-fc.7   True        False         False      61m
kube-storage-version-migrator              4.8.0-fc.7   True        False         False      27m
machine-api                                4.8.0-fc.7   True        False         False      58m
machine-approver                           4.8.0-fc.7   True        False         False      63m
machine-config                             4.8.0-fc.7   True        False         False      63m
marketplace                                4.8.0-fc.7   True        False         False      63m
monitoring                                 4.8.0-fc.7   True        False         False      4m6s
network                                    4.8.0-fc.7   True        False         False      64m
node-tuning                                4.8.0-fc.7   True        False         False      63m
openshift-apiserver                        4.8.0-fc.7   True        False         False      4m40s
openshift-controller-manager               4.8.0-fc.7   True        False         False      62m
openshift-samples                          4.8.0-fc.7   True        False         False      59m
operator-lifecycle-manager                 4.8.0-fc.7   True        False         False      63m
operator-lifecycle-manager-catalog         4.8.0-fc.7   True        False         False      63m
operator-lifecycle-manager-packageserver   4.8.0-fc.7   True        False         False      4m30s
service-ca                                 4.8.0-fc.7   True        False         False      64m
storage                                    4.8.0-fc.7   True        False         False      64m

If everything looked good from the steps above we can go ahead and remove the cluster network object configuration object by setting it to null:

$ oc patch Network.operator.openshift.io cluster --type='merge' --patch '{ "spec": { "migration": null } }'
network.operator.openshift.io/cluster patched

We can also remove the customer configuration for OpenShiftSDN:

$ oc patch Network.operator.openshift.io cluster --type='merge' --patch '{ "spec": { "defaultNetwork": { "openshiftSDNConfig": null } } }'
network.operator.openshift.io/cluster patched (no change)

And finally we can remove the old openshift-sdn:

$ oc delete namespace openshift-sdn
namespace "openshift-sdn" deleted

And just for one final confirmation of success lets confirm the OVNKubernetes pods are up and running:

$ oc get pods -n openshift-ovn-kubernetes
NAME                   READY   STATUS    RESTARTS   AGE
ovnkube-master-5v8ch   6/6     Running   14         50m
ovnkube-master-kmwkp   6/6     Running   6          50m
ovnkube-master-zlgnv   6/6     Running   14         50m
ovnkube-node-7vrmq     4/4     Running   4          50m
ovnkube-node-kz4l9     4/4     Running   4          50m
ovnkube-node-nhbdz     4/4     Running   5          50m
ovnkube-node-nwnnk     4/4     Running   5          50m
ovnkube-node-t27gb     4/4     Running   5          50m

No comments: