Adding VmWare Worker Node to OpenShift Cluster the BareMetal IPI Way

 

In a previous blog I discussed how one could provide Intelligent Platform Management Interface (IPMI) capabilities to a VmWare virtual machine.  I also eluded to being able to deploy OpenShift Baremetal IPI on VmWare virtual machines given the IPMI requirement was met for the purpose of a non production lab scenario.   However since I do not have enough lab equipment to run a full blown VmWare ESXi with enough virtual machines to mimic an OpenShift Baremetal IPI deployment, I will do the next best thing and demonstrate how to add a VmWare virtual machine acting as an OpenShift worker using the scale up capability.

Before we get started though lets review the lab setup for this exercise.   The diagram below shows that we have a 3 master cluster on a RHEL KVM hypervisor node.  These nodes while virtual are using VBMC to enable IPMI and hence the cluster was deployed as a OpenShift Baremetal IPI cluster.   We have an additional worker we would like to add that resides on an ESXi hypervisor host.   Using the virtualbmcforvsphere container (discussed in a previous blog) we can mimic IPMI for that worker node and thus treat it like a baremetal node.

Now that we have an understanding of the lab layout lets get to adding the additional VmWare worker node to our cluster.   The first step is to create the vmware-bmh.yaml which will contain the secret information for the IPMI credentials base64 encoded and the baremetal host information:

$ cat << EOF > ~/vmware-bmh.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: worker-4-bmc-secret
type: Opaque
data:
  username: YWRtaW4=
  password: cGFzc3dvcmQ=
---
apiVersion: metal3.io/v1alpha1
kind: BareMetalHost
metadata:
  name: worker-4
spec:
  online: true
  bootMACAddress: 00:50:56:83:da:a1
  bmc:
    address: ipmi://192.168.0.10:6801
    credentialsName: worker-4-bmc-secret
EOF

Once we have created the vmware-bmh.yaml file we can go ahead and create the resources with the oc command below:

$ oc create -f vmware-bmh.yaml -n openshift-machine-api
secret/worker-4-bmc-secret created
baremetalhost.metal3.io/worker-4 created	

Once the command is executed this will kick off the process of registering the node in ironic, turning the node on via IPMI and then inspecting the node to determine its resource properties.  The video below will show what is happening on the console of the worker node during this process:

Besides watching from the console, we can also run some oc commands to see the status of the worker node during this process as well:

$ oc get baremetalhosts -n openshift-machine-api
NAME       STATE                    CONSUMER               ONLINE   ERROR
master-0   externally provisioned   kni20-cmq65-master-0   true     
master-1   externally provisioned   kni20-cmq65-master-1   true     
master-2   externally provisioned   kni20-cmq65-master-2   true     
worker-4   registering                                     true 

$ oc get baremetalhosts -n openshift-machine-api
NAME       STATE                    CONSUMER               ONLINE   ERROR
master-0   externally provisioned   kni20-cmq65-master-0   true     
master-1   externally provisioned   kni20-cmq65-master-1   true     
master-2   externally provisioned   kni20-cmq65-master-2   true     
worker-4   inspecting                                      true     

$ oc get baremetalhosts -n openshift-machine-api
NAME       STATE                    CONSUMER               ONLINE   ERROR
master-0   externally provisioned   kni20-cmq65-master-0   true     
master-1   externally provisioned   kni20-cmq65-master-1   true     
master-2   externally provisioned   kni20-cmq65-master-2   true     
worker-4   match profile                                   true     

$ oc get baremetalhosts -n openshift-machine-api
NAME       STATE                    CONSUMER               ONLINE   ERROR
master-0   externally provisioned   kni20-cmq65-master-0   true     
master-1   externally provisioned   kni20-cmq65-master-1   true     
master-2   externally provisioned   kni20-cmq65-master-2   true     
worker-4   ready                                           true  

Once the process is complete the new worker node will be marked ready and left powered on.  Now we can move onto scaling up the cluster.   To do this we first need to find the name of the machineset which in this case is kni20-cmq65-worker-0.  With that information we can then scale up the node count from 0 to 1 and this will trigger the provisioning process:

$ oc -n openshift-machine-api get machineset
NAME                   DESIRED   CURRENT   READY   AVAILABLE   AGE
kni20-cmq65-worker-0   0         0                             17h

$ oc -n openshift-machine-api scale machineset kni20-cmq65-worker-0 --replicas=1
machineset.machine.openshift.io/kni20-cmq65-worker-0 scaled

The video below will show what happens during the scaling process from the worker nodes console point of view.  In summary what will happen is the node will turn on, an RHCOS image will get written, the node will reboot, the ostree will get updated, the node will reboot again and finally the services to enable the node to join the cluster will start:

Besides watching from the console of the worker node we can also following along at the cli with the oc command to show the state of the worker node:

$ oc get baremetalhosts -n openshift-machine-api
NAME       STATE                    CONSUMER                     ONLINE   ERROR
master-0   externally provisioned   kni20-cmq65-master-0         true     
master-1   externally provisioned   kni20-cmq65-master-1         true     
master-2   externally provisioned   kni20-cmq65-master-2         true     
worker-4   provisioning             kni20-cmq65-worker-0-lhd92   true 

And again using the oc command we can see the worker node has been provisioned:

$ oc get baremetalhosts -n openshift-machine-api
NAME       STATE                    CONSUMER                     ONLINE   ERROR
master-0   externally provisioned   kni20-cmq65-master-0         true     
master-1   externally provisioned   kni20-cmq65-master-1         true     
master-2   externally provisioned   kni20-cmq65-master-2         true     
worker-4   provisioned              kni20-cmq65-worker-0-lhd92   true 

Once the worker node has shown provisioned and the node has rebooted the second time, we can then follow the status of the worker node with the oc get nodes command: 

$ oc get nodes
NAME                             STATUS     ROLES           AGE   VERSION
master-0.kni20.schmaustech.com   Ready      master,worker   17h   v1.22.0-rc.0+a44d0f0
master-1.kni20.schmaustech.com   Ready      master,worker   17h   v1.22.0-rc.0+a44d0f0
master-2.kni20.schmaustech.com   Ready      master,worker   17h   v1.22.0-rc.0+a44d0f0
worker-4.kni20.schmaustech.com   NotReady   worker          39s   v1.22.0-rc.0+a44d0f0

Finally after the scaling process is completed and the worker node should display that it is ready and joined to the cluster:

$ oc get nodes
NAME                             STATUS   ROLES           AGE   VERSION
master-0.kni20.schmaustech.com   Ready    master,worker   17h   v1.22.0-rc.0+a44d0f0
master-1.kni20.schmaustech.com   Ready    master,worker   17h   v1.22.0-rc.0+a44d0f0
master-2.kni20.schmaustech.com   Ready    master,worker   17h   v1.22.0-rc.0+a44d0f0
worker-4.kni20.schmaustech.com   Ready    worker          58s   v1.22.0-rc.0+a44d0f0

Hopefully this provides a good example of how to use VmWare virtual machines to simulate baremetal nodes for OpenShift IPI deployments.