Sunday, November 17, 2019

Deploying OpenShift IPI Baremetal Disconnected


The following blog is a write up of the steps I used to generate a OpenShift IPI disconnected baremetal install. In this configuration we first mirror down the images to a local repository. Then deploy the Openshift cluster onto virtual machines that are mimicking a baremetal environment with vBMC being used as the IPMI interface into the virtual machines.  Please note that while this was demonstrated in a virtual setting these steps should also work with physical hardware.

Lab Setup:

Physical node specification:

Processors: i7 - 8 vcpus
Memory: 32gb
Disk: 512gb M2 SSD
Single NIC with 2 vlans (external and provisioning) tagged in
Nested virtualization should be enabled

Virtual machine Node specification:

Processor: Passthrough from physical node - 4 vcpus
Memory: 16gb
Disk: 60gb raw image
Two NICs ens3 (provisioning) ens4 (external) - no tags - dhcp interfaces
IPMI: Provided via centralized Virtual BMC controller

Lab Diagram:


Predefined DNS records:

*.apps.kni5  IN A 192.168.0.197
ns1.kni5  IN A 192.168.0.198
api.kni5  IN A 192.168.0.199
master-0.kni5         IN A 192.168.0.200
master-1.kni5         IN A 192.168.0.201
master-2.kni5         IN A 192.168.0.202

Cluster and user definitions:

Cluster Name: kni5
Domain Name: schmaustech.com
Username for installation: bschmaus

Preparing Provisioning Node:

The first step in preparing the provisioning node is to install, kickstart or image the provisioning node with RHEL8 and register the host.  I leverage an ISO image local on NUC-1 with an embedded kickstart file that way I can rebuild my provisioning node on the fly and start off with a fresh install.

Once the provisioning node is installed make sure the following packages are installed:

kexec-tools
@development (package group)
git
usbredir
golang
libXv
virt-install
libvirt
libvirt-devel
libselinux-utils
qemu-kvm
mkisofs

Next lets make sure the username used for installation has passwordless sudo access.  This is more about convenience give a lot of the commands require root and or sudo access:

# cat << EOF > /etc/sudoers.d/openshift
Defaults:bschmaus !requiretty
bschmaus ALL = (root) NOPASSWD:ALL
EOF
# chmod 600 /etc/sudoers.d/openshift

Unfortunately at this time selinux does need to be set to permissive so we will do that now:

# sudo setenforce permissive
# sudo sed -i "s/=enforcing/=permissive/g" /etc/selinux/config

For the bootstrap node to boot on this virtual machine we need to ensure a default storage pool exists.  On a RHEL8 installation this pool does not seem to exist out of the box so lets create it:

# sudo virsh pool-define-as --name default --type dir --target /var/lib/libvirt/images
# sudo virsh pool-start default
# sudo virsh pool-autostart default
# sudo usermod --append --groups libvirt bschmaus

The Openshift installer expects there to be a baremetal and provisioning interface on the provisioning node so lets configure them with the following:

# export PROV_CONN=ens3
# export MAIN_CONN=ens4
# sudo nmcli connection add ifname provisioning type bridge con-name provisioning
# sudo nmcli con add type bridge-slave ifname "$PROV_CONN" master provisioning
# sudo nmcli connection add ifname baremetal type bridge con-name baremetal
# sudo nmcli con add type bridge-slave ifname "$MAIN_CONN" master baremetal
# sudo nmcli con down "System $MAIN_CONN"; sudo pkill dhclient; sudo dhclient baremetal
# sudo nmcli connection modify provisioning ipv4.addresses 172.22.0.1/24 ipv4.method manual
# sudo nmcli con down provisioning
# sudo nmcli con up provisioning
# sudo ip a show $PROV_CONN;ip a show $MAIN_CONN; ip a show provisioning; ip a show baremetal
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master provisioning state UP group default qlen 1000
    link/ether 52:54:00:9e:01:ec brd ff:ff:ff:ff:ff:ff
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master baremetal state UP group default qlen 1000
    link/ether 52:54:00:07:98:b1 brd ff:ff:ff:ff:ff:ff
8: provisioning: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:9e:01:ec brd ff:ff:ff:ff:ff:ff
    inet 172.22.0.1/24 brd 172.22.0.255 scope global noprefixroute provisioning
       valid_lft forever preferred_lft forever
    inet6 fe80::b189:6c77:d795:57dc/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
7: baremetal: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:07:98:b1 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.246/24 brd 192.168.0.255 scope global dynamic noprefixroute baremetal
       valid_lft 366sec preferred_lft 366sec
    inet6 fe80::495e:1100:2ad3:851e/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

At this time we should also pull the latest oc binary and openshift-baremetal-install binary.  The oc command will be used to mirror the repository and also validate the cluster is up once installation is complete:

# export VERSION=$(curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/latest/release.txt | grep 'Name:' | awk -F: '{print $2}' | xargs)
# export RELEASE_IMAGE=$(curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/latest/release.txt | grep 'Pull From: quay.io' | awk -F ' ' '{print $3}' | xargs)
# export pullsecret_file=/home/bschmaus/pull-secret.json
# export cmd=openshift-baremetal-install
# export extract_dir=$(pwd)
# curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/latest/openshift-client-linux-$VERSION.tar.gz | tar zxvf - oc
# sudo cp /home/bschmaus/oc /usr/local/bin/oc
# /usr/local/bin/oc adm release extract --registry-config "${pullsecret_file}" --command=$cmd --to "${extract_dir}" ${RELEASE_IMAGE}
# sudo cp /home/bschmaus/openshift-baremetal-install /usr/local/bin/openshift-baremetal-install 

Create Initial Install-Config.yaml and Local Image Repository:

Now that we have prepared the provisioning host, we need to first create our initial install-config.yaml file.   The file should look similar to the sample below but adjusted for your environment (Note: RELEASEVERSION should be typed as in example as we will change that in later step):

apiVersion: v1
baseDomain: schmaustech.com
metadata:
  name: kni5
networking:
  machineCIDR: 192.168.0.0/24
compute:
- name: worker
  replicas: 0
controlPlane:
  name: master
  replicas: 3
  platform:
    baremetal: {}
platform:
  baremetal:
    apiVIP: 192.168.0.199
    ingressVIP: 192.168.0.197
    dnsVIP: 192.168.0.198
    hosts:
      - name: master-0
        role: master
        bmc:
          address: ipmi://192.168.0.11:6241
          username: admin
          password: password
        bootMACAddress: 52:54:00:3d:04:ae
        hardwareProfile: default
      - name: master-1
        role: master
        bmc:
          address: ipmi://192.168.0.11:6242
          username: admin
          password: password
        bootMACAddress: 52:54:00:0f:91:f3
        hardwareProfile: default
      - name: master-2
        role: master
        bmc:
          address: ipmi://192.168.0.11:6243
          username: admin
          password: password
        bootMACAddress: 52:54:00:ee:d2:f2
        hardwareProfile: default
sshKey: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDP+5QkRCiuhsYItXj7DzLcOIs2RbCgpMzDtPlt/hfLnDkLGozYIFapMp+o4l+6ornbZ3L+hYE0T8SyvyYVWfm1XpPcVgUIW6qp7yfEyTSRhpGnoY74PD33FIf6BtU2HoFLWjQcE6OrQOF0wijI3fgL0jSzvAxvYoXU/huMx/kI2jBcWEq5cADRfvpeYXhVEJLrIIOepoAZE1syaPT7jQEoLDfvxrDZPKObCOI2vzLiAQXI7gK1uc9YDb6IEA/4Ik4eV2R1+VCgKhgk5RUqn69+8a1o783g1tChKuLwA4K9lyEAbFBwlHMctfNOLeC1w+bYpDXH/3GydcYfq79/18dVd+xEUlzzC+2/qycWG36C1MxUZa2fXvSRWLnpkLcxtIes4MikFeIr3jkJlFUzITigzvFrKa2IKaJzQ53WsE++LVnKJfcFNLtWfdEOZMowG/KtgzSSac/iVEJRM2YTIJsQsqhhI4PTrqVlUy/NwcXOFfUF/NkF2deeUZ21Cdn+bKZDKtFu2x+ujyAWZKNq570YaFT3a4TrL6WmE9kdHnJOXYR61Tiq/1fU+y0fv1d0f1cYr4+mNRCGIZoQOgJraF7/YluLB23INkJgtbah/0t1xzSsQ59gzFhRlLkW9gQDekj2tOGJmZIuYCnTXGiqXHnri2yAPexgRiaFjoM3GCpsWw== bschmaus@lap1.schmaustech.com'
imageContentSources:
- mirrors:
  - rhel8-ocp-auto.schmaustech.com:5000/ocp4/openshift4
  source: registry.svc.ci.openshift.org/ocp/RELEASEVERSION
- mirrors:
  - rhel8-ocp-auto.schmaustech.com:5000/ocp4/openshift4
  source: registry.svc.ci.openshift.org/ocp/release
pullSecret: 'PULL SECRET HERE'

Lets also create a pull-secret.json file that has the OpenShift pull-secret credentials so we can access the Openshift repository (Note: PULL-SECRET-JSON needs to be the actual pull-secret one gets from OpenShift):

# echo 'PULL-SECRET-JSON' > /home/bschmaus/pull-secret-json

Now that we have the initial install-config.yaml that we will use to deploy the OpenShift cluster lets change gears and configure the local image repository on the provisioning node.   The steps below outline what is needed to configure the repository:

# sudo yum -y install podman httpd httpd-tools
# sudo mkdir -p /opt/registry/{auth,certs,data}
# sudo openssl req -newkey rsa:4096 -nodes -sha256 -keyout /opt/registry/certs/domain.key -x509 -days 365 -out /opt/registry/certs/domain.crt -subj "/C=US/ST=Minnesota/L=Brooklyn Park/O=Red Hat/OU=Engineering/CN=rhel8-ocp-auto.schmaustech.com"
# sudo cp /opt/registry/certs/domain.crt /etc/pki/ca-trust/source/anchors/
# sudo update-ca-trust extract
# sudo htpasswd -bBc /opt/registry/auth/htpasswd dummy dummy
# sudo firewall-cmd --add-port=5000/tcp --zone=libvirt  --permanent
# sudo firewall-cmd --add-port=5000/tcp --zone=public   --permanent
# sudo firewall-cmd --add-service=http  --permanent
# sudo firewall-cmd --reload
# sudo podman create --name poc-registry -p 5000:5000 -v /opt/registry/data:/var/lib/registry:z -v /opt/registry/auth:/auth:z -e "REGISTRY_AUTH=htpasswd" -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry" -e "REGISTRY_HTTP_SECRET=ALongRandomSecretForRegistry" -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd -v /opt/registry/certs:/certs:z -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key docker.io/library/registry:2
# sudo podman start poc-registry

Test that the repository is working with the following test:

# curl -u dummy:dummy -k https://rhel8-ocp-auto.schmaustech.com:5000/v2/_catalog


Update Install-Config.yaml Pull Secret & Certificate:

Now we need to update the pull-secret in our install-config.yaml file to reference the local repository credentials we used above.  To do this we first need to capture the base64 output for the user/password we configured for the local mirror which was dummy and dummy:

# echo -n 'dummy:dummy' | base64 -w0
ZHVtbXk6ZHVtbXk=

With the output above place it into a string like the sample below making sure to update the repository hostname as well to match the environment.  Save the contents to a file called local_pull_secret:

# cat << 'EOF' > /home/bschmaus/local_pull_secret
pullSecret: '{ "auths": { "rhel8-ocp-auto.schmaustech.com:5000": {"auth": "ZHVtbXk6ZHVtbXk=","email": "bschmaus@redhat.com"} } }'
EOF

Now lets inject that local_pull_secret into our install-config.yaml file:

# sed  -i '/^pullSecret/d' /home/bschmaus/install-config.yaml
# cat /home/bschmaus/local_pull_secret >> /home/bschmaus/install-config.yaml

Since we also created a cert above and our install-config.yaml file will need to connect to the local repository lets go ahead and add that cert to the install-config.yaml:

# sudo cp /opt/registry/certs/domain.crt /home/bschmaus/domain.crt
# sed -i -e 's/^/  /' /home/bschmaus/domain.crt
# echo "additionalTrustBundle: |" >> /home/bschmaus/install-config.yaml
# cat /home/bschmaus/domain.crt >> /home/bschmaus/install-config.yaml

Once the install-config.yaml file is updated with both the local repository pull-secret and domain cert our install-config.yaml will look something like the following:

apiVersion: v1
baseDomain: schmaustech.com
metadata:
  name: kni5
networking:
  machineCIDR: 192.168.0.0/24
compute:
- name: worker
  replicas: 0
controlPlane:
  name: master
  replicas: 3
  platform:
    baremetal: {}
platform:
  baremetal:
    apiVIP: 192.168.0.199
    ingressVIP: 192.168.0.197
    dnsVIP: 192.168.0.198
    hosts:
      - name: master-0
        role: master
        bmc:
          address: ipmi://192.168.0.11:6241
          username: admin
          password: password
        bootMACAddress: 52:54:00:3d:04:ae
        hardwareProfile: default
      - name: master-1
        role: master
        bmc:
          address: ipmi://192.168.0.11:6242
          username: admin
          password: password
        bootMACAddress: 52:54:00:0f:91:f3
        hardwareProfile: default
      - name: master-2
        role: master
        bmc:
          address: ipmi://192.168.0.11:6243
          username: admin
          password: password
        bootMACAddress: 52:54:00:ee:d2:f2
        hardwareProfile: default
sshKey: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDP+5QkRCiuhsYItXj7DzLcOIs2RbCgpMzDtPlt/hfLnDkLGozYIFapMp+o4l+6ornbZ3L+hYE0T8SyvyYVWfm1XpPcVgUIW6qp7yfEyTSRhpGnoY74PD33FIf6BtU2HoFLWjQcE6OrQOF0wijI3fgL0jSzvAxvYoXU/huMx/kI2jBcWEq5cADRfvpeYXhVEJLrIIOepoAZE1syaPT7jQEoLDfvxrDZPKObCOI2vzLiAQXI7gK1uc9YDb6IEA/4Ik4eV2R1+VCgKhgk5RUqn69+8a1o783g1tChKuLwA4K9lyEAbFBwlHMctfNOLeC1w+bYpDXH/3GydcYfq79/18dVd+xEUlzzC+2/qycWG36C1MxUZa2fXvSRWLnpkLcxtIes4MikFeIr3jkJlFUzITigzvFrKa2IKaJzQ53WsE++LVnKJfcFNLtWfdEOZMowG/KtgzSSac/iVEJRM2YTIJsQsqhhI4PTrqVlUy/NwcXOFfUF/NkF2deeUZ21Cdn+bKZDKtFu2x+ujyAWZKNq570YaFT3a4TrL6WmE9kdHnJOXYR61Tiq/1fU+y0fv1d0f1cYr4+mNRCGIZoQOgJraF7/YluLB23INkJgtbah/0t1xzSsQ59gzFhRlLkW9gQDekj2tOGJmZIuYCnTXGiqXHnri2yAPexgRiaFjoM3GCpsWw== bschmaus@bschmaus.remote.csb'
imageContentSources:
- mirrors:
  - rhel8-ocp-auto.schmaustech.com:5000/ocp4/openshift4
  source: registry.svc.ci.openshift.org/ocp/RELEASEVERSION
- mirrors:
  - rhel8-ocp-auto.schmaustech.com:5000/ocp4/openshift4
  source: registry.svc.ci.openshift.org/ocp/release
pullSecret: '{ "auths": { "rhel8-ocp-auto.schmaustech.com:5000": {"auth": "ZHVtbXk6ZHVtbXk=","email": "bschmaus@redhat.com"} } }'
additionalTrustBundle: |
  -----BEGIN CERTIFICATE-----
  MIIF9zCCA9+gAwIBAgIUJhBYhR40iyQOEWifRhKAjwupm4gwDQYJKoZIhvcNAQEL
  BQAwgYoxCzAJBgNVBAYTAlVTMRIwEAYDVQQIDAlNaW5uZXNvdGExFjAUBgNVBAcM
  DUJyb29rbHluIFBhcmsxEDAOBgNVBAoMB1JlZCBIYXQxFDASBgNVBAsMC0VuZ2lu
  ZWVyaW5nMScwJQYDVQQDDB5yaGVsOC1vY3AtYXV0by5zY2htYXVzdGVjaC5jb20w
  HhcNMTkxMTE3MjEwNjEzWhcNMjAxMTE2MjEwNjEzWjCBijELMAkGA1UEBhMCVVMx
  EjAQBgNVBAgMCU1pbm5lc290YTEWMBQGA1UEBwwNQnJvb2tseW4gUGFyazEQMA4G
  A1UECgwHUmVkIEhhdDEUMBIGA1UECwwLRW5naW5lZXJpbmcxJzAlBgNVBAMMHnJo
  ZWw4LW9jcC1hdXRvLnNjaG1hdXN0ZWNoLmNvbTCCAiIwDQYJKoZIhvcNAQEBBQAD
  ggIPADCCAgoCggIBAMiiLRYITGwaOXext9cTOpemfwBjfKx0eREEpqww9nz5qRPn
  a0NK9Q5zgpieQvGWAN8y9c6wK0aUS4SRcFSx8RVbjP2L4p3ii0UsH6xP6JqHtfIg
  ynbqco6CyfSq7k+GwmoMLFx0Tki3Ta6syVQ9pN/YctTPIeIBMOXkSeOxNkn8dtdG
  oiCXS2OB0cW+wf3INp48Cc2zbbc+QmMw/LQlJIZcrP/C/Luh+fGiOt5XbrFD2Ain
  FYkKHOEzGucuEHDC7f/wLqLkwUf6iN4aB0fVU7kdYn1C1aZJhmlY5tSzR3eRvNsL
  QgFPfkHI647OTbDA8R4VNRRgkza+dEgEvLUbAyeQ7JztcFfnncWlDuvZdj6MCstR
  MND4lw3Ig0AK9PYXb5ui5g57ms7tInmqVJKb/9Xp9DKSWEUEOe+TQBeZ6spWmQjg
  GcPwuX5g+9RHh4iYmClxclDrZ6k6o6NQY9ldXQeI6kNVHacvg2S7iaWXFrshDtk4
  ss5Lvlag4/+yLJUIiWeIyG8yBkDlH7d+OV8F9rhf86J7tgbcA+pxms0v11Ot0I4n
  GibtZDmU0SPpsdAQzv798jhRiwJJMjh7Gw3EnnwxC/3GIK0LqqcKxMd3VmLRQYtb
  VFKYJyK6LBml/iONAmCIcaLMc7CcoaalM7pmWAdyis3FFdIsYDFOQP7pWPHlAgMB
  AAGjUzBRMB0GA1UdDgQWBBSVX7d9U2nc5LgiF7CFYABdTvEn8DAfBgNVHSMEGDAW
  gBSVX7d9U2nc5LgiF7CFYABdTvEn8DAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3
  DQEBCwUAA4ICAQCOzQ5oPbfAAinjhGRXuBPjlkm0bfKxW3sd+ExFOopVRYlCepzg
  CQ3iiMkD2ygI6Ga1B8NhQ1kHkvZ9vISAG3R1Ew65Od/qoZ0vuCUlo9fG761Yp1i6
  w/hR1MmcE0PL2EkyVz+GwNYjq445YD0aI3m3El9VPKoJPB6VHCBmII0qcCCTpag2
  Uq77vzk4M8HYV/1lqRKH601pkaZOUul7KuS2a4GzNUNr90Fps1I0s45CE9NpyS0s
  6cDd3RITlTHoNRBTeXWy5I5lWTqeJx5lwiGc+lkXQYCY/sVOtATwTSp6EqzvS08r
  Q/4sZDxmcmGwig+afBa8Of71ndhzd5MxJHyOrhsfsGfwH6ThB8SaCliyvC/160MT
  3LuXLn6OzGLcoLMcOhlJhhy7H7DnJeellVlSc/FHVr17IKInBd7viF4Sw5NGjH/d
  q5peosB2tDkaGgOtgIMsuA7aYrilV+3ZZ3nx1Yipwju+9hU4ncCcO16OGC/bgRym
  Bg6W8b9HZ+v1dvmh7aYHKDdZCXcNX6W/bWVC/rBpo4Cq+0jJso77CQYj95EdooSQ
  kqc6bj8BpHfxU6o6nZ1Aqtfw17yPeqh6sfByn1yfLuhPBUGuU0mZQAmFGkwSP+HH
  ZqLMKxvpJe5ufLz97O8gjCh38XV6mt3VUfVEn4Yrx5M2RtEMeuEtbYMCvw==
  -----END CERTIFICATE-----

Mirror Images:

Now that we have configured all the services and configuration files we need for our OpenShift IPI deployment we can now begin the process of mirroring the images to the local respository:

# LATEST_CI_IMAGE=$(curl https://openshift-release.svc.ci.openshift.org/api/v1/releasestream/4.3.0-0.ci/latest | grep -o 'registry.svc.ci.openshift.org[^"]\+')
# export OPENSHIFT_RELEASE_IMAGE="${OPENSHIFT_RELEASE_IMAGE:-$LATEST_CI_IMAGE}"
# export GOPATH=/home/bschmaus/go
# export OCP_RELEASE=`echo $LATEST_CI_IMAGE|cut -d: -f2`
# export UPSTREAM_REPO=$LATEST_CI_IMAGE
# export LOCAL_REG='rhel8-ocp-auto.schmaustech.com:5000'
# export LOCAL_REPO='ocp4/openshift4'
# export LOCAL_SECRET_JSON="${HOME}/pull-secret.json"
# export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=${LOCAL_REG}/${LOCAL_REPO}:${OCP_RELEASE}
# /usr/local/bin/oc adm release mirror -a $LOCAL_SECRET_JSON --from=$UPSTREAM_REPO --to-release-image=$LOCAL_REG/$LOCAL_REPO:$OCP_RELEASE --to=$LOCAL_REG/$LOCAL_REPO

Adjust ImageContentSources in Install-Config.yaml:

Recall we original defined the install-config.yaml with a line that had RELEASEVERSION in all caps.   Since we declared the OCP_RELEASE variable in the previous steps, we are now ready to update the install-config.yaml and replace RELEASEVERSION with the proper version defined for our local repository.  The following steps initiate that change:

# NEW_RELEASE=`echo $OCP_RELEASE|sed s/.0-0.ci//g`
# sed -i s/RELEASEVERSION/$NEW_RELEASE/g /home/bschmaus/install-config.yaml

Deploying the OpenShift Cluster:

Finally after all of the steps above, we can begin the actual deployment using the commands below:


# mkdir /home/bschmaus/ocp
# cp /home/bschmaus/install-config.yaml /home/bschmaus/ocp 
# /usr/local/bin/openshift-baremetal-install --dir /home/bschmaus/ocp --log-level debug create cluster

If all the steps were followed, the cluster should successfully deploy with the exception of Metal3 container as there is an issue with this that needs to be addressed.

We can check if the cluster nodes are online with the following:

# export KUBECONFIG=/home/bschmaus/ocp/auth/kubeconfig
# oc get nodes
NAME                            STATUS   ROLES           AGE   VERSION
master-0.kni5.schmaustech.com   Ready    master,worker   10h   v1.16.2
master-1.kni5.schmaustech.com   Ready    master,worker   10h   v1.16.2
master-2.kni5.schmaustech.com   Ready    master,worker   10h   v1.16.2

We can also view all the pods to validate things are online as well:

# oc get pods --all-namespaces
NAMESPACE                                               NAME                                                              READY   STATUS                            RESTARTS   AGE
openshift-apiserver-operator                            openshift-apiserver-operator-589544b58f-hpzdj                     1/1     Running                           2          10h
openshift-apiserver                                     apiserver-gq5wv                                                   1/1     Running                           0          9h
openshift-apiserver                                     apiserver-gsds9                                                   1/1     Running                           0          9h
openshift-apiserver                                     apiserver-kqw6d                                                   1/1     Running                           0          9h
openshift-authentication-operator                       authentication-operator-58d65b5d94-7s225                          1/1     Running                           0          9h
openshift-authentication                                oauth-openshift-58c95b9459-jvnx5                                  1/1     Running                           0          9h
openshift-authentication                                oauth-openshift-58c95b9459-z9cbp                                  1/1     Running                           0          9h
openshift-cloud-credential-operator                     cloud-credential-operator-8c9748878-55n7k                         1/1     Running                           3          10h
openshift-cluster-machine-approver                      machine-approver-6485cf466b-m6r9m                                 2/2     Running                           0          10h
openshift-cluster-node-tuning-operator                  cluster-node-tuning-operator-7668d5c85c-lt8vh                     1/1     Running                           0          9h
openshift-cluster-node-tuning-operator                  tuned-fssr9                                                       1/1     Running                           0          9h
openshift-cluster-node-tuning-operator                  tuned-nzq4q                                                       1/1     Running                           0          9h
openshift-cluster-node-tuning-operator                  tuned-znl95                                                       1/1     Running                           0          9h
openshift-cluster-samples-operator                      cluster-samples-operator-66fd64c57b-swnvx                         2/2     Running                           0          9h
openshift-cluster-storage-operator                      cluster-storage-operator-698c8fc449-hzbqp                         1/1     Running                           0          9h
openshift-cluster-version                               cluster-version-operator-7449dc5b9c-2kcb8                         1/1     Running                           0          10h
openshift-console-operator                              console-operator-67bdf96b5b-lgzj4                                 1/1     Running                           0          9h
openshift-console                                       console-6df4667b8c-4bw9l                                          1/1     Running                           0          9h
openshift-console                                       console-6df4667b8c-m2pl7                                          1/1     Running                           1          9h
openshift-console                                       downloads-65fdcc888-29t6m                                         1/1     Running                           0          9h
openshift-console                                       downloads-65fdcc888-vh6tj                                         1/1     Running                           0          9h
openshift-controller-manager-operator                   openshift-controller-manager-operator-69bb4c6545-m9hf4            1/1     Running                           2          10h
openshift-controller-manager                            controller-manager-drrvv                                          1/1     Running                           0          9h
openshift-controller-manager                            controller-manager-fck8g                                          1/1     Running                           0          9h
openshift-controller-manager                            controller-manager-gnj6b                                          1/1     Running                           0          9h
openshift-dns-operator                                  dns-operator-54d6dbb59b-wrjtl                                     1/1     Running                           0          10h
openshift-dns                                           dns-default-dl2lq                                                 2/2     Running                           0          9h
openshift-dns                                           dns-default-vs8xd                                                 2/2     Running                           0          9h
openshift-dns                                           dns-default-wn4px                                                 2/2     Running                           0          9h
openshift-etcd                                          etcd-member-master-0.kni5.schmaustech.com                         2/2     Running                           0          10h
openshift-etcd                                          etcd-member-master-1.kni5.schmaustech.com                         2/2     Running                           0          10h
openshift-etcd                                          etcd-member-master-2.kni5.schmaustech.com                         2/2     Running                           0          10h
openshift-image-registry                                cluster-image-registry-operator-788f556d9d-l9hrh                  2/2     Running                           0          9h
openshift-ingress-operator                              ingress-operator-6f8d45d96f-4kw7x                                 1/1     Running                           0          9h
openshift-ingress                                       router-default-5675955655-4wqw4                                   1/1     Running                           0          9h
openshift-ingress                                       router-default-5675955655-rvjmq                                   1/1     Running                           0          9h
openshift-insights                                      insights-operator-69b4497995-ltggd                                1/1     Running                           3          10h
openshift-kni-infra                                     coredns-master-0.kni5.schmaustech.com                             1/1     Running                           0          10h
openshift-kni-infra                                     coredns-master-1.kni5.schmaustech.com                             1/1     Running                           0          10h
openshift-kni-infra                                     coredns-master-2.kni5.schmaustech.com                             1/1     Running                           0          10h
openshift-kni-infra                                     haproxy-master-0.kni5.schmaustech.com                             2/2     Running                           2          10h
openshift-kni-infra                                     haproxy-master-1.kni5.schmaustech.com                             2/2     Running                           2          10h
openshift-kni-infra                                     haproxy-master-2.kni5.schmaustech.com                             2/2     Running                           2          10h
openshift-kni-infra                                     keepalived-master-0.kni5.schmaustech.com                          2/2     Running                           0          10h
openshift-kni-infra                                     keepalived-master-1.kni5.schmaustech.com                          2/2     Running                           0          10h
openshift-kni-infra                                     keepalived-master-2.kni5.schmaustech.com                          2/2     Running                           0          10h
openshift-kni-infra                                     mdns-publisher-master-0.kni5.schmaustech.com                      1/1     Running                           0          10h
openshift-kni-infra                                     mdns-publisher-master-1.kni5.schmaustech.com                      1/1     Running                           0          10h
openshift-kni-infra                                     mdns-publisher-master-2.kni5.schmaustech.com                      1/1     Running                           0          10h
openshift-kube-apiserver-operator                       kube-apiserver-operator-79cc8666bd-sbcbq                          1/1     Running                           2          10h
openshift-kube-apiserver                                installer-2-master-0.kni5.schmaustech.com                         0/1     Completed                         0          9h
openshift-kube-apiserver                                installer-2-master-1.kni5.schmaustech.com                         0/1     Completed                         0          9h
openshift-kube-apiserver                                installer-2-master-2.kni5.schmaustech.com                         0/1     Completed                         0          9h
openshift-kube-apiserver                                installer-3-master-1.kni5.schmaustech.com                         0/1     Completed                         0          9h
openshift-kube-apiserver                                installer-5-master-0.kni5.schmaustech.com                         0/1     Completed                         0          9h
openshift-kube-apiserver                                installer-5-master-1.kni5.schmaustech.com                         0/1     Completed                         0          9h
openshift-kube-apiserver                                installer-5-master-2.kni5.schmaustech.com                         0/1     Completed                         0          9h
openshift-kube-apiserver                                kube-apiserver-master-0.kni5.schmaustech.com                      3/3     Running                           0          9h
openshift-kube-apiserver                                kube-apiserver-master-1.kni5.schmaustech.com                      3/3     Running                           0          9h
openshift-kube-apiserver                                kube-apiserver-master-2.kni5.schmaustech.com                      3/3     Running                           1          9h
openshift-kube-apiserver                                revision-pruner-2-master-0.kni5.schmaustech.com                   0/1     Completed                         0          9h
openshift-kube-apiserver                                revision-pruner-2-master-1.kni5.schmaustech.com                   0/1     Completed                         0          9h
openshift-kube-apiserver                                revision-pruner-2-master-2.kni5.schmaustech.com                   0/1     Completed                         0          9h
openshift-kube-apiserver                                revision-pruner-3-master-1.kni5.schmaustech.com                   0/1     Completed                         0          9h
openshift-kube-apiserver                                revision-pruner-5-master-0.kni5.schmaustech.com                   0/1     Completed                         0          9h
openshift-kube-apiserver                                revision-pruner-5-master-1.kni5.schmaustech.com                   0/1     Completed                         0          9h
openshift-kube-apiserver                                revision-pruner-5-master-2.kni5.schmaustech.com                   0/1     Completed                         0          9h
openshift-kube-controller-manager-operator              kube-controller-manager-operator-d46bf7586-ctjcd                  1/1     Running                           2          10h
openshift-kube-controller-manager                       installer-2-master-0.kni5.schmaustech.com                         0/1     Completed                         0          9h
openshift-kube-controller-manager                       installer-3-master-0.kni5.schmaustech.com                         0/1     Completed                         0          9h
openshift-kube-controller-manager                       installer-3-master-1.kni5.schmaustech.com                         0/1     Completed                         0          9h
openshift-kube-controller-manager                       installer-3-master-2.kni5.schmaustech.com                         0/1     Completed                         0          9h
openshift-kube-controller-manager                       installer-4-master-0.kni5.schmaustech.com                         0/1     Completed                         0          9h
openshift-kube-controller-manager                       installer-4-master-1.kni5.schmaustech.com                         0/1     Completed                         0          9h
openshift-kube-controller-manager                       installer-4-master-2.kni5.schmaustech.com                         0/1     Completed                         0          9h
openshift-kube-controller-manager                       installer-5-master-0.kni5.schmaustech.com                         0/1     Completed                         0          9h
openshift-kube-controller-manager                       installer-5-master-1.kni5.schmaustech.com                         0/1     Completed                         0          9h
openshift-kube-controller-manager                       installer-5-master-2.kni5.schmaustech.com                         0/1     Completed                         0          9h
openshift-kube-controller-manager                       kube-controller-manager-master-0.kni5.schmaustech.com             3/3     Running                           0          9h
openshift-kube-controller-manager                       kube-controller-manager-master-1.kni5.schmaustech.com             3/3     Running                           1          9h
openshift-kube-controller-manager                       kube-controller-manager-master-2.kni5.schmaustech.com             3/3     Running                           1          9h
openshift-kube-controller-manager                       revision-pruner-2-master-0.kni5.schmaustech.com                   0/1     Completed                         0          9h
openshift-kube-controller-manager                       revision-pruner-3-master-0.kni5.schmaustech.com                   0/1     Completed                         0          9h
openshift-kube-controller-manager                       revision-pruner-3-master-1.kni5.schmaustech.com                   0/1     Completed                         0          9h
openshift-kube-controller-manager                       revision-pruner-3-master-2.kni5.schmaustech.com                   0/1     Completed                         0          9h
openshift-kube-controller-manager                       revision-pruner-4-master-0.kni5.schmaustech.com                   0/1     Completed                         0          9h
openshift-kube-controller-manager                       revision-pruner-4-master-1.kni5.schmaustech.com                   0/1     Completed                         0          9h
openshift-kube-controller-manager                       revision-pruner-4-master-2.kni5.schmaustech.com                   0/1     Completed                         0          9h
openshift-kube-controller-manager                       revision-pruner-5-master-0.kni5.schmaustech.com                   0/1     Completed                         0          9h
openshift-kube-controller-manager                       revision-pruner-5-master-1.kni5.schmaustech.com                   0/1     Completed                         0          9h
openshift-kube-controller-manager                       revision-pruner-5-master-2.kni5.schmaustech.com                   0/1     Completed                         0          9h
openshift-kube-scheduler-operator                       openshift-kube-scheduler-operator-6ff9678df8-8xfnv                1/1     Running                           2          10h
openshift-kube-scheduler                                installer-2-master-0.kni5.schmaustech.com                         0/1     Completed                         0          9h
openshift-kube-scheduler                                installer-3-master-1.kni5.schmaustech.com                         0/1     Completed                         0          9h
openshift-kube-scheduler                                installer-5-master-0.kni5.schmaustech.com                         0/1     Completed                         0          9h
openshift-kube-scheduler                                installer-5-master-1.kni5.schmaustech.com                         0/1     Completed                         0          9h
openshift-kube-scheduler                                installer-5-master-2.kni5.schmaustech.com                         0/1     Completed                         0          9h
openshift-kube-scheduler                                installer-6-master-0.kni5.schmaustech.com                         0/1     Completed                         0          9h
openshift-kube-scheduler                                installer-6-master-1.kni5.schmaustech.com                         0/1     Completed                         0          9h
openshift-kube-scheduler                                installer-6-master-2.kni5.schmaustech.com                         0/1     Completed                         0          9h
openshift-kube-scheduler                                openshift-kube-scheduler-master-0.kni5.schmaustech.com            1/1     Running                           1          9h
openshift-kube-scheduler                                openshift-kube-scheduler-master-1.kni5.schmaustech.com            1/1     Running                           0          9h
openshift-kube-scheduler                                openshift-kube-scheduler-master-2.kni5.schmaustech.com            1/1     Running                           1          9h
openshift-kube-scheduler                                revision-pruner-2-master-0.kni5.schmaustech.com                   0/1     Completed                         0          9h
openshift-kube-scheduler                                revision-pruner-3-master-1.kni5.schmaustech.com                   0/1     Completed                         0          9h
openshift-kube-scheduler                                revision-pruner-5-master-0.kni5.schmaustech.com                   0/1     Completed                         0          9h
openshift-kube-scheduler                                revision-pruner-5-master-1.kni5.schmaustech.com                   0/1     Completed                         0          9h
openshift-kube-scheduler                                revision-pruner-5-master-2.kni5.schmaustech.com                   0/1     Completed                         0          9h
openshift-kube-scheduler                                revision-pruner-6-master-0.kni5.schmaustech.com                   0/1     Completed                         0          9h
openshift-kube-scheduler                                revision-pruner-6-master-1.kni5.schmaustech.com                   0/1     Completed                         0          9h
openshift-kube-scheduler                                revision-pruner-6-master-2.kni5.schmaustech.com                   0/1     Completed                         0          9h
openshift-machine-api                                   cluster-autoscaler-operator-7f977b7c45-975p5                      2/2     Running                           0          9h
openshift-machine-api                                   machine-api-controllers-556b5ffc85-nvm9k                          4/4     Running                           0          9h
openshift-machine-api                                   machine-api-operator-8589cc9889-wp8k6                             2/2     Running                           1          10h
openshift-machine-api                                   metal3-68b9cf8bf-kmfw2                                            0/8     Init:CreateContainerConfigError   0          9h
openshift-machine-config-operator                       etcd-quorum-guard-6f6574bc9b-czhjz                                1/1     Running                           0          9h
openshift-machine-config-operator                       etcd-quorum-guard-6f6574bc9b-gd5ms                                1/1     Running                           0          9h
openshift-machine-config-operator                       etcd-quorum-guard-6f6574bc9b-zzjnb                                1/1     Running                           0          9h
openshift-machine-config-operator                       machine-config-controller-9d8c59488-dm74p                         1/1     Running                           1          9h
openshift-machine-config-operator                       machine-config-daemon-224vq                                       2/2     Running                           0          9h
openshift-machine-config-operator                       machine-config-daemon-hxhbc                                       2/2     Running                           0          9h
openshift-machine-config-operator                       machine-config-daemon-ngxbt                                       2/2     Running                           0          9h
openshift-machine-config-operator                       machine-config-operator-6cbbd79995-6b277                          1/1     Running                           1          10h
openshift-machine-config-operator                       machine-config-server-lswt8                                       1/1     Running                           0          9h
openshift-machine-config-operator                       machine-config-server-mrvln                                       1/1     Running                           0          9h
openshift-machine-config-operator                       machine-config-server-nshlv                                       1/1     Running                           0          9h
openshift-marketplace                                   certified-operators-7c44559446-w4bl2                              1/1     Running                           0          9h
openshift-marketplace                                   community-operators-64bfb7b678-7gs7q                              1/1     Running                           0          9h
openshift-marketplace                                   marketplace-operator-7776f6c9ff-mxls8                             1/1     Running                           0          9h
openshift-marketplace                                   redhat-operators-864d755755-49vh2                                 1/1     Running                           0          9h
openshift-monitoring                                    alertmanager-main-0                                               3/3     Running                           0          9h
openshift-monitoring                                    alertmanager-main-1                                               3/3     Running                           0          9h
openshift-monitoring                                    alertmanager-main-2                                               3/3     Running                           0          9h
openshift-monitoring                                    cluster-monitoring-operator-7c66dc45b4-8x6kl                      1/1     Running                           0          9h
openshift-monitoring                                    grafana-668586776b-dnfj7                                          2/2     Running                           0          9h
openshift-monitoring                                    kube-state-metrics-75df8cfbdf-bgfjv                               3/3     Running                           0          9h
openshift-monitoring                                    node-exporter-7b4s9                                               2/2     Running                           0          9h
openshift-monitoring                                    node-exporter-n459c                                               2/2     Running                           0          9h
openshift-monitoring                                    node-exporter-vvfh2                                               2/2     Running                           0          9h
openshift-monitoring                                    openshift-state-metrics-5b995c8497-2sv4m                          3/3     Running                           0          9h
openshift-monitoring                                    prometheus-adapter-bb778b866-cccw5                                1/1     Running                           0          9h
openshift-monitoring                                    prometheus-adapter-bb778b866-jxlwf                                1/1     Running                           0          9h
openshift-monitoring                                    prometheus-k8s-0                                                  7/7     Running                           1          9h
openshift-monitoring                                    prometheus-k8s-1                                                  7/7     Running                           1          9h
openshift-monitoring                                    prometheus-operator-6c4f54f97c-242ft                              1/1     Running                           0          9h
openshift-monitoring                                    thanos-querier-bf4f5dd76-bn8rb                                    4/4     Running                           0          9h
openshift-monitoring                                    thanos-querier-bf4f5dd76-lcs69                                    4/4     Running                           0          9h
openshift-multus                                        multus-admission-controller-899jw                                 1/1     Running                           1          9h
openshift-multus                                        multus-admission-controller-bwgcb                                 1/1     Running                           1          9h
openshift-multus                                        multus-admission-controller-s4rdl                                 1/1     Running                           1          9h
openshift-multus                                        multus-mzfjv                                                      1/1     Running                           0          10h
openshift-multus                                        multus-qxq9z                                                      1/1     Running                           0          10h
openshift-multus                                        multus-zfs6k                                                      1/1     Running                           0          10h
openshift-network-operator                              network-operator-55b786448c-fkt2l                                 1/1     Running                           0          10h
openshift-operator-lifecycle-manager                    catalog-operator-75b65486c-6xhwk                                  1/1     Running                           0          10h
openshift-operator-lifecycle-manager                    olm-operator-76cfbdc87f-cr4qm                                     1/1     Running                           0          10h
openshift-operator-lifecycle-manager                    packageserver-579759bb6d-44d84                                    1/1     Running                           1          9h
openshift-operator-lifecycle-manager                    packageserver-579759bb6d-x7srm                                    1/1     Running                           1          9h
openshift-sdn                                           ovs-8pmrz                                                         1/1     Running                           0          10h
openshift-sdn                                           ovs-npwxk                                                         1/1     Running                           0          10h
openshift-sdn                                           ovs-svmwk                                                         1/1     Running                           0          10h
openshift-sdn                                           sdn-controller-d8rd8                                              1/1     Running                           0          10h
openshift-sdn                                           sdn-controller-pjqxt                                              1/1     Running                           0          10h
openshift-sdn                                           sdn-controller-zr2kr                                              1/1     Running                           0          10h
openshift-sdn                                           sdn-q4w8d                                                         1/1     Running                           0          10h
openshift-sdn                                           sdn-v72dr                                                         1/1     Running                           0          10h
openshift-sdn                                           sdn-zjtc2                                                         1/1     Running                           0          10h
openshift-service-ca-operator                           service-ca-operator-748f8bff-r4g94                                1/1     Running                           1          10h
openshift-service-ca                                    apiservice-cabundle-injector-549875965f-2dlq9                     1/1     Running                           1          9h
openshift-service-ca                                    configmap-cabundle-injector-6c49fc5d79-k4mbg                      1/1     Running                           1          9h
openshift-service-ca                                    service-serving-cert-signer-5fc7c8df6f-rlp6v                      1/1     Running                           1          9h
openshift-service-catalog-apiserver-operator            openshift-service-catalog-apiserver-operator-c7c9d4494-rqmv5      1/1     Running                           0          9h
openshift-service-catalog-controller-manager-operator   openshift-service-catalog-controller-manager-operator-587bhqzmm   1/1     Running                           0          9h