Deploying a single node OpenShift via the Assisted Installer has made it very easy to stand up a one node cluster. However this means having nodes that have connectivity to the internet. But what if the environment is disconnected? In the following blog I will show how one can use the openshift-install binary to deploy a single node OpenShift that is in a disconnected environment without the assisted installer.
Before we begin lets cover what this blog already assumes exists as prerequisites:
- Podman, the oc binary and the openshift-install binary already exist on the system
- A disconnected registry is already configured and has the mirrored contents of the images for a given OpenShift release.
- A physical baremetal node with the ability to boot an ISO image
- DNS entries for basic baremetal IPI requirements exist. My environment is below:
master-0.kni20.schmaustech.com IN A 192.168.0.210*.apps.kni20.schmaustech.com IN A 192.168.0.210api.kni20.schmaustech.com IN A 192.168.0.210api-int.kni20.schmaustech.com IN A 192.168.0.210
First lets verify the version of OpenShift we will be deploying by looking at the output of the oc version and openshift-install version:
$ oc version Client Version: 4.8.12 $ ./openshift-install version ./openshift-install 4.8.12 built from commit 450e95767d89f809cb1afe5a142e9c824a269de8 release image quay.io/openshift-release-dev/ocp-release@sha256:c3af995af7ee85e88c43c943e0a64c7066d90e77fafdabc7b22a095e4ea3c25a
Looks like we will be deploying a version of 4.8.12. Ensure the disconnected registry being used has the images for 4.8.12 mirrored. If not use procedure like I have used in one of my previous blogs to mirror the 4.8.12 images.
Now lets pull down a few files we will need for our deployment iso. We need to pull down both the coreos-installer and the rhcos live iso:
$ wget https://mirror.openshift.com/pub/openshift-v4/clients/coreos-installer/v0.8.0-3/coreos-installer --2021-09-15 10:10:26-- https://mirror.openshift.com/pub/openshift-v4/clients/coreos-installer/v0.8.0-3/coreos-installer Resolving mirror.openshift.com (mirror.openshift.com)... 54.172.173.155, 54.173.18.88 Connecting to mirror.openshift.com (mirror.openshift.com)|54.172.173.155|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 7649968 (7.3M) Saving to: ‘coreos-installer’ coreos-installer 100%[=====================================================================================================================>] 7.29M 8.83MB/s in 0.8s 2021-09-15 10:10:27 (8.83 MB/s) - ‘coreos-installer’ saved [7649968/7649968] $ wget https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/latest/4.8.2/rhcos-4.8.2-x86_64-live.x86_64.iso --2021-09-15 10:10:40-- https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/latest/4.8.2/rhcos-4.8.2-x86_64-live.x86_64.iso Resolving mirror.openshift.com (mirror.openshift.com)... 54.172.173.155, 54.173.18.88 Connecting to mirror.openshift.com (mirror.openshift.com)|54.172.173.155|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 1031798784 (984M) [application/octet-stream] Saving to: ‘rhcos-4.8.2-x86_64-live.x86_64.iso’ rhcos-4.8.2-x86_64-live.x86_64.iso 100%[=====================================================================================================================>] 984.00M 11.2MB/s in 93s 2021-09-15 10:12:13 (10.6 MB/s) - ‘rhcos-4.8.2-x86_64-live.x86_64.iso’ saved [1031798784/1031798784]
Set the execution bit on the coreos-installer which is a utility to embed the ignition file we will generate:
$ chmod 755 coreos-installer
Lets go ahead now and create an install-config.yaml for our single node deployment. Notice some of the differences in this install-config.yaml. Specifically we have no worker nodes defined, one master node defined and then we have the BootstrapInPlace section which tells us to use the sda disk in the node. We also have our imageContentSources which tells the installer to use the registry mirror.
$ cat << EOF > install-config.yaml
apiVersion: v1beta4
baseDomain: schmaustech.com
metadata:
name: kni20
networking:
networkType: OVNKubernetes
machineCIDR: 192.168.0.0/24
compute:
- name: worker
replicas: 0
controlPlane:
name: master
replicas: 1
platform:
none: {}
BootstrapInPlace:
InstallationDisk: /dev/sda
pullSecret: '{ "auths": { "rhel8-ocp-auto.schmaustech.com:5000": {"auth": "ZHVtbXk6ZHVtbXk=","email": "bschmaus@schmaustech.com" } } }'
sshKey: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDP+5QkRCiuhsYItXj7DzLcOIs2RbCgpMzDtPlt/hfLnDkLGozYIFapMp+o4l+6ornbZ3L+hYE0T8SyvyYVWfm1XpPcVgUIW6qp7yfEyTSRhpGnoY74PD33FIf6BtU2HoFLWjQcE6OrQOF0wijI3fgL0jSzvAxvYoXU/huMx/kI2jBcWEq5cADRfvpeYXhVEJLrIIOepoAZE1syaPT7jQEoLDfvxrDZPKObCOI2vzLiAQXI7gK1uc9YDb6IEA/4Ik4eV2R1+VCgKhgk5RUqn69+8a1o783g1tChKuLwA4K9lyEAbFBwlHMctfNOLeC1w+bYpDXH/3GydcYfq79/18dVd+xEUlzzC+2/qycWG36C1MxUZa2fXvSRWLnpkLcxtIes4MikFeIr3jkJlFUzITigzvFrKa2IKaJzQ53WsE++LVnKJfcFNLtWfdEOZMowG/KtgzSSac/iVEJRM2YTIJsQsqhhI4PTrqVlUy/NwcXOFfUF/NkF2deeUZ21Cdn+bKZDKtFu2x+ujyAWZKNq570YaFT3a4TrL6WmE9kdHnJOXYR61Tiq/1fU+y0fv1d0f1cYr4+mNRCGIZoQOgJraF7/YluLB23INkJgtbah/0t1xzSsQ59gzFhRlLkW9gQDekj2tOGJmZIuYCnTXGiqXHnri2yAPexgRiaFjoM3GCpsWw== bschmaus@bschmaus.remote.csb'
imageContentSources:
- mirrors:
- rhel8-ocp-auto.schmaustech.com:5000/ocp4/openshift4
source: quay.io/openshift-release-dev/ocp-release
- mirrors:
- rhel8-ocp-auto.schmaustech.com:5000/ocp4/openshift4
source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
additionalTrustBundle: |
-----BEGIN CERTIFICATE-----
MIIF7zCCA9egAwIBAgIUeecEs+U5psgJ0aFgc4Q5dGVrAFcwDQYJKoZIhvcNAQEL
BQAwgYYxCzAJBgNVBAYTAlVTMRYwFAYDVQQIDA1Ob3J0aENhcm9saW5hMRAwDgYD
VQQHDAdSYWxlaWdoMRAwDgYDVQQKDAdSZWQgSGF0MRIwEAYDVQQLDAlNYXJrZXRp
bmcxJzAlBgNVBAMMHnJoZWw4LW9jcC1hdXRvLnNjaG1hdXN0ZWNoLmNvbTAeFw0y
MTA2MDkxMDM5MDZaFw0yMjA2MDkxMDM5MDZaMIGGMQswCQYDVQQGEwJVUzEWMBQG
A1UECAwNTm9ydGhDYXJvbGluYTEQMA4GA1UEBwwHUmFsZWlnaDEQMA4GA1UECgwH
UmVkIEhhdDESMBAGA1UECwwJTWFya2V0aW5nMScwJQYDVQQDDB5yaGVsOC1vY3At
YXV0by5zY2htYXVzdGVjaC5jb20wggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK
AoICAQC9exAg3Ie3N3mkrQKseyri1VP2IPTc+pUEiVCPisIQAhRUfHhPR1HT7EF7
SwaxrWjpfh9aYBPDEF3uLFQvzDEJWCh5PF55jwn3aABFGKEhfVBKd+es6nXnYaCS
8CgLS2qM9x4WiuZxrntfB16JrjP+CrTvlAbE4DIMlDQLgh8+hDw9VPlbzY+MI+WC
cYues1Ne+JZ5dZcKmCZ3zrVToPjreWZUuhSygci2xIQZxwWNmTvAgi+CAiQZS7VF
RmKjj2H/o/d3I+XSS2261I8aXCAw4/3vaM9aci0eHeEhLIMrhv86WycOjcYL1Z6R
n55diwDTSyrTo/B4zsQbmYUc8rP+pR2fyRJEGFVJ4ejcj2ZF5EbgUKupyU2gh/qt
QeYtJ+6uAr9S5iQIcq9qvD9nhAtm3DnBb065X4jVPl2YL4zsbOS1gjoa6dRbFuvu
f3SdsbQRF/YJWY/7j6cUaueCQOlXZRNhbQQHdIdBWFObw0QyyYtI831ue1MHPG0C
nsAriPOkRzBBq+BPmS9CqcRDGqh+nd9m9UPVDoBshwaziSqaIK2hvfCAVb3BPJES
CXKuIaP2IRzTjse58aAzsRW3W+4e/v9fwAOaE8nS7i+v8wrqcFgJ489HnVq+kRNc
VImv5dBKg2frzXs1PpnWkE4u2VJagKn9B2zva2miRQ+LyvLLDwIDAQABo1MwUTAd
BgNVHQ4EFgQUbcE9mpTkOK2ypIrURf+xYR08OAAwHwYDVR0jBBgwFoAUbcE9mpTk
OK2ypIrURf+xYR08OAAwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOC
AgEANTjx04NoiIyw9DyvszwRdrSGPO3dy1gk3jh+Du6Dpqqku3Mwr2ktaSCimeZS
4zY4S5mRCgZRwDKu19z0tMwbVDyzHPFJx+wqBpZKkD1FvOPKjKLewtiW2z8AP/kF
gl5UUNuwvGhOizazbvd1faQ8jMYoZKifM8On6IpFgqXCx98/GOWvnjn2t8YkMN3x
blKVm5N7eGy9LeiGRoiCJqcyfGqdAdg+Z+J94AHEZb3OxG8uHLrtmz0BF3A+8V2H
hofYI0spx5y9OcPin2yLm9DeCwWAA7maqdImBG/QpQCjcPW3Yzz9VytIMajPdnvd
vbJF5GZNj7ods1AykCCJjGy6n9WCf3a4VLnZWtUTbtz0nrIjJjsdlXZqby5BCF0G
iqWbg0j8onl6kmbMAhssRTlvL8w90F1IK3Hk+lz0Qy8rqZX2ohObtEYGMIAOdFJ1
iPQrbksXOBpZNtm1VAved41sYt1txS2WZQgfklIXOhNOu4r32ZGKas4EJml0l0wL
2P65PkPEa7AOeqwP0y6eGoNG9qFSl+yArycZGWudp88977H6CcdkdEcQzmjg5+TD
9GHm3drUYGqBJDvIemQaXfnwy9Gxx+oBDpXLXOuo+edK812zh/q7s2FELfH5ZieE
Q3dIH8UGsnjYxv8G3O23cYKZ1U0iiu9QvPRFm0F8JuFZqLQ=
-----END CERTIFICATE-----
EOF
Once we have the install-config.yaml created lets use the openshift-install binary to generate a singe node openshift ignition config:
$ ~/openshift-install --dir=./ create single-node-ignition-config INFO Consuming Install Config from target directory WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings INFO Single-Node-Ignition-Config created in: . and auth $ ls -lart total 1017468 -rwxr-xr-x. 1 bschmaus bschmaus 7649968 Apr 27 00:49 coreos-installer -rw-rw-r--. 1 bschmaus bschmaus 1031798784 Jul 22 13:10 rhcos-4.8.2-x86_64-live.x86_64.iso -rw-r--r--. 1 bschmaus bschmaus 3667 Sep 15 10:35 install-config.yaml.save drwx------. 27 bschmaus bschmaus 8192 Sep 15 10:39 .. drwxr-x---. 2 bschmaus bschmaus 50 Sep 15 10:45 auth -rw-r-----. 1 bschmaus bschmaus 284253 Sep 15 10:45 bootstrap-in-place-for-live-iso.ign -rw-r-----. 1 bschmaus bschmaus 1865601 Sep 15 10:45 .openshift_install_state.json -rw-rw-r--. 1 bschmaus bschmaus 213442 Sep 15 10:45 .openshift_install.log -rw-r-----. 1 bschmaus bschmaus 98 Sep 15 10:45 metadata.json drwxrwxr-x. 3 bschmaus bschmaus 247 Sep 15 10:45 .
Now lets take that bootstrap-in-place-for-live-iso.ign config we generated and use the coreos-installer to embed it into the rhcos live iso image. There will be no output upon completion so I usually echo the $? to confirm it ended with a good exit status.
$ ./coreos-installer iso ignition embed -fi bootstrap-in-place-for-live-iso.ign rhcos-4.8.2-x86_64-live.x86_64.iso $ echo $? 0
Since I am using a virtual machine as my single node openshift node I need to copy the boot iso over to my hypervisor host. If this were a real baremetal server like Dell one might mount the iso image via virtual media or as another method write the iso to a USB device and physically plug it into the node being used for this singe node deployment.
$ scp rhcos-4.8.2-x86_64-live.x86_64.iso root@192.168.0.20:/var/lib/libvirt/images/ root@192.168.0.20's password: rhcos-4.8.2-x86_64-live.x86_64.iso 100% 984MB 86.0MB/s 00:11
Once I have the live iso over on my hypervisor host I will use Virt-Manager to set the cdrom to boot from the live iso:
Next I will start the virtual machine. If using a physical host power on the node. The screen should be similar:
Once the virtual machine has booted we will see the console and login prompt. After a few minutes the machine will reboot.
If the ignition file was embedded without errors we should be able to login using the core user and associated key that was set in the install-config.yaml we used. Once inside the node we should be able to use crictl ps to confirm containers are being started:
$ ssh core@192.168.0.210 The authenticity of host '192.168.0.210 (192.168.0.210)' can't be established. ECDSA key fingerprint is SHA256:B24X/7PH3+kGWwmUKPc/E+2Rg3YYsmYHISCOHfbGthg. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '192.168.0.210' (ECDSA) to the list of known hosts. Red Hat Enterprise Linux CoreOS 48.84.202109100857-0 Part of OpenShift 4.8, RHCOS is a Kubernetes native operating system managed by the Machine Config Operator (`clusteroperator/machine-config`). WARNING: Direct SSH access to machines is not recommended; instead, make configuration changes via `machineconfig` objects: https://docs.openshift.com/container-platform/4.8/architecture/architecture-rhcos.html --- [core@master-0 ~]$ sudo crictl ps CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID a3792d71875ab aeee3c4eb8828bef375fa5f81bf524e84d12a0264c126b0f97703a3e5ebc06a8 17 seconds ago Running sbdb 0 4de60fd9cc622 733326d7246f8 dfd1e2430556eb4a9de83031a82c62c06debca6095dd63553ed38bd486374ac8 17 seconds ago Running kube-rbac-proxy 0 4de60fd9cc622 7df7efd52c7f9 de195e3670ad1b3dd892d5a289aa83ce12122001faf02a56facb8fa4720ceaa3 44 seconds ago Running kube-multus-additional-cni-plugins 0 aab58f11b1f0a ce602f830cb44 aeee3c4eb8828bef375fa5f81bf524e84d12a0264c126b0f97703a3e5ebc06a8 48 seconds ago Running ovnkube-node 0 f0fea8120b806 d17912e8c762d quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b7edfdb1dd3510c1a8d74144ae89fbe61a28f519781088ead1cb5e560158657 48 seconds ago Running kube-rbac-proxy 0 f0fea8120b806 f6cf9e739714e aeee3c4eb8828bef375fa5f81bf524e84d12a0264c126b0f97703a3e5ebc06a8 49 seconds ago Running ovn-acl-logging 0 f0fea8120b806 232e663c0b190 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03dc4dd87f6e52ad54718f31de9edfc763ce5a001d5bdff6c95fe85275fb64de 49 seconds ago Running northd 0 4de60fd9cc622 7b4b432b988d8 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03dc4dd87f6e52ad54718f31de9edfc763ce5a001d5bdff6c95fe85275fb64de 49 seconds ago Running ovn-controller 0 f0fea8120b806 5596f6644e1bb quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1fec937521df496277f7f934c079ebf48baccd8f76a5bfcc793e7c441976e6b5 About a minute ago Running kube-multus 0 7f4536275fb42 51b1c4da641f4 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70ffc0ed147222ab1bea6207af5415f11450c86a9de2979285ba1324f6e904c2 About a minute ago Running network-operator 0 ea0f3c0bb9567 b4b46f8f5de1c quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:66fa2d7a5b2be88b76b5a8fa6f330bc64b57ce0fa9b8ea29e96a4c77df90f7cd 2 minutes ago Running kube-apiserver-insecure-readyz 0 e3a4d81e4e99a e49ce4745cefd c7dbf8655b94a464b0aa15734fbd887bec8cdda46bbb3580954bf36961b4ac78 2 minutes ago Running kube-controller-manager 1 3cbc2d942afd8 7bd9f40dd40a3 c7dbf8655b94a464b0aa15734fbd887bec8cdda46bbb3580954bf36961b4ac78 2 minutes ago Running kube-apiserver 0 e3a4d81e4e99a e319800865018 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:80d0fcaf10fd289e31383062293cadb91ca6f7852a82f864c088679905f67859 2 minutes ago Running cluster-policy-controller 0 3cbc2d942afd8 d1e26854fc700 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e9de94a775df9cd6f86712410794393aa58f07374f294ba5a7b503f9fb23cf42 2 minutes ago Running kube-scheduler 0 0ae8507e3280a e95cef37125c4 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:622d9bb3fe4e540054f54ec260a7e3e4f16892260658dbe32ee4750c27a94158 2 minutes ago Running etcd 0 dcd694d4f9317 [core@master-0 ~]$
Further once we have confirmed containers are starting we can also use the kubeconfig and show the node state:
$ export KUBECONFIG=./auth/kubeconfig $ oc get nodes NAME STATUS ROLES AGE VERSION master-0.kni20.schmaustech.com Ready master,worker 21m v1.21.1+d8043e1
Now we can get the cluster operator states with the oc command to confirm when installation has completed. If there are still False's under AVAILABLE then the installation is still progressing:
$ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.8.12 False True False 17m baremetal 4.8.12 True False False 11m cloud-credential 4.8.12 True False False 3m37s cluster-autoscaler 4.8.12 True False False 11m config-operator 4.8.12 True False False 17m console 4.8.12 False True False 7m35s csi-snapshot-controller 4.8.12 True False False 7m56s dns 4.8.12 True False False 9m2s etcd 4.8.12 True False False 12m image-registry 4.8.12 True False False 7m48s ingress 4.8.12 True False False 8m53s insights 4.8.12 True False False 12m kube-apiserver 4.8.12 True True False 7m53s kube-controller-manager 4.8.12 True False False 10m kube-scheduler 4.8.12 True False False 11m kube-storage-version-migrator 4.8.12 True False False 17m machine-api 4.8.12 True False False 11m machine-approver 4.8.12 True False False 16m machine-config True marketplace 4.8.12 True False False 16m monitoring 4.8.12 True False False 6m18s network 4.8.12 True False False 17m node-tuning 4.8.12 True False False 11m openshift-apiserver 4.8.12 True False False 7m45s openshift-controller-manager 4.8.12 True False False 7m53s openshift-samples 4.8.12 True False False 8m operator-lifecycle-manager 4.8.12 True False False 17m operator-lifecycle-manager-catalog 4.8.12 True False False 12m operator-lifecycle-manager-packageserver 4.8.12 True False False 8m56s service-ca 4.8.12 True False False 17m storage 4.8.12 True False False 11m
Finally though after about 30 - 60 minutes we can finally see our single node cluster has completed installation:
$ oc get co NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE authentication 4.8.12 True False False 6m55s baremetal 4.8.12 True False False 19m cloud-credential 4.8.12 True False False 10m cluster-autoscaler 4.8.12 True False False 18m config-operator 4.8.12 True False False 24m console 4.8.12 True False False 7m1s csi-snapshot-controller 4.8.12 True False False 15m dns 4.8.12 True False False 16m etcd 4.8.12 True False False 19m image-registry 4.8.12 True False False 15m ingress 4.8.12 True False False 16m insights 4.8.12 True False False 19m kube-apiserver 4.8.12 True False False 15m kube-controller-manager 4.8.12 True False False 18m kube-scheduler 4.8.12 True False False 18m kube-storage-version-migrator 4.8.12 True False False 24m machine-api 4.8.12 True False False 19m machine-approver 4.8.12 True False False 24m machine-config 4.8.12 True False False 5m45s marketplace 4.8.12 True False False 24m monitoring 4.8.12 True False False 13m network 4.8.12 True False False 25m node-tuning 4.8.12 True False False 19m openshift-apiserver 4.8.12 True False False 15m openshift-controller-manager 4.8.12 True False False 15m openshift-samples 4.8.12 True False False 15m operator-lifecycle-manager 4.8.12 True False False 24m operator-lifecycle-manager-catalog 4.8.12 True False False 19m operator-lifecycle-manager-packageserver 4.8.12 True False False 16m service-ca 4.8.12 True False False 24m storage 4.8.12 True False False 19m
