Saturday, October 08, 2022

Deploy Microshift on Apple M1 Virtual Machine

I have been experimenting recently with the Apple Virtualization Framework in the Ventura MacOS Beta. I have written up a previous blog around using Red Hat Advanced Cluster Management for Kubernetes to deploy Single Node OpenShift on a virtual machine. There is also a video me installing Red Hat Enterprise Linux 9 on a similarly configured virtual machine on YouTube. Today however I want to explore installing Microshift on a similar configuration like I used in the previous write up and videos. After all Microshift is an optimized version of OpenShift & Kubernetes for small form factor and edge type environments. Let's explore what a basic installation looks like then in the remainder of this blog.

Lab Environment

The following lab environment was created in order to test this experiment, which includes the following:

  • MacBook Pro
    • M1 Max Processor
    • 32GB of memory
    • 1.8TB SSD
  • MacOS Ventura Beta 8
  • UTM Virtualization UI
  • 1 Virtual Machine using Apple Virtualization Framework
    • 8 cores of vCPU
    • 24GB memory
    • 120GB disk
    • Fedora 35 aarch64 installed
    • Static ip address configured

Now that we have a brief overview of the environment let't move onto installing Microshift.

Configure & Deploy Microshift

Installing Microshift is not that difficult to get up and running and takes just a few simple steps. To begin this process let's first log into the Fedora 35 host via ssh.

$ ssh bschmaus@10.0.0.25 The authenticity of host '10.0.0.25 (10.0.0.25)' can't be established. ECDSA key fingerprint is SHA256:OK9JNTWDGmYDnsb+ka4ynw91ihXGnsaa+Np8ExmR7is. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '10.0.0.25' (ECDSA) to the list of known hosts. bschmaus@10.0.0.25's password: Web console: https://m1:9090/ or https://10.0.0.25:9090/

Once we have established our ssh connection I want to execute a few commands to show this is indeed a Apple Virtualization Framework virtual machine running on an M1 process which has an Arm core. First we will run dmidecode to show the system information.

$ sudo dmidecode|more # dmidecode 3.3 Getting SMBIOS data from sysfs. SMBIOS 3.3.0 present. Table at 0x64BDE9000. Handle 0x0000, DMI type 1, 27 bytes System Information Manufacturer: Apple Inc. Product Name: Apple Virtualization Generic Platform Version: 1 Serial Number: Virtualization-c99bfd1d-3630-4c9c-815a-c4aff99a4e9b UUID: 1dfd9bc9-3036-9c4c-815a-c4aff99a4e9b Wake-up Type: Power Switch SKU Number: Not Specified Family: Not Specified (...)

Next let's display the Fedora version and the kernel version.

$ cat /etc/fedora-release Fedora release 35 (Thirty Five) $ uname -a Linux m1 5.14.10-300.fc35.aarch64 #1 SMP Thu Oct 7 20:32:40 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

With the confirmation of the system out of the way we can begin installing Microshift. The first step is to pin cri-o to the 1.21 version.

$ sudo dnf module enable -y cri-o:1.21 Fedora 35 - aarch64 11 MB/s | 75 MB 00:06 Fedora 35 openh264 (From Cisco) - aarch64 2.0 kB/s | 2.5 kB 00:01 Fedora Modular 35 - aarch64 1.3 MB/s | 3.2 MB 00:02 Fedora 35 - aarch64 - Updates 4.4 MB/s | 31 MB 00:07 Fedora Modular 35 - aarch64 - Updates 2.0 MB/s | 3.7 MB 00:01 Dependencies resolved. =================================================================================================================================================================================================================== Package Architecture Version Repository Size =================================================================================================================================================================================================================== Enabling module streams: cri-o 1.21 Transaction Summary =================================================================================================================================================================================================================== Complete!

With crio pinned to the version we need it at we can now install the cri-o and cri-tools to the virtual machine. Additional dependencies will also be pulled in during the process.

$ sudo dnf install -y cri-o cri-tools Last metadata expiration check: 0:03:11 ago on Sat 08 Oct 2022 09:04:19 AM CDT. Dependencies resolved. =================================================================================================================================================================================================================== Package Architecture Version Repository Size =================================================================================================================================================================================================================== Installing: cri-o aarch64 1.21.3-1.module_f35+13330+6bc9c749 updates-modular 21 M cri-tools aarch64 1.19.0-1.module_f35+12974+2bc66b5d updates-modular 5.5 M Installing dependencies: conmon aarch64 2:2.1.0-2.fc35 updates 53 k container-selinux noarch 2:2.169.0-1.fc35 fedora 50 k containernetworking-plugins aarch64 1.1.0-1.fc35 updates 7.9 M containers-common noarch 4:1-45.fc35 updates 76 k criu aarch64 3.16-2.fc35 fedora 511 k fuse-common aarch64 3.10.5-1.fc35 fedora 8.3 k fuse3 aarch64 3.10.5-1.fc35 fedora 54 k fuse3-libs aarch64 3.10.5-1.fc35 fedora 90 k libbsd aarch64 0.10.0-8.fc35 fedora 105 k libnet aarch64 1.2-4.fc35 fedora 60 k libslirp aarch64 4.6.1-2.fc35 fedora 72 k runc aarch64 2:1.1.3-1.fc35 updates 2.8 M socat aarch64 1.7.4.2-1.fc35 updates 300 k Installing weak dependencies: aardvark-dns aarch64 1.0.3-1.fc35 updates 1.0 M fuse-overlayfs aarch64 1.9-1.fc35 updates 67 k netavark aarch64 1.0.3-1.fc35 updates 2.0 M slirp4netns aarch64 1.1.12-2.fc35 fedora 56 k Transaction Summary =================================================================================================================================================================================================================== Install 19 Packages Total download size: 41 M Installed size: 212 M Downloading Packages: (1/19): fuse-common-3.10.5-1.fc35.aarch64.rpm 33 kB/s | 8.3 kB 00:00 (2/19): container-selinux-2.169.0-1.fc35.noarch.rpm 142 kB/s | 50 kB 00:00 (3/19): fuse3-3.10.5-1.fc35.aarch64.rpm 372 kB/s | 54 kB 00:00 (4/19): fuse3-libs-3.10.5-1.fc35.aarch64.rpm 704 kB/s | 90 kB 00:00 (5/19): criu-3.16-2.fc35.aarch64.rpm 959 kB/s | 511 kB 00:00 (6/19): libbsd-0.10.0-8.fc35.aarch64.rpm 700 kB/s | 105 kB 00:00 (7/19): libnet-1.2-4.fc35.aarch64.rpm 752 kB/s | 60 kB 00:00 (8/19): libslirp-4.6.1-2.fc35.aarch64.rpm 955 kB/s | 72 kB 00:00 (9/19): slirp4netns-1.1.12-2.fc35.aarch64.rpm 689 kB/s | 56 kB 00:00 (10/19): conmon-2.1.0-2.fc35.aarch64.rpm 101 kB/s | 53 kB 00:00 (11/19): containers-common-1-45.fc35.noarch.rpm 548 kB/s | 76 kB 00:00 (12/19): fuse-overlayfs-1.9-1.fc35.aarch64.rpm 163 kB/s | 67 kB 00:00 (13/19): aardvark-dns-1.0.3-1.fc35.aarch64.rpm 654 kB/s | 1.0 MB 00:01 (14/19): netavark-1.0.3-1.fc35.aarch64.rpm 449 kB/s | 2.0 MB 00:04 (15/19): socat-1.7.4.2-1.fc35.aarch64.rpm 542 kB/s | 300 kB 00:00 (16/19): runc-1.1.3-1.fc35.aarch64.rpm 522 kB/s | 2.8 MB 00:05 (17/19): containernetworking-plugins-1.1.0-1.fc35.aarch64.rpm 1.0 MB/s | 7.9 MB 00:08 (18/19): cri-tools-1.19.0-1.module_f35+12974+2bc66b5d.aarch64.rpm 2.0 MB/s | 5.5 MB 00:02 (19/19): cri-o-1.21.3-1.module_f35+13330+6bc9c749.aarch64.rpm 4.8 MB/s | 21 MB 00:04 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 3.3 MB/s | 41 MB 00:12 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Running scriptlet: container-selinux-2:2.169.0-1.fc35.noarch 1/19 Installing : container-selinux-2:2.169.0-1.fc35.noarch 1/19 Running scriptlet: container-selinux-2:2.169.0-1.fc35.noarch 1/19 Installing : containernetworking-plugins-1.1.0-1.fc35.aarch64 2/19 Installing : fuse3-libs-3.10.5-1.fc35.aarch64 3/19 Installing : socat-1.7.4.2-1.fc35.aarch64 4/19 Installing : conmon-2:2.1.0-2.fc35.aarch64 5/19 Installing : aardvark-dns-1.0.3-1.fc35.aarch64 6/19 Installing : netavark-1.0.3-1.fc35.aarch64 7/19 Installing : libslirp-4.6.1-2.fc35.aarch64 8/19 Installing : slirp4netns-1.1.12-2.fc35.aarch64 9/19 Installing : libnet-1.2-4.fc35.aarch64 10/19 Installing : libbsd-0.10.0-8.fc35.aarch64 11/19 Installing : criu-3.16-2.fc35.aarch64 12/19 Installing : runc-2:1.1.3-1.fc35.aarch64 13/19 Installing : fuse-common-3.10.5-1.fc35.aarch64 14/19 Installing : fuse3-3.10.5-1.fc35.aarch64 15/19 Installing : fuse-overlayfs-1.9-1.fc35.aarch64 16/19 Running scriptlet: fuse-overlayfs-1.9-1.fc35.aarch64 16/19 Installing : containers-common-4:1-45.fc35.noarch 17/19 Installing : cri-o-1.21.3-1.module_f35+13330+6bc9c749.aarch64 18/19 Running scriptlet: cri-o-1.21.3-1.module_f35+13330+6bc9c749.aarch64 18/19 Installing : cri-tools-1.19.0-1.module_f35+12974+2bc66b5d.aarch64 19/19 Running scriptlet: container-selinux-2:2.169.0-1.fc35.noarch 19/19 Running scriptlet: cri-tools-1.19.0-1.module_f35+12974+2bc66b5d.aarch64 19/19 Verifying : container-selinux-2:2.169.0-1.fc35.noarch 1/19 Verifying : criu-3.16-2.fc35.aarch64 2/19 Verifying : fuse-common-3.10.5-1.fc35.aarch64 3/19 Verifying : fuse3-3.10.5-1.fc35.aarch64 4/19 Verifying : fuse3-libs-3.10.5-1.fc35.aarch64 5/19 Verifying : libbsd-0.10.0-8.fc35.aarch64 6/19 Verifying : libnet-1.2-4.fc35.aarch64 7/19 Verifying : libslirp-4.6.1-2.fc35.aarch64 8/19 Verifying : slirp4netns-1.1.12-2.fc35.aarch64 9/19 Verifying : aardvark-dns-1.0.3-1.fc35.aarch64 10/19 Verifying : conmon-2:2.1.0-2.fc35.aarch64 11/19 Verifying : containernetworking-plugins-1.1.0-1.fc35.aarch64 12/19 Verifying : containers-common-4:1-45.fc35.noarch 13/19 Verifying : fuse-overlayfs-1.9-1.fc35.aarch64 14/19 Verifying : netavark-1.0.3-1.fc35.aarch64 15/19 Verifying : runc-2:1.1.3-1.fc35.aarch64 16/19 Verifying : socat-1.7.4.2-1.fc35.aarch64 17/19 Verifying : cri-o-1.21.3-1.module_f35+13330+6bc9c749.aarch64 18/19 Verifying : cri-tools-1.19.0-1.module_f35+12974+2bc66b5d.aarch64 19/19 Installed: aardvark-dns-1.0.3-1.fc35.aarch64 conmon-2:2.1.0-2.fc35.aarch64 container-selinux-2:2.169.0-1.fc35.noarch containernetworking-plugins-1.1.0-1.fc35.aarch64 containers-common-4:1-45.fc35.noarch cri-o-1.21.3-1.module_f35+13330+6bc9c749.aarch64 cri-tools-1.19.0-1.module_f35+12974+2bc66b5d.aarch64 criu-3.16-2.fc35.aarch64 fuse-common-3.10.5-1.fc35.aarch64 fuse-overlayfs-1.9-1.fc35.aarch64 fuse3-3.10.5-1.fc35.aarch64 fuse3-libs-3.10.5-1.fc35.aarch64 libbsd-0.10.0-8.fc35.aarch64 libnet-1.2-4.fc35.aarch64 libslirp-4.6.1-2.fc35.aarch64 netavark-1.0.3-1.fc35.aarch64 runc-2:1.1.3-1.fc35.aarch64 slirp4netns-1.1.12-2.fc35.aarch64 socat-1.7.4.2-1.fc35.aarch64 Complete!

After cri-o is installed we can enable it and start it immediately.

$ sudo systemctl enable crio --now Created symlink /etc/systemd/system/multi-user.target.wants/crio.service → /usr/lib/systemd/system/crio.service.

The Microshift packages are in the extra COPR repository so we will need to enable that repository.

$ sudo dnf copr enable -y @redhat-et/microshift /usr/lib/python3.10/site-packages/dnf-plugins/copr.py:433: DeprecationWarning: distro.linux_distribution() is deprecated. It should only be used as a compatibility shim with Python's platform.linux_distribution(). Please use distro.id(), distro.version() and distro.name() instead. dist = linux_distribution() Enabling a Copr repository. Please note that this repository is not part of the main distribution, and quality may vary. The Fedora Project does not exercise any power over the contents of this repository beyond the rules outlined in the Copr FAQ at <https://docs.pagure.org/copr.copr/user_documentation.html#what-i-can-build-in-copr>, and packages are not held to any quality or security level. Please do not file bug reports about these packages in Fedora Bugzilla. In case of problems, contact the owner of this repository. Repository successfully enabled.

Now that the extra repository is enabled we can install the Microshift rpm package and its dependencies on the virtual machine.

$ sudo dnf install -y microshift Copr repo for microshift owned by @redhat-et 4.1 kB/s | 3.5 kB 00:00 Dependencies resolved. =================================================================================================================================================================================================================== Package Architecture Version Repository Size =================================================================================================================================================================================================================== Installing: microshift aarch64 4.8.0-2022_04_20_141053.fc35 copr:copr.fedorainfracloud.org:group_redhat-et:microshift 31 M Upgrading: selinux-policy noarch 35.19-1.fc35 updates 60 k selinux-policy-targeted noarch 35.19-1.fc35 updates 6.3 M Installing dependencies: conntrack-tools aarch64 1.4.5-8.fc35 fedora 201 k libnetfilter_cthelper aarch64 1.0.0-20.fc35 fedora 22 k libnetfilter_cttimeout aarch64 1.0.0-18.fc35 fedora 23 k libnetfilter_queue aarch64 1.0.2-18.fc35 fedora 26 k microshift-selinux noarch 4.8.0-2022_04_20_141053.fc35 copr:copr.fedorainfracloud.org:group_redhat-et:microshift 20 k Transaction Summary =================================================================================================================================================================================================================== Install 6 Packages Upgrade 2 Packages Total download size: 37 M Downloading Packages: (1/8): microshift-selinux-4.8.0-2022_04_20_141053.fc35.noarch.rpm 61 kB/s | 20 kB 00:00 (2/8): conntrack-tools-1.4.5-8.fc35.aarch64.rpm 575 kB/s | 201 kB 00:00 (3/8): libnetfilter_cttimeout-1.0.0-18.fc35.aarch64.rpm 227 kB/s | 23 kB 00:00 (4/8): libnetfilter_cthelper-1.0.0-20.fc35.aarch64.rpm 101 kB/s | 22 kB 00:00 (5/8): libnetfilter_queue-1.0.2-18.fc35.aarch64.rpm 247 kB/s | 26 kB 00:00 (6/8): selinux-policy-35.19-1.fc35.noarch.rpm 91 kB/s | 60 kB 00:00 (7/8): microshift-4.8.0-2022_04_20_141053.fc35.aarch64.rpm 13 MB/s | 31 MB 00:02 (8/8): selinux-policy-targeted-35.19-1.fc35.noarch.rpm 920 kB/s | 6.3 MB 00:07 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 4.4 MB/s | 37 MB 00:08 Copr repo for microshift owned by @redhat-et 3.1 kB/s | 1.0 kB 00:00 Importing GPG key 0xC6BC2A0E: Userid : "@redhat-et_microshift (None) <@redhat-et#microshift@copr.fedorahosted.org>" Fingerprint: C950 F16F F4CC 32E0 BCDA 1DF3 4730 C786 C6BC 2A0E From : https://download.copr.fedorainfracloud.org/results/@redhat-et/microshift/pubkey.gpg Key imported successfully Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Running scriptlet: selinux-policy-targeted-35.19-1.fc35.noarch 1/1 Preparing : 1/1 Upgrading : selinux-policy-35.19-1.fc35.noarch 1/10 Running scriptlet: selinux-policy-35.19-1.fc35.noarch 1/10 Running scriptlet: selinux-policy-targeted-35.19-1.fc35.noarch 2/10 Upgrading : selinux-policy-targeted-35.19-1.fc35.noarch 2/10 Running scriptlet: selinux-policy-targeted-35.19-1.fc35.noarch 2/10 Installing : microshift-selinux-4.8.0-2022_04_20_141053.fc35.noarch 3/10 Running scriptlet: microshift-selinux-4.8.0-2022_04_20_141053.fc35.noarch 3/10 Installing : libnetfilter_queue-1.0.2-18.fc35.aarch64 4/10 Installing : libnetfilter_cttimeout-1.0.0-18.fc35.aarch64 5/10 Installing : libnetfilter_cthelper-1.0.0-20.fc35.aarch64 6/10 Installing : conntrack-tools-1.4.5-8.fc35.aarch64 7/10 Running scriptlet: conntrack-tools-1.4.5-8.fc35.aarch64 7/10 Installing : microshift-4.8.0-2022_04_20_141053.fc35.aarch64 8/10 Running scriptlet: microshift-4.8.0-2022_04_20_141053.fc35.aarch64 8/10 Running scriptlet: selinux-policy-35.3-1.20211019git94970fc.fc35.noarch 9/10 Cleanup : selinux-policy-35.3-1.20211019git94970fc.fc35.noarch 9/10 Running scriptlet: selinux-policy-35.3-1.20211019git94970fc.fc35.noarch 9/10 Cleanup : selinux-policy-targeted-35.3-1.20211019git94970fc.fc35.noarch 10/10 Running scriptlet: selinux-policy-targeted-35.3-1.20211019git94970fc.fc35.noarch 10/10 Running scriptlet: selinux-policy-targeted-35.19-1.fc35.noarch 10/10 Running scriptlet: microshift-selinux-4.8.0-2022_04_20_141053.fc35.noarch 10/10 Running scriptlet: selinux-policy-targeted-35.3-1.20211019git94970fc.fc35.noarch 10/10 Verifying : microshift-4.8.0-2022_04_20_141053.fc35.aarch64 1/10 Verifying : microshift-selinux-4.8.0-2022_04_20_141053.fc35.noarch 2/10 Verifying : conntrack-tools-1.4.5-8.fc35.aarch64 3/10 Verifying : libnetfilter_cthelper-1.0.0-20.fc35.aarch64 4/10 Verifying : libnetfilter_cttimeout-1.0.0-18.fc35.aarch64 5/10 Verifying : libnetfilter_queue-1.0.2-18.fc35.aarch64 6/10 Verifying : selinux-policy-35.19-1.fc35.noarch 7/10 Verifying : selinux-policy-35.3-1.20211019git94970fc.fc35.noarch 8/10 Verifying : selinux-policy-targeted-35.19-1.fc35.noarch 9/10 Verifying : selinux-policy-targeted-35.3-1.20211019git94970fc.fc35.noarch 10/10 Upgraded: selinux-policy-35.19-1.fc35.noarch selinux-policy-targeted-35.19-1.fc35.noarch Installed: conntrack-tools-1.4.5-8.fc35.aarch64 libnetfilter_cthelper-1.0.0-20.fc35.aarch64 libnetfilter_cttimeout-1.0.0-18.fc35.aarch64 libnetfilter_queue-1.0.2-18.fc35.aarch64 microshift-4.8.0-2022_04_20_141053.fc35.aarch64 microshift-selinux-4.8.0-2022_04_20_141053.fc35.noarch Complete!

Once the Microshift package installation is complete we need to next open a few firewall rules and then enable Microshift to start via systemd and start it.

$ sudo firewall-cmd --zone=trusted --add-source=10.42.0.0/16 --permanent success $ sudo firewall-cmd --zone=public --add-port=80/tcp --permanent success $ sudo firewall-cmd --zone=public --add-port=443/tcp --permanent success $ sudo firewall-cmd --zone=public --add-port=5353/udp --permanent success $ sudo firewall-cmd --reload success $ sudo systemctl enable microshift --now Created symlink /etc/systemd/system/multi-user.target.wants/microshift.service → /usr/lib/systemd/system/microshift.service.

While Microshift is starting we can pull down our oc/kubectl binaries which will enable us to interact with Microshift.

$ curl -O https://mirror.openshift.com/pub/openshift-v4/$(uname -m)/clients/ocp/stable/openshift-client-linux.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 48.4M 100 48.4M 0 0 9425k 0 0:00:05 0:00:05 --:--:-- 12.0M

We can untar the binaries directly into the location where they will be available in our default path.

$ sudo tar -xf openshift-client-linux.tar.gz -C /usr/local/bin oc kubectl

Now lets make a hidden kube directory in our home directory and mirror a copy of the kubeconfig for the Microshift deployment there.

$ mkdir ~/.kube $ sudo cat /var/lib/microshift/resources/kubeadmin/kubeconfig > ~/.kube/config

At this point we should be able to interact with the Microshift deployment using oc commands we are familiar with. In the example below I ran the command a few times to show the pods creating and then their final running state.

$ oc get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system kube-flannel-ds-l6v7b 0/1 Init:0/2 0 5s openshift-dns dns-default-4qxss 0/2 ContainerCreating 0 5s openshift-dns node-resolver-k2hkq 0/1 ContainerCreating 0 5s openshift-ingress router-default-85bcfdd948-xgw2l 0/1 Pending 0 9s openshift-service-ca service-ca-7764c85869-nvwcl 0/1 Pending 0 10s $ oc get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system kube-flannel-ds-l6v7b 1/1 Running 0 32s kubevirt-hostpath-provisioner kubevirt-hostpath-provisioner-qdlxs 0/1 ContainerCreating 0 2s openshift-dns dns-default-4qxss 0/2 ContainerCreating 0 32s openshift-dns node-resolver-k2hkq 1/1 Running 0 32s openshift-ingress router-default-85bcfdd948-xgw2l 0/1 Pending 0 36s openshift-service-ca service-ca-7764c85869-nvwcl 0/1 Pending 0 37s $ oc get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system kube-flannel-ds-l6v7b 1/1 Running 0 46s kubevirt-hostpath-provisioner kubevirt-hostpath-provisioner-qdlxs 1/1 Running 0 16s openshift-dns dns-default-4qxss 0/2 ContainerCreating 0 46s openshift-dns node-resolver-k2hkq 1/1 Running 0 46s openshift-ingress router-default-85bcfdd948-xgw2l 0/1 ContainerCreating 0 50s openshift-service-ca service-ca-7764c85869-nvwcl 1/1 Running 0 51s $ oc get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system kube-flannel-ds-l6v7b 1/1 Running 0 53s kubevirt-hostpath-provisioner kubevirt-hostpath-provisioner-qdlxs 1/1 Running 0 23s openshift-dns dns-default-4qxss 0/2 ContainerCreating 0 53s openshift-dns node-resolver-k2hkq 1/1 Running 0 53s openshift-ingress router-default-85bcfdd948-xgw2l 0/1 ContainerCreating 0 57s openshift-service-ca service-ca-7764c85869-nvwcl 1/1 Running 0 58s $ oc get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system kube-flannel-ds-l6v7b 1/1 Running 0 100s kubevirt-hostpath-provisioner kubevirt-hostpath-provisioner-qdlxs 1/1 Running 0 70s openshift-dns dns-default-4qxss 2/2 Running 0 100s openshift-dns node-resolver-k2hkq 1/1 Running 0 100s openshift-ingress router-default-85bcfdd948-xgw2l 1/1 Running 0 104s openshift-service-ca service-ca-7764c85869-nvwcl 1/1 Running 0 105s

Hopefully this gives an idea on how easy deploying Microshift can be and also opens up a lot of possibilities for edge developers to spawn their very own personal development environment of Microshift!

Saturday, May 21, 2022

Check For Expired Certificates on OpenShift


OpenShift has a lot of certificates associated to the services it runs.  With that in mind it might make sense to check on those certificates every once and a while with some kind of simple report.   I have had customers make this request on occasion and it got me thinking about a quick and dirty way to visualize this.   The following blog show the fruits of this simple task in a simple bash script.

First lets go ahead and create the certs-expired.sh script: 

$ cat << EOF > ~/certs-expired.sh
#!/bin/bash

format="%-8s%-8s%-60s%-26s%-60s\n"
printf "$format" STATE DAYS NAME EXPIRY NAMESPACE
printf "$format" ----- ---- ---- ------ ---------

oc get secrets -A -o go-template='{{range .items}}{{if eq .type "kubernetes.io/tls"}}{{.metadata.namespace}}{{" "}}{{.metadata.name}}{{" "}}{{index .data "tls.crt"}}{{"\n"}}{{end}}{{end}}' | while read namespace name cert
do
  certdate=`echo $cert | base64 -d | openssl x509 -noout -enddate|cut -d= -f2`
  epochcertdate=$(date -d "$certdate" +"%s")
  currentdate=$(date +%s)
  if ((epochcertdate > currentdate)); then
    datediff=$((epochcertdate-currentdate))
    state="OK"
  else
    state="EXPIRED"
    datediff=$((currentdate-epochcertdate))
  fi
  days=$((datediff/86400))
  certdate=`echo $cert | base64 -d | openssl x509 -noout -enddate| cut -d= -f2`
  printf "$format" "$state" "$days" "$name" "$certdate" "$namespace" 
done

EOF

The script makes the assumptions that the oc binary is in the current path variable and that the kubeconfig has been set.   This ensures that the oc command inside the script can pull the appropriate data.   If those assumptions are met we can go ahead and run the script.  I chose just to issue a bash against the script but we could have also set the file with the execute permissions.   When we execute the script we can see the output below:

$ bash certs-expired.sh 
STATE   DAYS    NAME                                                        EXPIRY                    NAMESPACE                                                   
-----   ----    ----                                                        ------                    ---------                                                   
OK      715     openshift-apiserver-operator-serving-cert                   May  5 21:33:47 2024 GMT  openshift-apiserver-operator                                
OK      3635    etcd-client                                                 May  3 21:13:54 2032 GMT  openshift-apiserver                                         
OK      715     serving-cert                                                May  5 21:33:52 2024 GMT  openshift-apiserver                                         
OK      715     serving-cert                                                May  5 21:33:59 2024 GMT  openshift-authentication-operator                           
OK      715     v4-0-config-system-serving-cert                             May  5 21:33:49 2024 GMT  openshift-authentication                                    
OK      715     cloud-credential-operator-serving-cert                      May  5 21:33:50 2024 GMT  openshift-cloud-credential-operator                         
OK      715     machine-approver-tls                                        May  5 21:33:48 2024 GMT  openshift-cluster-machine-approver                          
OK      715     node-tuning-operator-tls                                    May  5 21:33:47 2024 GMT  openshift-cluster-node-tuning-operator                      
OK      715     samples-operator-tls                                        May  5 21:37:44 2024 GMT  openshift-cluster-samples-operator                          
OK      715     cluster-storage-operator-serving-cert                       May  5 21:33:55 2024 GMT  openshift-cluster-storage-operator                          
OK      715     csi-snapshot-webhook-secret                                 May  5 21:33:47 2024 GMT  openshift-cluster-storage-operator                          
OK      715     serving-cert                                                May  5 21:33:54 2024 GMT  openshift-cluster-storage-operator                          
OK      715     cluster-version-operator-serving-cert                       May  5 21:33:52 2024 GMT  openshift-cluster-version                                   
OK      15      kube-controller-manager-client-cert-key                     Jun  5 21:33:41 2022 GMT  openshift-config-managed                                    
OK      15      kube-scheduler-client-cert-key                              Jun  5 21:33:34 2022 GMT  openshift-config-managed                                    
OK      715     config-operator-serving-cert                                May  5 21:33:47 2024 GMT  openshift-config-operator                                   
OK      3635    etcd-client                                                 May  3 21:13:54 2032 GMT  openshift-config                                            
OK      3635    etcd-metric-client                                          May  3 21:13:54 2032 GMT  openshift-config                                            
OK      3635    etcd-metric-signer                                          May  3 21:13:54 2032 GMT  openshift-config                                            
OK      3635    etcd-signer                                                 May  3 21:13:54 2032 GMT  openshift-config                                            
OK      715     serving-cert                                                May  5 21:41:37 2024 GMT  openshift-console-operator                                  
OK      715     console-serving-cert                                        May  5 21:42:15 2024 GMT  openshift-console                                           
OK      715     openshift-controller-manager-operator-serving-cert          May  5 21:33:47 2024 GMT  openshift-controller-manager-operator                       
OK      715     serving-cert                                                May  5 21:33:56 2024 GMT  openshift-controller-manager                                
OK      715     metrics-tls                                                 May  5 21:33:58 2024 GMT  openshift-dns-operator                                      
OK      715     dns-default-metrics-tls                                     May  5 21:34:59 2024 GMT  openshift-dns                                               
OK      3635    etcd-client                                                 May  3 21:13:54 2032 GMT  openshift-etcd-operator                                     
OK      715     etcd-operator-serving-cert                                  May  5 21:33:57 2024 GMT  openshift-etcd-operator                                     
OK      3635    etcd-client                                                 May  3 21:13:54 2032 GMT  openshift-etcd                                              
OK      1080    etcd-peer-asus-vm1.kni.schmaustech.com                      May  5 21:51:28 2025 GMT  openshift-etcd                                              
OK      1080    etcd-peer-asus1-vm2.kni.schmaustech.com                     May  5 21:33:23 2025 GMT  openshift-etcd                                              
OK      1080    etcd-peer-asus1-vm3.kni.schmaustech.com                     May  5 21:33:24 2025 GMT  openshift-etcd                                              
OK      1080    etcd-serving-asus-vm1.kni.schmaustech.com                   May  5 21:51:28 2025 GMT  openshift-etcd                                              
OK      1080    etcd-serving-asus1-vm2.kni.schmaustech.com                  May  5 21:33:23 2025 GMT  openshift-etcd                                              
OK      1080    etcd-serving-asus1-vm3.kni.schmaustech.com                  May  5 21:33:24 2025 GMT  openshift-etcd                                              
OK      1080    etcd-serving-metrics-asus-vm1.kni.schmaustech.com           May  5 21:51:27 2025 GMT  openshift-etcd                                              
OK      1080    etcd-serving-metrics-asus1-vm2.kni.schmaustech.com          May  5 21:33:23 2025 GMT  openshift-etcd                                              
OK      1080    etcd-serving-metrics-asus1-vm3.kni.schmaustech.com          May  5 21:33:24 2025 GMT  openshift-etcd                                              
OK      715     serving-cert                                                May  5 21:33:59 2024 GMT  openshift-etcd                                              
OK      715     image-registry-operator-tls                                 May  5 21:33:58 2024 GMT  openshift-image-registry                                    
OK      715     metrics-tls                                                 May  5 21:33:55 2024 GMT  openshift-ingress-operator                                  
OK      715     router-ca                                                   May  5 21:35:59 2024 GMT  openshift-ingress-operator                                  
OK      715     router-certs-default                                        May  5 21:36:01 2024 GMT  openshift-ingress                                           
OK      715     router-metrics-certs-default                                May  5 21:36:00 2024 GMT  openshift-ingress                                           
OK      715     openshift-insights-serving-cert                             May  5 21:33:51 2024 GMT  openshift-insights                                          
OK      15      aggregator-client-signer                                    Jun  6 16:21:59 2022 GMT  openshift-kube-apiserver-operator                           
OK      715     kube-apiserver-operator-serving-cert                        May  5 21:33:54 2024 GMT  openshift-kube-apiserver-operator                           
OK      350     kube-apiserver-to-kubelet-signer                            May  6 21:09:57 2023 GMT  openshift-kube-apiserver-operator                           
OK      350     kube-control-plane-signer                                   May  6 21:09:57 2023 GMT  openshift-kube-apiserver-operator                           
OK      3635    loadbalancer-serving-signer                                 May  3 21:09:52 2032 GMT  openshift-kube-apiserver-operator                           
OK      3635    localhost-recovery-serving-signer                           May  3 21:33:29 2032 GMT  openshift-kube-apiserver-operator                           
OK      3635    localhost-serving-signer                                    May  3 21:09:50 2032 GMT  openshift-kube-apiserver-operator                           
OK      105     node-system-admin-client                                    Sep  3 21:33:40 2022 GMT  openshift-kube-apiserver-operator                           
OK      350     node-system-admin-signer                                    May  6 21:33:29 2023 GMT  openshift-kube-apiserver-operator                           
OK      3635    service-network-serving-signer                              May  3 21:09:51 2032 GMT  openshift-kube-apiserver-operator                           
OK      15      aggregator-client                                           Jun  6 16:21:59 2022 GMT  openshift-kube-apiserver                                    
OK      15      check-endpoints-client-cert-key                             Jun  5 21:33:46 2022 GMT  openshift-kube-apiserver                                    
OK      15      control-plane-node-admin-client-cert-key                    Jun  5 21:33:53 2022 GMT  openshift-kube-apiserver                                    
OK      3635    etcd-client                                                 May  3 21:13:54 2032 GMT  openshift-kube-apiserver                                    
OK      3635    etcd-client-10                                              May  3 21:13:54 2032 GMT  openshift-kube-apiserver                                    
OK      3635    etcd-client-11                                              May  3 21:13:54 2032 GMT  openshift-kube-apiserver                                    
OK      3635    etcd-client-12                                              May  3 21:13:54 2032 GMT  openshift-kube-apiserver                                    
OK      3635    etcd-client-8                                               May  3 21:13:54 2032 GMT  openshift-kube-apiserver                                    
OK      3635    etcd-client-9                                               May  3 21:13:54 2032 GMT  openshift-kube-apiserver                                    
OK      15      external-loadbalancer-serving-certkey                       Jun  5 21:33:52 2022 GMT  openshift-kube-apiserver                                    
OK      15      internal-loadbalancer-serving-certkey                       Jun  5 21:33:34 2022 GMT  openshift-kube-apiserver                                    
OK      15      kubelet-client                                              Jun  5 21:33:34 2022 GMT  openshift-kube-apiserver                                    
OK      3635    localhost-recovery-serving-certkey                          May  3 21:33:29 2032 GMT  openshift-kube-apiserver                                    
OK      3635    localhost-recovery-serving-certkey-10                       May  3 21:33:29 2032 GMT  openshift-kube-apiserver                                    
OK      3635    localhost-recovery-serving-certkey-11                       May  3 21:33:29 2032 GMT  openshift-kube-apiserver                                    
OK      3635    localhost-recovery-serving-certkey-12                       May  3 21:33:29 2032 GMT  openshift-kube-apiserver                                    
OK      3635    localhost-recovery-serving-certkey-8                        May  3 21:33:29 2032 GMT  openshift-kube-apiserver                                    
OK      3635    localhost-recovery-serving-certkey-9                        May  3 21:33:29 2032 GMT  openshift-kube-apiserver                                    
OK      15      localhost-serving-cert-certkey                              Jun  5 21:33:34 2022 GMT  openshift-kube-apiserver                                    
OK      15      service-network-serving-certkey                             Jun  5 21:33:33 2022 GMT  openshift-kube-apiserver                                    
OK      15      csr-signer                                                  Jun  6 16:26:40 2022 GMT  openshift-kube-controller-manager-operator                  
OK      45      csr-signer-signer                                           Jul  6 16:22:14 2022 GMT  openshift-kube-controller-manager-operator                  
OK      715     kube-controller-manager-operator-serving-cert               May  5 21:33:57 2024 GMT  openshift-kube-controller-manager-operator                  
OK      15      csr-signer                                                  Jun  6 16:26:40 2022 GMT  openshift-kube-controller-manager                           
OK      15      kube-controller-manager-client-cert-key                     Jun  5 21:33:41 2022 GMT  openshift-kube-controller-manager                           
OK      715     serving-cert                                                May  5 21:33:51 2024 GMT  openshift-kube-controller-manager                           
OK      715     serving-cert-2                                              May  5 21:33:51 2024 GMT  openshift-kube-controller-manager                           
OK      715     serving-cert-3                                              May  5 21:33:51 2024 GMT  openshift-kube-controller-manager                           
OK      715     serving-cert-4                                              May  5 21:33:51 2024 GMT  openshift-kube-controller-manager                           
OK      715     serving-cert-5                                              May  5 21:33:51 2024 GMT  openshift-kube-controller-manager                           
OK      715     serving-cert-6                                              May  5 21:33:51 2024 GMT  openshift-kube-controller-manager                           
OK      715     serving-cert-7                                              May  5 21:33:51 2024 GMT  openshift-kube-controller-manager                           
OK      715     kube-scheduler-operator-serving-cert                        May  5 21:33:50 2024 GMT  openshift-kube-scheduler-operator                           
OK      15      kube-scheduler-client-cert-key                              Jun  5 21:33:34 2022 GMT  openshift-kube-scheduler                                    
OK      715     serving-cert                                                May  5 21:33:59 2024 GMT  openshift-kube-scheduler                                    
OK      715     serving-cert-3                                              May  5 21:33:59 2024 GMT  openshift-kube-scheduler                                    
OK      715     serving-cert-4                                              May  5 21:33:59 2024 GMT  openshift-kube-scheduler                                    
OK      715     serving-cert-5                                              May  5 21:33:59 2024 GMT  openshift-kube-scheduler                                    
OK      715     serving-cert-6                                              May  5 21:33:59 2024 GMT  openshift-kube-scheduler                                    
OK      715     serving-cert-7                                              May  5 21:33:59 2024 GMT  openshift-kube-scheduler                                    
OK      715     serving-cert                                                May  5 21:34:00 2024 GMT  openshift-kube-storage-version-migrator-operator            
OK      725     diskmaker-metric-serving-cert                               May 15 23:33:46 2024 GMT  openshift-local-storage                                     
OK      715     baremetal-operator-webhook-server-cert                      May  5 21:36:34 2024 GMT  openshift-machine-api                                       
OK      715     cluster-autoscaler-operator-cert                            May  5 21:34:01 2024 GMT  openshift-machine-api                                       
OK      715     cluster-baremetal-operator-tls                              May  5 21:33:58 2024 GMT  openshift-machine-api                                       
OK      715     cluster-baremetal-webhook-server-cert                       May  5 21:33:48 2024 GMT  openshift-machine-api                                       
OK      715     machine-api-controllers-tls                                 May  5 21:33:47 2024 GMT  openshift-machine-api                                       
OK      715     machine-api-operator-tls                                    May  5 21:33:56 2024 GMT  openshift-machine-api                                       
OK      715     machine-api-operator-webhook-cert                           May  5 21:33:53 2024 GMT  openshift-machine-api                                       
OK      715     proxy-tls                                                   May  5 21:34:00 2024 GMT  openshift-machine-config-operator                           
OK      715     marketplace-operator-metrics                                May  5 21:33:50 2024 GMT  openshift-marketplace                                       
OK      715     alertmanager-main-tls                                       May  5 21:45:20 2024 GMT  openshift-monitoring                                        
OK      715     cluster-monitoring-operator-tls                             May  5 21:33:52 2024 GMT  openshift-monitoring                                        
OK      715     grafana-tls                                                 May  5 21:45:20 2024 GMT  openshift-monitoring                                        
OK      715     kube-state-metrics-tls                                      May  5 21:35:59 2024 GMT  openshift-monitoring                                        
OK      715     node-exporter-tls                                           May  5 21:35:59 2024 GMT  openshift-monitoring                                        
OK      715     openshift-state-metrics-tls                                 May  5 21:35:58 2024 GMT  openshift-monitoring                                        
OK      715     prometheus-adapter-tls                                      May  5 21:35:59 2024 GMT  openshift-monitoring                                        
OK      715     prometheus-k8s-thanos-sidecar-tls                           May  5 21:45:22 2024 GMT  openshift-monitoring                                        
OK      715     prometheus-k8s-tls                                          May  5 21:45:21 2024 GMT  openshift-monitoring                                        
OK      715     prometheus-operator-tls                                     May  5 21:35:43 2024 GMT  openshift-monitoring                                        
OK      715     telemeter-client-tls                                        May  5 21:37:44 2024 GMT  openshift-monitoring                                        
OK      715     thanos-querier-tls                                          May  5 21:35:58 2024 GMT  openshift-monitoring                                        
OK      715     metrics-daemon-secret                                       May  5 21:33:56 2024 GMT  openshift-multus                                            
OK      715     multus-admission-controller-secret                          May  5 21:33:48 2024 GMT  openshift-multus                                            
OK      3635    etcd-client                                                 May  3 21:13:54 2032 GMT  openshift-oauth-apiserver                                   
OK      715     serving-cert                                                May  5 21:34:01 2024 GMT  openshift-oauth-apiserver                                   
OK      715     catalog-operator-serving-cert                               May  5 21:33:47 2024 GMT  openshift-operator-lifecycle-manager                        
OK      715     olm-operator-serving-cert                                   May  5 21:33:48 2024 GMT  openshift-operator-lifecycle-manager                        
OK      714     packageserver-service-cert                                  May  4 21:34:44 2024 GMT  openshift-operator-lifecycle-manager                        
OK      0       pprof-cert                                                  May 21 18:30:03 2022 GMT  openshift-operator-lifecycle-manager                        
OK      3635    ovn-ca                                                      May  3 21:27:45 2032 GMT  openshift-ovn-kubernetes                                    
OK      167     ovn-cert                                                    Nov  5 09:27:45 2022 GMT  openshift-ovn-kubernetes                                    
OK      715     ovn-master-metrics-cert                                     May  5 21:33:53 2024 GMT  openshift-ovn-kubernetes                                    
OK      715     ovn-node-metrics-cert                                       May  5 21:33:49 2024 GMT  openshift-ovn-kubernetes                                    
OK      3635    signer-ca                                                   May  3 21:27:46 2032 GMT  openshift-ovn-kubernetes                                    
OK      167     signer-cert                                                 Nov  5 09:27:46 2022 GMT  openshift-ovn-kubernetes                                    
OK      715     serving-cert                                                May  5 21:33:54 2024 GMT  openshift-service-ca-operator                               
OK      775     signing-key                                                 Jul  4 21:33:37 2024 GMT  openshift-service-ca                                        
OK      725     noobaa-db-serving-cert                                      May 15 23:42:26 2024 GMT  openshift-storage                                           
OK      725     noobaa-mgmt-serving-cert                                    May 15 23:42:26 2024 GMT  openshift-storage                                           
OK      725     noobaa-operator-service-cert                                May 16 06:23:29 2024 GMT  openshift-storage                                           
OK      725     noobaa-s3-serving-cert                                      May 15 23:42:26 2024 GMT  openshift-storage                                           
OK      725     ocs-storagecluster-cos-ceph-rgw-tls-cert                    May 15 23:41:32 2024 GMT  openshift-storage                                           
OK      725     odf-console-serving-cert                                    May 15 23:27:38 2024 GMT  openshift-storage   

The output of the script is simple.  The first column contains the state of the certificate.  If its okay then it just says OK and if its expired the field will say EXPIRED.   The next column tells us how many days until the certificate expires and if the number is negative then the certificate is expired and has been for that many days.   The third column tells us the certificates name while the fourth gives us the actual expiry date.   Finally the last column provides the namespace the certificate is in.

Again just a simple script but provides an example of how we can see this type of information.  However if one has a fleet of clusters then configuring a Red Hat Advanced Cluster Management Certificate Policy Controller might be a more effective method at expired certificate management.

Friday, May 20, 2022

Install OpenShift with Agent Installer


There are so many ways to install OpenShift: Assisted Installer, UPI, IPI, Red Hat Advanced Cluster Management and ZTP.  However I have always longed for a single ISO image I could just boot my physical hardware and it would form a OpenShift cluster.  Well that dream is on course to become a reality with the Agent Installer a tool that can generate an ephemeral OpenShift installation image.  In the following blog I will demonstrate how to use this early incarnation of the tool.

As I stated the Agent Installer generates a single ISO image that one would use to boot all of the nodes they would want to be part of a newly deployed cluster.  However this current example may change some as the code gets developed and merged into the mainstream Openshift installer.  However if one is interested in exploring this new method the following can be a preview of what is to come.

The first step in trying out the Agent Installer is to grab the OpenShift installer source code from Github and checkout the agent-installer branch:

$ git clone https://github.com/openshift/installer
Cloning into 'installer'...
remote: Enumerating objects: 204497, done.
remote: Counting objects: 100% (210/210), done.
remote: Compressing objects: 100% (130/130), done.
remote: Total 204497 (delta 99), reused 153 (delta 70), pack-reused 204287
Receiving objects: 100% (204497/204497), 873.44 MiB | 10.53 MiB/s, done.
Resolving deltas: 100% (132947/132947), done.
Updating files: 100% (86883/86883), done.

$ git checkout 88db7ef
Updating files: 100% (23993/23993), done.
Note: switching to '88db7ef'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 88db7eff2 Fix unnecessary delays in start-cluster-installation

$ git branch
* (HEAD detached at 88db7eff2)
  master

Once we have the source code checked out we need to go ahead and build the the OpenShift install binary:

$ hack/build.sh
+ minimum_go_version=1.17
++ go version
++ cut -d ' ' -f 3
+ current_go_version=go1.17.7
++ version 1.17.7
++ IFS=.
++ printf '%03d%03d%03d\n' 1 17 7
++ unset IFS
++ version 1.17
++ IFS=.
++ printf '%03d%03d%03d\n' 1 17
++ unset IFS
+ '[' 001017007 -lt 001017000 ']'
+ make -C terraform all
make: Entering directory '/home/bschmaus/installer/terraform'
cd providers/alicloud; \
if [ -f main.go ]; then path="."; else path=./vendor/`grep _ tools.go|awk '{ print $2 }'|sed 's|"||g'`; fi; \
go build -ldflags "-s -w" -o ../../bin/terraform-provider-alicloud "$path"; \
zip -1j ../../bin/terraform-provider-alicloud.zip ../../bin/terraform-provider-alicloud;
  adding: terraform-provider-alicloud (deflated 81%)
cd providers/aws; \
if [ -f main.go ]; then path="."; else path=./vendor/`grep _ tools.go|awk '{ print $2 }'|sed 's|"||g'`; fi; \
go build -ldflags "-s -w" -o ../../bin/terraform-provider-aws "$path"; \
zip -1j ../../bin/terraform-provider-aws.zip ../../bin/terraform-provider-aws;
  adding: terraform-provider-aws (deflated 75%)
cd providers/azureprivatedns; \
if [ -f main.go ]; then path="."; else path=./vendor/`grep _ tools.go|awk '{ print $2 }'|sed 's|"||g'`; fi; \
go build -ldflags "-s -w" -o ../../bin/terraform-provider-azureprivatedns "$path"; \
zip -1j ../../bin/terraform-provider-azureprivatedns.zip ../../bin/terraform-provider-azureprivatedns;
  adding: terraform-provider-azureprivatedns (deflated 62%)
cd providers/azurerm; \
if [ -f main.go ]; then path="."; else path=./vendor/`grep _ tools.go|awk '{ print $2 }'|sed 's|"||g'`; fi; \
go build -ldflags "-s -w" -o ../../bin/terraform-provider-azurerm "$path"; \
zip -1j ../../bin/terraform-provider-azurerm.zip ../../bin/terraform-provider-azurerm;
  adding: terraform-provider-azurerm (deflated 77%)
cd providers/azurestack; \
if [ -f main.go ]; then path="."; else path=./vendor/`grep _ tools.go|awk '{ print $2 }'|sed 's|"||g'`; fi; \
go build -ldflags "-s -w" -o ../../bin/terraform-provider-azurestack "$path"; \
zip -1j ../../bin/terraform-provider-azurestack.zip ../../bin/terraform-provider-azurestack;
  adding: terraform-provider-azurestack (deflated 64%)
cd providers/google; \
if [ -f main.go ]; then path="."; else path=./vendor/`grep _ tools.go|awk '{ print $2 }'|sed 's|"||g'`; fi; \
go build -ldflags "-s -w" -o ../../bin/terraform-provider-google "$path"; \
zip -1j ../../bin/terraform-provider-google.zip ../../bin/terraform-provider-google;
  adding: terraform-provider-google (deflated 68%)
cd providers/ibm; \
if [ -f main.go ]; then path="."; else path=./vendor/`grep _ tools.go|awk '{ print $2 }'|sed 's|"||g'`; fi; \
go build -ldflags "-s -w" -o ../../bin/terraform-provider-ibm "$path"; \
zip -1j ../../bin/terraform-provider-ibm.zip ../../bin/terraform-provider-ibm;
  adding: terraform-provider-ibm (deflated 67%)
cd providers/ignition; \
if [ -f main.go ]; then path="."; else path=./vendor/`grep _ tools.go|awk '{ print $2 }'|sed 's|"||g'`; fi; \
go build -ldflags "-s -w" -o ../../bin/terraform-provider-ignition "$path"; \
zip -1j ../../bin/terraform-provider-ignition.zip ../../bin/terraform-provider-ignition;
  adding: terraform-provider-ignition (deflated 61%)
cd providers/ironic; \
if [ -f main.go ]; then path="."; else path=./vendor/`grep _ tools.go|awk '{ print $2 }'|sed 's|"||g'`; fi; \
go build -ldflags "-s -w" -o ../../bin/terraform-provider-ironic "$path"; \
zip -1j ../../bin/terraform-provider-ironic.zip ../../bin/terraform-provider-ironic;
  adding: terraform-provider-ironic (deflated 60%)
cd providers/libvirt; \
if [ -f main.go ]; then path="."; else path=./vendor/`grep _ tools.go|awk '{ print $2 }'|sed 's|"||g'`; fi; \
go build -ldflags "-s -w" -o ../../bin/terraform-provider-libvirt "$path"; \
zip -1j ../../bin/terraform-provider-libvirt.zip ../../bin/terraform-provider-libvirt;
  adding: terraform-provider-libvirt (deflated 61%)
cd providers/local; \
if [ -f main.go ]; then path="."; else path=./vendor/`grep _ tools.go|awk '{ print $2 }'|sed 's|"||g'`; fi; \
go build -ldflags "-s -w" -o ../../bin/terraform-provider-local "$path"; \
zip -1j ../../bin/terraform-provider-local.zip ../../bin/terraform-provider-local;
  adding: terraform-provider-local (deflated 59%)
cd providers/nutanix; \
if [ -f main.go ]; then path="."; else path=./vendor/`grep _ tools.go|awk '{ print $2 }'|sed 's|"||g'`; fi; \
go build -ldflags "-s -w" -o ../../bin/terraform-provider-nutanix "$path"; \
zip -1j ../../bin/terraform-provider-nutanix.zip ../../bin/terraform-provider-nutanix;
  adding: terraform-provider-nutanix (deflated 60%)
cd providers/openstack; \
if [ -f main.go ]; then path="."; else path=./vendor/`grep _ tools.go|awk '{ print $2 }'|sed 's|"||g'`; fi; \
go build -ldflags "-s -w" -o ../../bin/terraform-provider-openstack "$path"; \
zip -1j ../../bin/terraform-provider-openstack.zip ../../bin/terraform-provider-openstack;
  adding: terraform-provider-openstack (deflated 62%)
cd providers/ovirt; \
if [ -f main.go ]; then path="."; else path=./vendor/`grep _ tools.go|awk '{ print $2 }'|sed 's|"||g'`; fi; \
go build -ldflags "-s -w" -o ../../bin/terraform-provider-ovirt "$path"; \
zip -1j ../../bin/terraform-provider-ovirt.zip ../../bin/terraform-provider-ovirt;
  adding: terraform-provider-ovirt (deflated 66%)
cd providers/random; \
if [ -f main.go ]; then path="."; else path=./vendor/`grep _ tools.go|awk '{ print $2 }'|sed 's|"||g'`; fi; \
go build -ldflags "-s -w" -o ../../bin/terraform-provider-random "$path"; \
zip -1j ../../bin/terraform-provider-random.zip ../../bin/terraform-provider-random;
  adding: terraform-provider-random (deflated 59%)
cd providers/vsphere; \
if [ -f main.go ]; then path="."; else path=./vendor/`grep _ tools.go|awk '{ print $2 }'|sed 's|"||g'`; fi; \
go build -ldflags "-s -w" -o ../../bin/terraform-provider-vsphere "$path"; \
zip -1j ../../bin/terraform-provider-vsphere.zip ../../bin/terraform-provider-vsphere;
  adding: terraform-provider-vsphere (deflated 68%)
cd providers/vsphereprivate; \
if [ -f main.go ]; then path="."; else path=./vendor/`grep _ tools.go|awk '{ print $2 }'|sed 's|"||g'`; fi; \
go build -ldflags "-s -w" -o ../../bin/terraform-provider-vsphereprivate "$path"; \
zip -1j ../../bin/terraform-provider-vsphereprivate.zip ../../bin/terraform-provider-vsphereprivate;
  adding: terraform-provider-vsphereprivate (deflated 69%)
cd terraform; \
go build -ldflags "-s -w" -o ../bin/terraform ./vendor/github.com/hashicorp/terraform
make: Leaving directory '/home/bschmaus/installer/terraform'
+ copy_terraform_to_mirror
++ go env GOOS
++ go env GOARCH
+ TARGET_OS_ARCH=linux_amd64
+ rm -rf '/home/bschmaus/installer/pkg/terraform/providers/mirror/*/'
+ find /home/bschmaus/installer/terraform/bin/ -maxdepth 1 -name 'terraform-provider-*.zip' -exec bash -c '
      providerName="$(basename "$1" | cut -d - -f 3 | cut -d . -f 1)"
      targetOSArch="$2"
      dstDir="${PWD}/pkg/terraform/providers/mirror/openshift/local/$providerName"
      mkdir -p "$dstDir"
      echo "Copying $providerName provider to mirror"
      cp "$1" "$dstDir/terraform-provider-${providerName}_1.0.0_${targetOSArch}.zip"
    ' shell '{}' linux_amd64 ';'
Copying alicloud provider to mirror
Copying aws provider to mirror
Copying azureprivatedns provider to mirror
Copying azurerm provider to mirror
Copying azurestack provider to mirror
Copying google provider to mirror
Copying ibm provider to mirror
Copying ignition provider to mirror
Copying ironic provider to mirror
Copying libvirt provider to mirror
Copying local provider to mirror
Copying nutanix provider to mirror
Copying openstack provider to mirror
Copying ovirt provider to mirror
Copying random provider to mirror
Copying vsphere provider to mirror
Copying vsphereprivate provider to mirror
+ mkdir -p /home/bschmaus/installer/pkg/terraform/providers/mirror/terraform/
+ cp /home/bschmaus/installer/terraform/bin/terraform /home/bschmaus/installer/pkg/terraform/providers/mirror/terraform/
+ MODE=release
++ git rev-parse --verify 'HEAD^{commit}'
+ GIT_COMMIT=d74e210f30edf110764d87c8223a18b8a9952253
++ git describe --always --abbrev=40 --dirty
+ GIT_TAG=unreleased-master-6040-gd74e210f30edf110764d87c8223a18b8a9952253
+ DEFAULT_ARCH=amd64
+ GOFLAGS=-mod=vendor
+ LDFLAGS=' -X github.com/openshift/installer/pkg/version.Raw=unreleased-master-6040-gd74e210f30edf110764d87c8223a18b8a9952253 -X github.com/openshift/installer/pkg/version.Commit=d74e210f30edf110764d87c8223a18b8a9952253 -X github.com/openshift/installer/pkg/version.defaultArch=amd64'
+ TAGS=
+ OUTPUT=bin/openshift-install
+ export CGO_ENABLED=0
+ CGO_ENABLED=0
+ case "${MODE}" in
+ LDFLAGS=' -X github.com/openshift/installer/pkg/version.Raw=unreleased-master-6040-gd74e210f30edf110764d87c8223a18b8a9952253 -X github.com/openshift/installer/pkg/version.Commit=d74e210f30edf110764d87c8223a18b8a9952253 -X github.com/openshift/installer/pkg/version.defaultArch=amd64 -s -w'
+ TAGS=' release'
+ test '' '!=' y
+ go generate ./data
writing assets_vfsdata.go
+ echo ' release'
+ grep -q libvirt
+ go build -mod=vendor -ldflags ' -X github.com/openshift/installer/pkg/version.Raw=unreleased-master-6040-gd74e210f30edf110764d87c8223a18b8a9952253 -X github.com/openshift/installer/pkg/version.Commit=d74e210f30edf110764d87c8223a18b8a9952253 -X github.com/openshift/installer/pkg/version.defaultArch=amd64 -s -w' -tags ' release' -o bin/openshift-install ./cmd/openshift-install 

Once the OpenShift install binary is built we next need to create a manifests directory under the installer directory.  In this manifest directory we will be creating six files that basically give the Agent Installer the blueprints of what our cluster should look like.  First lets create the directory:

$ pwd 
/home/bschmaus/installer

$ mkdir manifests

With the directory created we can move onto creating the agent cluster install resource file.  This file specifies the clusters configuration such as number of control plane and/or worker nodes, the api and ingress vip and the cluster networking.   In my example I will be deploying a 3 node compact cluster which referenced a cluster deployment named kni22:

$ cat << EOF > ./manifests/agent-cluster-install.yaml
apiVersion: extensions.hive.openshift.io/v1beta1
kind: AgentClusterInstall
metadata:
  name: kni22
  namespace: kni22
spec:
  apiVIP: 192.168.0.125
  ingressVIP: 192.168.0.126
  clusterDeploymentRef:
    name: kni22
  imageSetRef:
    name: openshift-v4.10.0
  networking:
    clusterNetwork:
    - cidr: 10.128.0.0/14
      hostPrefix: 23
    serviceNetwork:
    - 172.30.0.0/16
  provisionRequirements:
    controlPlaneAgents: 3
    workerAgents: 0 
  sshPublicKey: 'INSERT PUBLIC SSH KEY HERE'
EOF

Next we will create the cluster deployment resource file which defines the cluster name, domain, and other details:

$ cat << EOF > ./manifests/cluster-deployment.yaml
apiVersion: hive.openshift.io/v1
kind: ClusterDeployment
metadata:
  name: kni22
  namespace: kni22
spec:
  baseDomain: schmaustech.com
  clusterInstallRef:
    group: extensions.hive.openshift.io
    kind: AgentClusterInstall
    name: kni22-agent-cluster-install
    version: v1beta1
  clusterName: kni22
  controlPlaneConfig:
    servingCertificates: {}
  platform:
    agentBareMetal:
      agentSelector:
        matchLabels:
          bla: aaa
  pullSecretRef:
    name: pull-secret
EOF

Moving on we now create the cluster image set resource file which contains OpenShift image information such as the repository and image name.  This will be the version of the cluster that gets deployed in our 3 node compact cluster.  In this example we are using 4.10.10:

$ cat << EOF > ./manifests/cluster-image-set.yaml
apiVersion: hive.openshift.io/v1
kind: ClusterImageSet
metadata:
  name: ocp-release-4.10.10-x86-64-for-4.10.0-0-to-4.11.0-0
spec:
  releaseImage: quay.io/openshift-release-dev/ocp-release:4.10.10-x86_64
EOF

Next we define the infrastructure environment file which  contains information for pulling OpenShift onto the target host nodes we are deploying to:

$ cat << EOF > ./manifests/infraenv.yaml 
apiVersion: agent-install.openshift.io/v1beta1
kind: InfraEnv
metadata:
  name: kni22
  namespace: kni22
spec:
  clusterRef:
    name: kni22  
    namespace: kni22
  pullSecretRef:
    name: pull-secret
  sshAuthorizedKey: 'INSERT PUBLIC SSH KEY HERE'
  nmStateConfigLabelSelector:
    matchLabels:
      kni22-nmstate-label-name: kni22-nmstate-label-value
EOF

The next file is the nmstate configuration file and this file provides all the details for all of the host that will be booted using the ISO image we are going to create.   Since we have a 3 node compact cluster to deploy we notice that in the file below we have specified three nmstate configurations.  Each configuration is for a node and generates a static IP address on the nodes enp2s0 interface that matches the MAC address defined.   This enables the ISO to boot up and not necessarily require DHCP in the environment which is what a lot of customers are looking for.   Again my example has 3 configurations but if we had worker nodes we would add those in too.   Lets go ahead and create the file:

$ cat << EOF > ./manifests/nmstateconfig.yaml
---
apiVersion: agent-install.openshift.io/v1beta1
kind: NMStateConfig
metadata:
  name: mynmstateconfig01
  namespace: openshift-machine-api
  labels:
    kni22-nmstate-label-name: kni22-nmstate-label-value
spec:
  config:
    interfaces:
      - name: enp2s0
        type: ethernet
        state: up
        mac-address: 52:54:00:e7:05:72
        ipv4:
          enabled: true
          address:
            - ip: 192.168.0.116
              prefix-length: 24
          dhcp: false
    dns-resolver:
      config:
        server:
          - 192.168.0.10
    routes:
      config:
        - destination: 0.0.0.0/0
          next-hop-address: 192.168.0.1
          next-hop-interface: enp2s0
          table-id: 254
  interfaces:
    - name: "enp2s0"
      macAddress: 52:54:00:e7:05:72
---
apiVersion: agent-install.openshift.io/v1beta1
kind: NMStateConfig
metadata:
  name: mynmstateconfig02
  namespace: openshift-machine-api
  labels:
    kni22-nmstate-label-name: kni22-nmstate-label-value
spec:
  config:
    interfaces:
      - name: enp2s0
        type: ethernet
        state: up
        mac-address: 52:54:00:95:fd:f3
        ipv4:
          enabled: true
          address:
            - ip: 192.168.0.117
              prefix-length: 24
          dhcp: false
    dns-resolver:
      config:
        server:
          - 192.168.0.10
    routes:
      config:
        - destination: 0.0.0.0/0
          next-hop-address: 192.168.0.1
          next-hop-interface: enp2s0
          table-id: 254
  interfaces:
    - name: "enp2s0"
      macAddress: 52:54:00:95:fd:f3
---
apiVersion: agent-install.openshift.io/v1beta1
kind: NMStateConfig
metadata:
  name: mynmstateconfig03
  namespace: openshift-machine-api
  labels:
    kni22-nmstate-label-name: kni22-nmstate-label-value
spec:
  config:
    interfaces:
      - name: enp2s0
        type: ethernet
        state: up
        mac-address: 52:54:00:e8:b9:18
        ipv4:
          enabled: true
          address:
            - ip: 192.168.0.118
              prefix-length: 24
          dhcp: false
    dns-resolver:
      config:
        server:
          - 192.168.0.10
    routes:
      config:
        - destination: 0.0.0.0/0
          next-hop-address: 192.168.0.1
          next-hop-interface: enp2s0
          table-id: 254
  interfaces:
    - name: "enp2s0"
      macAddress: 52:54:00:e8:b9:18
EOF

The final file we need to create is the pull-secret resource file which contains the pull-secret values so that our cluster can pull in the required OpenShift images to instantiate the cluster:

$ cat << EOF > ./manifests/pull-secret.yaml 
apiVersion: v1
kind: Secret
type: kubernetes.io/dockerconfigjson
metadata:
  name: pull-secret
  namespace: kni22
stringData:
  .dockerconfigjson: 'INSERT JSON FORMATTED PULL-SECRET'
EOF

At this point we should now have our six required files defined to build our Agent Installer ISO:

$ ls -1 ./manifests/
agent-cluster-install.yaml
cluster-deployment.yaml
cluster-image-set.yaml
infraenv.yaml
nmstateconfig.yaml
pull-secret.yaml 

We are now ready to use the Openshift install binary we compiled earlier with the Agent Installer code to generate our ephemeral OpenShift ISO.   We do this by issuing the following command which introduces the agent option.  This in turn will read in the manifest details we generated and download the corresponding RHCOS image and then inject our details into the image writing out a file called agent.iso:

$ bin/openshift-install agent create image 
INFO adding MAC interface map to host static network config - Name:  enp2s0  MacAddress: 52:54:00:e7:05:72 
INFO adding MAC interface map to host static network config - Name:  enp2s0  MacAddress: 52:54:00:95:fd:f3 
INFO adding MAC interface map to host static network config - Name:  enp2s0  MacAddress: 52:54:00:e8:b9:18 
INFO[0000] Adding NMConnection file <enp2s0 .nmconnection="">  pkg=manifests
INFO[0000] Adding NMConnection file <enp2s0 .nmconnection="">  pkg=manifests
INFO[0001] Adding NMConnection file <enp2s0 .nmconnection="">  pkg=manifests
INFO[0001] Start configuring static network for 3 hosts  pkg=manifests
INFO[0001] Adding NMConnection file <enp2s0 .nmconnection="">  pkg=manifests
INFO[0001] Adding NMConnection file <enp2s0 .nmconnection="">  pkg=manifests
INFO[0001] Adding NMConnection file <enp2s0 .nmconnection="">  pkg=manifests
INFO Obtaining RHCOS image file from 'https://rhcos-redirector.apps.art.xq1c.p1.openshiftapps.com/art/storage/releases/rhcos-4.11/411.85.202203181601-0/x86_64/rhcos-411.85.202203181601-0-live.x86_64.iso' 
INFO   

Once the agent create image command completes we are left with a agent.iso image which is in fact our OpenShift install ISO:

$ ls -l ./output/
total 1073152
-rw-rw-r--. 1 bschmaus bschmaus 1098907648 May 20 08:55 agent.iso

Since the nodes I will be using to demonstrate this 3 node compact cluster are virtual machines all on the same KVM hypervisor I will go ahead and copy the agent.iso image over to that host:

$ scp ./output/agent.iso root@192.168.0.22:/var/lib/libvirt/images/
root@192.168.0.22's password: 
agent.iso   

With the image moved over to the hypervisor host I went ahead and ensured each virtual machine we are using (asus3-vm[1-3]) has the image set.  Further the hosts are designed boot off the ISO if the disk is empty.  We can confirm everything is ready with the following output:

# virsh list --all
 Id   Name        State
----------------------------
 -    asus3-vm1   shut off
 -    asus3-vm2   shut off
 -    asus3-vm3   shut off
 -    asus3-vm4   shut off
 -    asus3-vm5   shut off
 -    asus3-vm6   shut off

# virsh domblklist asus3-vm1
 Target   Source
---------------------------------------------------
 sda      /var/lib/libvirt/images/asus3-vm1.qcow2
 sdb      /var/lib/libvirt/images/agent.iso

# virsh domblklist asus3-vm2
 Target   Source
---------------------------------------------------
 sda      /var/lib/libvirt/images/asus3-vm2.qcow2
 sdb      /var/lib/libvirt/images/agent.iso

# virsh domblklist asus3-vm3
 Target   Source
---------------------------------------------------
 sda      /var/lib/libvirt/images/asus3-vm3.qcow2
 sdb      /var/lib/libvirt/images/agent.iso
 

# virsh start asus3-vm1
Domain asus3-vm1 started

Once the first virtual machine is started we can switch over to the console and watch it boot up:


During the boot process the system will come up to a standard login prompt on the console.  Then in the background on the host it will start pulling in the required containers to run the familiar Assisted Installer UI.  I gave this process about 5 minutes before I attempted to access the web UI.   To access the web UI we can point our browser to the ipaddress of node we just booted and port 8080:


We should see a visible kni22 cluster with a status of draft because no nodes have been associated to it yet.  Next we will click on kni22 to bring us into the configuration:


We can see the familiar Assisted Installer discovery screen and we can also see our first host is listed.   At this point lets turn on the other two nodes that will make up our 3 node compact cluster and let them also boot from the agent ISO we created.

After the other two  nodes have booted we should see them appear in the web UI.  We also can see that the node names are all localhost.  This was due to the fact I set static IP addresses in the nmstate.yaml above.   If we had gone with DHCP the names would have been set by DHCP.  Nevertheless though we can go ahead and edit each hostname and set it to the proper name and click next to continue:


There will be an additional configuration page where other configuration items will be set and could be changed if needed but we will click next through that screen to bring us to the summary page:



If everything looks correct we can go ahead and click on the install cluster button to start the deployment:




At this point the cluster installation begins.  I should point out however we will not be able to watch the installation complete from the web UI.  The reason being is that the other two nodes will get their RHCOS images written to disk, reboot and then instantiate part of the cluster.  At that point the first node, the one running the web UI, will also get its RHCOS image written to disk and reboot.  After that the web UI is not longer available to watch.   With that in mind I recommend grabbing the kubeconfig for the cluster by clicking on the download kubeconfig button.

Once the web UI is no longer accessible we can monitor the installation from command line using the kubeconfig we downloaded.   First lets see where the nodes are at:

$ export KUBECONFIG=/home/bschmaus/kubeconfig-kni22

$ oc get nodes
NAME        STATUS   ROLES           AGE   VERSION
asus3-vm1   Ready    master,worker   2m    v1.23.5+9ce5071
asus3-vm2   Ready    master,worker   29m   v1.23.5+9ce5071
asus3-vm3   Ready    master,worker   29m   v1.23.5+9ce5071

All the nodes are in a ready state and marked as both a control node and worker.   Now lets see where the cluster operators are at:

$ oc get co 
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.10.10   False       True          True       18m     WellKnownAvailable: The well-known endpoint is not yet available: need at least 3 kube-apiservers, got 2
baremetal                                  4.10.10   True        False         False      17m     
cloud-controller-manager                   4.10.10   True        False         False      29m     
cloud-credential                           4.10.10   True        False         False      34m     
cluster-autoscaler                         4.10.10   True        False         False      16m     
config-operator                            4.10.10   True        False         False      18m     
console                                    4.10.10   True        False         False      4m33s   
csi-snapshot-controller                    4.10.10   True        False         False      18m     
dns                                        4.10.10   True        False         False      17m     
etcd                                       4.10.10   True        True          False      16m     NodeInstallerProgressing: 1 nodes are at revision 0; 2 nodes are at revision 4; 0 nodes have achieved new revision 5
image-registry                             4.10.10   True        False         False      9m44s   
ingress                                    4.10.10   True        False         False      11m     
insights                                   4.10.10   True        False         False      12m     
kube-apiserver                             4.10.10   True        True          False      4m29s   NodeInstallerProgressing: 1 nodes are at revision 0; 2 nodes are at revision 6
kube-controller-manager                    4.10.10   True        True          False      14m     NodeInstallerProgressing: 1 nodes are at revision 0; 2 nodes are at revision 7
kube-scheduler                             4.10.10   True        True          False      14m     NodeInstallerProgressing: 1 nodes are at revision 0; 2 nodes are at revision 6
kube-storage-version-migrator              4.10.10   True        False         False      18m     
machine-api                                4.10.10   True        False         False      7m53s   
machine-approver                           4.10.10   True        False         False      17m     
machine-config                             4.10.10   True        False         False      17m     
marketplace                                4.10.10   True        False         False      16m     
monitoring                                 4.10.10   True        False         False      5m52s   
network                                    4.10.10   True        True          False      19m     DaemonSet "openshift-multus/network-metrics-daemon" is not available (awaiting 1 nodes)...
node-tuning                                4.10.10   True        False         False      15m     
openshift-apiserver                        4.10.10   True        False         False      4m45s   
openshift-controller-manager               4.10.10   True        False         False      15m     
openshift-samples                          4.10.10   True        False         False      7m37s   
operator-lifecycle-manager                 4.10.10   True        False         False      17m     
operator-lifecycle-manager-catalog         4.10.10   True        False         False      17m     
operator-lifecycle-manager-packageserver   4.10.10   True        False         False      11m     
service-ca                                 4.10.10   True        False         False      19m     
storage                                    4.10.10   True        False         False      19m  

The cluster operators are still rolling out so lets give it a few more minutes and we will check again:

$ oc get co
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.10.10   True        False         False      13m     
baremetal                                  4.10.10   True        False         False      31m     
cloud-controller-manager                   4.10.10   True        False         False      43m     
cloud-credential                           4.10.10   True        False         False      49m     
cluster-autoscaler                         4.10.10   True        False         False      31m     
config-operator                            4.10.10   True        False         False      33m     
console                                    4.10.10   True        False         False      19m     
csi-snapshot-controller                    4.10.10   True        False         False      33m     
dns                                        4.10.10   True        False         False      32m     
etcd                                       4.10.10   True        False         False      31m     
image-registry                             4.10.10   True        False         False      24m     
ingress                                    4.10.10   True        False         False      26m     
insights                                   4.10.10   True        False         False      27m     
kube-apiserver                             4.10.10   True        False         False      19m     
kube-controller-manager                    4.10.10   True        False         False      29m     
kube-scheduler                             4.10.10   True        False         False      28m     
kube-storage-version-migrator              4.10.10   True        False         False      33m     
machine-api                                4.10.10   True        False         False      22m     
machine-approver                           4.10.10   True        False         False      32m     
machine-config                             4.10.10   True        False         False      32m     
marketplace                                4.10.10   True        False         False      31m     
monitoring                                 4.10.10   True        False         False      20m     
network                                    4.10.10   True        False         False      34m     
node-tuning                                4.10.10   True        False         False      30m     
openshift-apiserver                        4.10.10   True        False         False      19m     
openshift-controller-manager               4.10.10   True        False         False      29m     
openshift-samples                          4.10.10   True        False         False      22m     
operator-lifecycle-manager                 4.10.10   True        False         False      32m     
operator-lifecycle-manager-catalog         4.10.10   True        False         False      32m     
operator-lifecycle-manager-packageserver   4.10.10   True        False         False      26m     
service-ca                                 4.10.10   True        False         False      34m     
storage                                    4.10.10   True        False         False      34m   

At this point our cluster installation is completed.  However I forgot to mention that while the web UI was up we should have ssh'd to the bootstrap node and shelled into the assisted installer container running to retrieve our kubeadmin password under the /data directory.   However I purposely skipped that part so I could show how we can just reset the kubeadmin password instead.

First I want to thank Andrew Block and his write up on how to do this here.  So lets go ahead and create the kubeadmin-rotate.go file here in the kuberotate directory we create:

 $ mkdir ~/kuberotate
$cd ~/kuberotate

$ cat << EOF > ./kubeadmin-rotate.go 
package main import ( "fmt" "crypto/rand" "golang.org/x/crypto/bcrypt" b64 "encoding/base64" "math/big" ) // generateRandomPasswordHash generates a hash of a random ASCII password // 5char-5char-5char-5char func generateRandomPasswordHash(length int) (string, string, error) { const ( lowerLetters = "abcdefghijkmnopqrstuvwxyz" upperLetters = "ABCDEFGHIJKLMNPQRSTUVWXYZ" digits = "23456789" all = lowerLetters + upperLetters + digits ) var password string for i := 0; i < length; i++ { n, err := rand.Int(rand.Reader, big.NewInt(int64(len(all)))) if err != nil { return "", "", err } newchar := string(all[n.Int64()]) if password == "" { password = newchar } if i < length-1 { n, err = rand.Int(rand.Reader, big.NewInt(int64(len(password)+1))) if err != nil { return "", "",err } j := n.Int64() password = password[0:j] + newchar + password[j:] } } pw := []rune(password) for _, replace := range []int{5, 11, 17} { pw[replace] = '-' } bytes, err := bcrypt.GenerateFromPassword([]byte(string(pw)), bcrypt.DefaultCost) if err != nil { return "", "",err } return string(pw), string(bytes), nil } func main() { password, hash, err := generateRandomPasswordHash(23) if err != nil { fmt.Println(err.Error()) return } fmt.Printf("Actual Password: %s\n", password) fmt.Printf("Hashed Password: %s\n", hash) fmt.Printf("Data to Change in Secret: %s\n", b64.StdEncoding.EncodeToString([]byte(hash))) } EOF

Next lets go ahead and initialize our go project:

$ go mod init kuberotate
go: creating new go.mod: module kuberotate

With the project initialized lets go ahead and pull in the module dependencies by executing a go mod tidy which will pull in the bcrypt module:

$ go mod tidy
go: finding module for package golang.org/x/crypto/bcrypt
go: found golang.org/x/crypto/bcrypt in golang.org/x/crypto v0.0.0-20220518034528-6f7dac969898

And finally since I just want to run the program instead of compile it I will just run a go run kubeadmin-rotate.go which will print out the password, a hashed password and a base64 encoded version of the hashed password:

$ go run kubeadmin-rotate.go 
Actual Password: gWdYr-62GLh-QIynG-Boj7n
Hashed Password: $2a$10$DN48Jp4YkuEEVMWZNyOR2.LkLn1ZZOJOtzR8c9detf1lVAQ2iVQGK
Data to Change in Secret: JDJhJDEwJERONDhKcDRZa3VFRVZNV1pOeU9SMi5Ma0xuMVpaT0pPdHpSOGM5ZGV0ZjFsVkFRMmlWUUdL

The last step is to patch the kubeadmin secret with the hashed password that was base64 encoded:

$ oc patch secret -n kube-system kubeadmin --type json -p '[{"op": "replace", "path": "/data/kubeadmin", "value": "JDJhJDEwJERONDhKcDRZa3VFRVZNV1pOeU9SMi5Ma0xuMVpaT0pPdHpSOGM5ZGV0ZjFsVkFRMmlWUUdL"}]'
secret/kubeadmin patched

Now we can go over to the OpenShift console and see if we can login.   And sure enough with the password we had above we can and confirm our 3 node OpenShift cluster installed by the agent installer is ready to be used for workloads:


Hopefully this blog was useful to provide a preview of what the agent installer will look like.  Keep in mind the code is under rapid development and so things could change but change is always good!