Saturday, February 15, 2020
OpenShift 4.3 Baremetal Deployment Troubleshooting Flow
The following is just a minimalist approach to troubleshooting an OpenShift 4.3 baremetal deployment. Obviously it does not cover everything in detail but it still provides an initial guide.
Tuesday, February 04, 2020
Carve up RedFish with Sushy-Tools
The Redfish API is a specification that delivers an industry standard protocol with a RESTful interface for the management of servers, storage, networking and converged infrastructure. Hardware vendors like Dell, HPE, Lenovo and Supermicro have all released servers with Redfish capabilities. However if you are a software engineer like myself that has barriers to getting access to Redfish capable hardware or hardware in general and find yourself often integrating and coding against virtual machines then Sushy tools might be exotic taste you crave.
If you are not familiar with Sushy-tools then you have come to the right place as this blog will show you how to acquire, configure and use Sushy-tools with your KVM virtual machines. The first thing we need to do is get Sushy tools and we have two options: install via Python pip or container. I am opting for using a container and the rest of this document will cover Sushy-tools from that standpoint although much of what we cover would apply to if it were installed with pip as well.
Installation/Configuration:
In order to use the Sushy tool container we first need to install podman:
# yum -y install podman
Next lets create a mount point for the container on our host:
# mkdir /opt/sushy
Now we can export our Sushy tool image to a variable and then create the container with podman:
# export SUSHY_TOOLS_IMAGE=${SUSHY_TOOLS_IMAGE:-"quay.io/metal3-io/sushy-tools"}
# podman create -d --net host --privileged --name sushy-tools -v "/opt/sushy":/root/sushy -v "/root/.ssh":/root/.ssh "${SUSHY_TOOLS_IMAGE}"
6ee305e4a435605ef8f0716878b6f06ad1eb38a12c8c896d779b65a4d1995ad3
Once the container is created we can start it up and confirm with a podman ps that it is running:
# podman start sushy-tools sushy-tools # sudo podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6ee305e4a435 quay.io/metal3-io/sushy-tools:latest /bin/sh -c sushy-... 23 seconds ago Up 4 seconds ago sushy-tools
Lets go ahead and peek inside and see what processes are running at startup. It will look similar to what we see below:
# podman exec -it 6ee305e4a435 /bin/bash # # ps -efl F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 4 S root 1 0 0 80 0 - 597 - 21:45 ? 00:00:00 /bin/sh -c sushy-emulator -i 0.0.0.0 -p 8000 --config /root/sushy/conf.py 4 S root 6 1 0 80 0 - 26468 x64_sy 21:45 ? 00:00:00 /usr/local/bin/python /usr/local/bin/sushy-emulator -i 0.0.0.0 -p 8000 --config /root/sushy/conf.py 4 S root 7 0 0 80 0 - 1438 - 21:47 pts/0 00:00:00 /bin/bash 0 R root 13 7 0 80 0 - 2348 - 21:48 pts/0 00:00:00 ps -efl
Now I have a libvirt KVM hypervisor host I would like to control the virtual machines on from a Redfish perspective. So the first thing I need to do is copy my public sshkey over to that host:
# ssh-copy-id root@192.168.0.5
Once the key is copied we can start a Sushy emulator process specific for that host with the following syntax. I am specifying a -i and ipaddress to ensure this port is listening so I can run my curl commands from remote hosts. Also note that you will need to open any firewall ports based on the port number specify in order to access the API.
# /usr/local/bin/sushy-emulator -i 192.168.0.11 -p 9005 --libvirt-uri qemu+ssh://root@192.168.0.5/system & [1] 18 # * Serving Flask app "sushy_tools.emulator.main" (lazy loading) * Environment: production WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Debug mode: off * Running on http://192.168.0.11:9005/ (Press CTRL+C to quit) # ps -efl F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 4 S root 1 0 0 80 0 - 597 - 21:54 ? 00:00:00 /bin/sh -c sushy-emulator -i 0.0.0.0 -p 8000 --config /root/sushy/conf.py 4 S root 6 1 0 80 0 - 44994 x64_sy 21:54 ? 00:00:00 /usr/local/bin/python /usr/local/bin/sushy-emulator -i 0.0.0.0 -p 8000 --config /root/sushy/conf.py 4 S root 10 0 0 80 0 - 1438 - 22:07 pts/0 00:00:00 /bin/bash 4 S root 18 10 0 80 0 - 26492 x64_sy 22:07 pts/0 00:00:00 /usr/local/bin/python /usr/local/bin/sushy-emulator -i 192.168.0.11 -p 9005 --libvirt-uri qemu+ssh://root@192.168.0.5/system 0 R root 19 10 0 80 0 - 2348 - 22:08 pts/0 00:00:00 ps -efl
After confirming our process is up and running we can move on to using Sushy tools to explore what virtual machines are on the hypervisor we specified in our previous command.
Using the Tools:
As stated before Sushy tools provides a Redfish emulated API for libvirt KVM. In the discussion above we successfully installed and configured Sushy tools and enabled them to access a hypervisor host at the address of 192.168.0.5. Now lets see what kind of information the Sushy tools will provide us.
If we run a curl command to retrieve the virtual machines on the host we can see there are currently 3 of them:
# curl 192.168.0.11:9005/redfish/v1/Systems
{
"@odata.type": "#ComputerSystemCollection.ComputerSystemCollection",
"Name": "Computer System Collection",
"Members@odata.count": 3,
"Members": [
{
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b"
},
{
"@odata.id": "/redfish/v1/Systems/04383b48-82c5-446a-a2db-5555b265ce75"
},
{
"@odata.id": "/redfish/v1/Systems/668d623c-2453-4c09-ac7d-8e509e91a337"
}
],
"@odata.context": "/redfish/v1/$metadata#ComputerSystemCollection.ComputerSystemCollection",
"@odata.id": "/redfish/v1/Systems",
"@Redfish.Copyright": "Copyright 2014-2016 Distributed Management Task Force, Inc. (DMTF). For the full DMTF copyright policy, see http://www.dmtf.org/about/policies/copyright."
}
Now lets explore the first system a little further and see what state it is in:
# curl 192.168.0.11:9005/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b
{
"@odata.type": "#ComputerSystem.v1_1_0.ComputerSystem",
"Id": "021359d3-9335-4e0c-800c-ad40232d804b",
"Name": "Jenkins",
"UUID": "021359d3-9335-4e0c-800c-ad40232d804b",
"Status": {
"State": "Enabled",
"Health": "OK",
"HealthRollUp": "OK"
},
"PowerState": "Off",
"Boot": {
"BootSourceOverrideEnabled": "Continuous"
},
"ProcessorSummary": {
"Count": 4,
"Status": {
"State": "Enabled",
"Health": "OK",
"HealthRollUp": "OK"
}
},
"MemorySummary": {
"TotalSystemMemoryGiB": 14,
"Status": {
"State": "Enabled",
"Health": "OK",
"HealthRollUp": "OK"
}
},
"Bios": {
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/BIOS"
},
"Processors": {
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/Processors"
},
"Memory": {
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/Memory"
},
"EthernetInterfaces": {
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/EthernetInterfaces"
},
"SimpleStorage": {
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/SimpleStorage"
},
"Storage": {
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/Storage"
},
"IndicatorLED": "Lit",
"Links": {
"Chassis": [
{
"@odata.id": "/redfish/v1/Chassis/15693887-7984-9484-3272-842188918912"
}
],
"ManagedBy": [
{
"@odata.id": "/redfish/v1/Managers/58893887-8974-2487-2389-841168418919"
}
]
},
"Actions": {
"#ComputerSystem.Reset": {
"target": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/Actions/ComputerSystem.Reset",
"ResetType@Redfish.AllowableValues": [
"On",
"ForceOff",
"GracefulShutdown",
"GracefulRestart",
"ForceRestart",
"Nmi",
"ForceOn"
]
}
},
"@odata.context": "/redfish/v1/$metadata#ComputerSystem.ComputerSystem",
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b",
"@Redfish.Copyright": "Copyright 2014-2016 Distributed Management Task Force, Inc. (DMTF). For the full DMTF copyright policy, see http://www.dmtf.org/about/policies/copyright."
}
We can see from the results we can learn quite a bit about the virtual machine like its name, status, power state, memory and cpu. I noticed this system was off and it appears to be the Jenkins CI/CD host. So lets show how Sushy tools can be useful and turn this machine back on:
# curl -d '{"ResetType":"On"}' -H "Content-Type: application/json" -X POST http://192.168.0.11:9005/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/Actions/ComputerSystem.Reset
After a few seconds lets confirm its on by running the following command we ran earlier:
# curl 192.168.0.11:9005/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b
{
"@odata.type": "#ComputerSystem.v1_1_0.ComputerSystem",
"Id": "021359d3-9335-4e0c-800c-ad40232d804b",
"Name": "Jenkins",
"UUID": "021359d3-9335-4e0c-800c-ad40232d804b",
"Status": {
"State": "Enabled",
"Health": "OK",
"HealthRollUp": "OK"
},
"PowerState": "On",
"Boot": {
"BootSourceOverrideEnabled": "Continuous"
},
"ProcessorSummary": {
"Count": 4,
"Status": {
"State": "Enabled",
"Health": "OK",
"HealthRollUp": "OK"
}
},
"MemorySummary": {
"TotalSystemMemoryGiB": 14,
"Status": {
"State": "Enabled",
"Health": "OK",
"HealthRollUp": "OK"
}
},
"Bios": {
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/BIOS"
},
"Processors": {
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/Processors"
},
"Memory": {
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/Memory"
},
"EthernetInterfaces": {
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/EthernetInterfaces"
},
"SimpleStorage": {
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/SimpleStorage"
},
"Storage": {
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/Storage"
},
"IndicatorLED": "Lit",
"Links": {
"Chassis": [
{
"@odata.id": "/redfish/v1/Chassis/15693887-7984-9484-3272-842188918912"
}
],
"ManagedBy": [
{
"@odata.id": "/redfish/v1/Managers/58893887-8974-2487-2389-841168418919"
}
]
},
"Actions": {
"#ComputerSystem.Reset": {
"target": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/Actions/ComputerSystem.Reset",
"ResetType@Redfish.AllowableValues": [
"On",
"ForceOff",
"GracefulShutdown",
"GracefulRestart",
"ForceRestart",
"Nmi",
"ForceOn"
]
}
},
"@odata.context": "/redfish/v1/$metadata#ComputerSystem.ComputerSystem",
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b",
"@Redfish.Copyright": "Copyright 2014-2016 Distributed Management Task Force, Inc. (DMTF). For the full DMTF copyright policy, see http://www.dmtf.org/about/policies/copyright."
}
Sure enough the system is now powered on and running all via the Sushy tools. While we are here lets take a look at the systems storage for fun:
# curl 192.168.0.11:9005/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/SimpleStorage
{
"@odata.type": "#SimpleStorageCollection.SimpleStorageCollection",
"Name": "Simple Storage Collection",
"Members@odata.count": 1,
"Members": [
{
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/SimpleStorage/virtio"
}
],
"Oem": {},
"@odata.context": "/redfish/v1/$metadata#SimpleStorageCollection.SimpleStorageCollection",
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/SimpleStorage"
}
The above command tells us there is one virtio disk. Now if we take the data id of that disk and curl on that we can get even further details about the disk, its capacity and status:
# curl 192.168.0.11:9005/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/SimpleStorage/virtio
{
"@odata.type": "#SimpleStorage.v1_2_0.SimpleStorage",
"Id": "virtio",
"Name": "virtio Controller",
"Devices": [
{
"@odata.type": "#SimpleStorage.v1_1_0.Device",
"Name": "Jenkins.qcow2",
"CapacityBytes": 64424509440,
"Status": {
"@odata.type": "#Resource.Status",
"State": "Enabled",
"Health": "OK"
}
}
],
"@odata.context": "/redfish/v1/$metadata#SimpleStorage.SimpleStorage",
"@odata.id": "/redfish/v1/Systems/021359d3-9335-4e0c-800c-ad40232d804b/SimpleStorage/virtio"
}
As you can see Sushy tools provides a way to enable the Redfish API for libvirt KVM virtual machines. Providing the ability to develop where the Redfish API is required without the need to invest in enterprise hardware.
Sunday, January 26, 2020
Hive on Minishift Deploying OpenShift 4.3 on Baremetal
Sometimes complexity can sting but this blog on using Hive to deploy OpenShift 4.3 on baremetal will make it sweeter than honey.
For those not familiar with Hive it is a API driven OpenShift 4 cluster and provisioning tool. Hive itself is an operator that runs in an existing OpenShift or Kubernetes cluster and allow for the provisioning and initial configuration of one or more OpenShift clusters.
This blog will walk through the current process of how to use Hive to deploy OpenShift 4.3 on baremetal nodes. Since I do not have enough resources to have two fully functional OpenShift clusters I will be deploying Hive in Minishift and then deploying OpenShift 4.3 on bare metal.
Minishift Installation & Configuration:
Minishift and Hive have a few requirements in order for them to work. The first thing we need to do on this fresh install of Red Hat Enterprise Linux 8 is install some package dependencies:
$ sudo yum -y install qemu-kvm* libvirt go git make
Once libvirt is installed the next step is to enable the service and configure the user we are using to be part of the libvirt group:
$ sudo systemctl enable --now libvirtd $ sudo usermod -a -G libvirt bschmaus $ newgrp - libvirt $ groups libvirt wheel bschmaus
Next lets confirm the host supports virtualization using virt-host-validate. All of the checks should come back as a PASS except IOMMU support. If they do not ensure the virtualized extensions in the BIOS are enabled. If the system is a virtual machine ensure nested virtualization is enabled on the hypervisor host and that the virtual machine has the CPU set to host-passthrough.
$ sudo virt-host-validate QEMU: Checking for hardware virtualization : PASS QEMU: Checking if device /dev/kvm exists : PASS QEMU: Checking if device /dev/kvm is accessible : PASS QEMU: Checking if device /dev/vhost-net exists : PASS QEMU: Checking if device /dev/net/tun exists : PASS QEMU: Checking for cgroup 'cpu' controller support : PASS QEMU: Checking for cgroup 'cpuacct' controller support : PASS QEMU: Checking for cgroup 'cpuset' controller support : PASS QEMU: Checking for cgroup 'memory' controller support : PASS QEMU: Checking for cgroup 'devices' controller support : PASS QEMU: Checking for cgroup 'blkio' controller support : PASS QEMU: Checking for device assignment IOMMU support : WARN (No ACPI DMAR table found, IOMMU either disabled in BIOS or not supported by this hardware platform)
Next lets grab the docker-machine binary:
$ curl --location https://github.com/docker/machine/releases/download/v0.16.1/docker-machine-Linux-`uname -i` > docker-machine $ chmod +x docker-machine $ sudo mv docker-machine /usr/local/bin/
We will also need the docker-machine-driver-kvm binary as well:
$ curl --location https://github.com/dhiltgen/docker-machine-kvm/releases/download/v0.10.0/docker-machine-driver-kvm-centos7 > docker-machine-driver-kvm $ chmod +x docker-machine-driver-kvm $ sudo mv docker-machine-driver-kvm /usr/local/bin/
Finally we can pull down the Minishift binary and set a soft link to minishift:
$ pwd /home/bschmaus $ mkdir bin $ wget -qO- https://github.com/minishift/minishift/releases/download/v1.34.2/minishift-1.34.2-linux-amd64.tgz | tar --extract --gzip --verbose -C ~/bin/ minishift-1.34.2-linux-amd64/ minishift-1.34.2-linux-amd64/LICENSE minishift-1.34.2-linux-amd64/README.ado $ ln -s ~/bin/minishift-1.34.2-linux-amd64/minishift ~/bin/minishift
Before we start Minishift we need to enable the admissions webhook addon that is required for the Hive deployment:
$ minishift addons enable admissions-webhook
At this point we are ready to launch Minishift. Many documents will show just executing minishift start but in my case I did not want to use the defaults for cpu/memory/disk resources and therefore I have specified how I want the Minishift virtual environment to be configured. The process of starting will take some time.
$ minishift start --cpus 6 --memory 12GB --disk-size 30GB
-- Starting profile 'minishift'
-- Check if deprecated options are used ... OK
-- Checking if https://github.com is reachable ... OK
-- Checking if requested OpenShift version 'v3.11.0' is valid ... OK
-- Checking if requested OpenShift version 'v3.11.0' is supported ... OK
-- Checking if requested hypervisor 'kvm' is supported on this platform ... OK
-- Checking if KVM driver is installed ...
Driver is available at /usr/local/bin/docker-machine-driver-kvm ...
Checking driver binary is executable ... OK
-- Checking if Libvirt is installed ... OK
-- Checking if Libvirt default network is present ... OK
-- Checking if Libvirt default network is active ... OK
-- Checking the ISO URL ... OK
-- Downloading OpenShift binary 'oc' version 'v3.11.0'
-- Downloading OpenShift v3.11.0 checksums ... OK
-- Checking if provided oc flags are supported ... OK
-- Starting the OpenShift cluster using 'kvm' hypervisor ...
-- Minishift VM will be configured with ...
Memory: 12 GB
vCPUs : 6
Disk size: 30 GB
-- Downloading ISO 'https://github.com/minishift/minishift-centos-iso/releases/download/v1.16.0/minishift-centos7.iso'
-- Starting Minishift VM ................. OK
-- Checking for IP address ... OK
-- Checking for nameservers ... OK
-- Checking if external host is reachable from the Minishift VM ...
Pinging 8.8.8.8 ... OK
-- Checking HTTP connectivity from the VM ...
Retrieving http://minishift.io/index.html ... OK
-- Checking if persistent storage volume is mounted ... OK
-- Checking available disk space ... 1% used OK
-- Writing current configuration for static assignment of IP address ... WARN
Importing 'openshift/origin-control-plane:v3.11.0' CACHE MISS
Importing 'openshift/origin-docker-registry:v3.11.0' CACHE MISS
Importing 'openshift/origin-haproxy-router:v3.11.0' CACHE MISS
-- OpenShift cluster will be configured with ...
Version: v3.11.0
-- Pulling the OpenShift Container Image ................. OK
-- Copying oc binary from the OpenShift container image to VM ... OK
-- Starting OpenShift cluster .........................................................
Getting a Docker client ...
Checking if image openshift/origin-control-plane:v3.11.0 is available ...
Pulling image openshift/origin-cli:v3.11.0
E0124 01:04:01.549540 2696 helper.go:173] Reading docker config from /home/docker/.docker/config.json failed: open /home/docker/.docker/config.json: no such file or directory, will attempt to pull image docker.io/openshift/origin-cli:v3.11.0 anonymously
Image pull complete
Pulling image openshift/origin-node:v3.11.0
E0124 01:04:02.757778 2696 helper.go:173] Reading docker config from /home/docker/.docker/config.json failed: open /home/docker/.docker/config.json: no such file or directory, will attempt to pull image docker.io/openshift/origin-node:v3.11.0 anonymously
Pulled 5/6 layers, 85% complete
Pulled 6/6 layers, 100% complete
Extracting
Image pull complete
Checking type of volume mount ...
Determining server IP ...
Using public hostname IP 192.168.42.173 as the host IP
Checking if OpenShift is already running ...
Checking for supported Docker version (=>1.22) ...
Checking if insecured registry is configured properly in Docker ...
Checking if required ports are available ...
Checking if OpenShift client is configured properly ...
Checking if image openshift/origin-control-plane:v3.11.0 is available ...
Starting OpenShift using openshift/origin-control-plane:v3.11.0 ...
I0124 01:04:20.142573 2696 config.go:40] Running "create-master-config"
I0124 01:04:23.992407 2696 config.go:46] Running "create-node-config"
I0124 01:04:25.078851 2696 flags.go:30] Running "create-kubelet-flags"
I0124 01:04:25.640019 2696 run_kubelet.go:49] Running "start-kubelet"
I0124 01:04:25.908558 2696 run_self_hosted.go:181] Waiting for the kube-apiserver to be ready ...
I0124 01:04:53.925511 2696 interface.go:26] Installing "kube-proxy" ...
I0124 01:04:53.925549 2696 interface.go:26] Installing "kube-dns" ...
I0124 01:04:53.925566 2696 interface.go:26] Installing "openshift-service-cert-signer-operator" ...
I0124 01:04:53.925581 2696 interface.go:26] Installing "openshift-apiserver" ...
I0124 01:04:53.925647 2696 apply_template.go:81] Installing "kube-dns"
I0124 01:04:53.925666 2696 apply_template.go:81] Installing "openshift-service-cert-signer-operator"
I0124 01:04:53.925651 2696 apply_template.go:81] Installing "openshift-apiserver"
I0124 01:04:53.925651 2696 apply_template.go:81] Installing "kube-proxy"
I0124 01:04:57.836630 2696 interface.go:41] Finished installing "kube-proxy" "kube-dns" "openshift-service-cert-signer-operator" "openshift-apiserver"
I0124 01:07:10.862452 2696 run_self_hosted.go:242] openshift-apiserver available
I0124 01:07:10.863431 2696 interface.go:26] Installing "openshift-controller-manager" ...
I0124 01:07:10.863453 2696 apply_template.go:81] Installing "openshift-controller-manager"
I0124 01:07:13.147594 2696 interface.go:41] Finished installing "openshift-controller-manager"
Adding default OAuthClient redirect URIs ...
Adding centos-imagestreams ...
Adding router ...
Adding persistent-volumes ...
Adding web-console ...
Adding registry ...
Adding sample-templates ...
I0124 01:07:13.167941 2696 interface.go:26] Installing "centos-imagestreams" ...
I0124 01:07:13.167974 2696 interface.go:26] Installing "openshift-router" ...
I0124 01:07:13.167988 2696 interface.go:26] Installing "persistent-volumes" ...
I0124 01:07:13.168012 2696 interface.go:26] Installing "openshift-web-console-operator" ...
I0124 01:07:13.168024 2696 interface.go:26] Installing "openshift-image-registry" ...
I0124 01:07:13.168035 2696 interface.go:26] Installing "sample-templates" ...
I0124 01:07:13.168129 2696 apply_list.go:67] Installing "centos-imagestreams"
I0124 01:07:13.168532 2696 interface.go:26] Installing "sample-templates/rails quickstart" ...
I0124 01:07:13.168557 2696 interface.go:26] Installing "sample-templates/sample pipeline" ...
I0124 01:07:13.168573 2696 interface.go:26] Installing "sample-templates/mongodb" ...
I0124 01:07:13.168586 2696 interface.go:26] Installing "sample-templates/postgresql" ...
I0124 01:07:13.168599 2696 interface.go:26] Installing "sample-templates/cakephp quickstart" ...
I0124 01:07:13.168614 2696 interface.go:26] Installing "sample-templates/django quickstart" ...
I0124 01:07:13.168628 2696 interface.go:26] Installing "sample-templates/nodejs quickstart" ...
I0124 01:07:13.168643 2696 interface.go:26] Installing "sample-templates/jenkins pipeline ephemeral" ...
I0124 01:07:13.168653 2696 apply_list.go:67] Installing "sample-templates/rails quickstart"
I0124 01:07:13.168656 2696 interface.go:26] Installing "sample-templates/mariadb" ...
I0124 01:07:13.168935 2696 interface.go:26] Installing "sample-templates/mysql" ...
I0124 01:07:13.168948 2696 interface.go:26] Installing "sample-templates/dancer quickstart" ...
I0124 01:07:13.169015 2696 apply_list.go:67] Installing "sample-templates/dancer quickstart"
I0124 01:07:13.169327 2696 apply_list.go:67] Installing "sample-templates/mariadb"
I0124 01:07:13.169432 2696 apply_template.go:81] Installing "openshift-web-console-operator"
I0124 01:07:13.169635 2696 apply_list.go:67] Installing "sample-templates/postgresql"
I0124 01:07:13.169931 2696 apply_list.go:67] Installing "sample-templates/mongodb"
I0124 01:07:13.170038 2696 apply_list.go:67] Installing "sample-templates/mysql"
I0124 01:07:13.170098 2696 apply_list.go:67] Installing "sample-templates/cakephp quickstart"
I0124 01:07:13.170197 2696 apply_list.go:67] Installing "sample-templates/sample pipeline"
I0124 01:07:13.170038 2696 apply_list.go:67] Installing "sample-templates/jenkins pipeline ephemeral"
I0124 01:07:13.170332 2696 apply_list.go:67] Installing "sample-templates/django quickstart"
I0124 01:07:13.170537 2696 apply_list.go:67] Installing "sample-templates/nodejs quickstart"
I0124 01:07:18.883279 2696 interface.go:41] Finished installing "sample-templates/rails quickstart" "sample-templates/sample pipeline" "sample-templates/mongodb" "sample-templates/postgresql" "sample-templates/cakephp quickstart" "sample-templates/django quickstart" "sample-templates/nodejs quickstart" "sample-templates/jenkins pipeline ephemeral" "sample-templates/mariadb" "sample-templates/mysql" "sample-templates/dancer quickstart"
I0124 01:08:08.797919 2696 interface.go:41] Finished installing "centos-imagestreams" "openshift-router" "persistent-volumes" "openshift-web-console-operator" "openshift-image-registry" "sample-templates"
Login to server ...
Creating initial project "myproject" ...
Server Information ...
OpenShift server started.
The server is accessible via web console at:
https://192.168.42.173:8443/console
You are logged in as:
User: developer
Password:
To login as administrator:
oc login -u system:admin
-- Applying addon 'admissions-webhook':..
-- Restarting Openshift API server ...
-- Restarting Kube API server .....
-- Successfully installed addon admission-webhook ... OK
-- Exporting of OpenShift images is occuring in background process with pid 3668.
Once Minishift is up and running we can now set the minishift oc environment up and then login to the OpenShift cluster Minishift is providing:
$ minishift oc-env export PATH="/home/bschmaus/.minishift/cache/oc/v3.11.0/linux:$PATH" # Run this command to configure your shell: # eval $(minishift oc-env) $ eval `minishift oc-env` $ oc login -u system:admin Logged into "https://192.168.42.173:8443" as "system:admin" using existing credentials. You have access to the following projects and can switch between them with 'oc project': default kube-dns kube-proxy kube-public kube-system * myproject openshift openshift-apiserver openshift-controller-manager openshift-core-operators openshift-infra openshift-node openshift-service-cert-signer openshift-web-console Using project "myproject".
At this point we are now ready to move on to the Hive portion of installing and configuring.
Hive Installation & Configuration:
Now that we have Minishift up and running we can deploy Hive. Hive requires a couple of additional components: Mockgen and Kustomize. MockGen generates mock implementations of Go interfaces and Kustomize traverses a Kubernetes/OpenShift manifest to add, remove or update configuration options without forking. Below we will setup the Go path, gather Mockgen, Kustomize and Hive:
$ mkdir ~/go/
$ export PATH=~/go/bin:$PATH
$ export GOROOT=/usr/lib/golang
$ export GOPATH=~/go
$ go get -v github.com/openshift/hive
github.com/openshift/hive (download)
package NaN,NaN github.com/openshift/hive: no Go files in /home/bschmaus/go/src/github.com/openshift/hive
$ go get -v github.com/golang/mock/mockgen
github.com/golang/mock (download)
Fetching https://golang.org/x/tools/go/packages?go-get=1
Parsing meta tags from https://golang.org/x/tools/go/packages?go-get=1 (status code 200)
get "golang.org/x/tools/go/packages": found meta tag get.metaImport{Prefix:"golang.org/x/tools", VCS:"git", RepoRoot:"https://go.googlesource.com/tools"} at https://golang.org/x/tools/go/packages?go-get=1
get "golang.org/x/tools/go/packages": verifying non-authoritative meta tag
Fetching https://golang.org/x/tools?go-get=1
Parsing meta tags from https://golang.org/x/tools?go-get=1 (status code 200)
golang.org/x/tools (download)
golang.org/x/tools/internal/packagesinternal
github.com/golang/mock/mockgen/model
golang.org/x/tools/go/internal/packagesdriver
golang.org/x/tools/go/internal/gcimporter
golang.org/x/tools/go/gcexportdata
golang.org/x/tools/go/packages
github.com/golang/mock/mockgen
$ cd $GOPATH/src/github.com/openshift/hive/
$ export PATH=$PATH:$GOPATH/bin
$ curl -sL https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv3.3.0/kustomize_v3.3.0_linux_amd64.tar.gz | sudo tar xvz -C /usr/local/bin/
kustomize
$ git fetch -a -v
From https://github.com/openshift/hive
= [up to date] v1 -> origin/v1
= [up to date] master -> origin/master
$ git checkout v1 Already on 'v1' Your branch is up to date with 'origin/v1'.
At this point we are ready to make and deploy Hive on our Minishift cluster:
$ DEPLOY_IMAGE=registry.svc.ci.openshift.org/openshift/hivev1:hive make deploy # The apis-path is explicitly specified so that CRDs are not created for v1alpha1 go run tools/vendor/sigs.k8s.io/controller-tools/cmd/controller-gen/main.go crd --apis-path=pkg/apis/hive/v1 CRD files generated, files can be found under path /home/bschmaus/go/src/github.com/openshift/hive/config/crds. oc apply -f config/crds customresourcedefinition.apiextensions.k8s.io/checkpoints.hive.openshift.io created customresourcedefinition.apiextensions.k8s.io/clusterdeployments.hive.openshift.io created customresourcedefinition.apiextensions.k8s.io/clusterdeprovisions.hive.openshift.io created customresourcedefinition.apiextensions.k8s.io/clusterimagesets.hive.openshift.io created customresourcedefinition.apiextensions.k8s.io/clusterprovisions.hive.openshift.io created customresourcedefinition.apiextensions.k8s.io/clusterstates.hive.openshift.io created customresourcedefinition.apiextensions.k8s.io/dnszones.hive.openshift.io created customresourcedefinition.apiextensions.k8s.io/hiveconfigs.hive.openshift.io created customresourcedefinition.apiextensions.k8s.io/machinepools.hive.openshift.io created customresourcedefinition.apiextensions.k8s.io/selectorsyncidentityproviders.hive.openshift.io created customresourcedefinition.apiextensions.k8s.io/selectorsyncsets.hive.openshift.io created customresourcedefinition.apiextensions.k8s.io/syncidentityproviders.hive.openshift.io created customresourcedefinition.apiextensions.k8s.io/syncsets.hive.openshift.io created customresourcedefinition.apiextensions.k8s.io/syncsetinstances.hive.openshift.io created go generate ./pkg/... ./cmd/... hack/update-bindata.sh # Deploy the operator manifests: mkdir -p overlays/deploy cp overlays/template/kustomization.yaml overlays/deploy cd overlays/deploy && kustomize edit set image registry.svc.ci.openshift.org/openshift/hive-v4.0:hive=registry.svc.ci.openshift.org/openshift/hivev1:hive kustomize build overlays/deploy | oc apply -f - namespace/hive created serviceaccount/hive-operator created clusterrole.rbac.authorization.k8s.io/hive-frontend created clusterrole.rbac.authorization.k8s.io/hive-operator-role created clusterrole.rbac.authorization.k8s.io/manager-role created clusterrole.rbac.authorization.k8s.io/system:openshift:hive:hiveadmission created rolebinding.rbac.authorization.k8s.io/extension-server-authentication-reader-hiveadmission created clusterrolebinding.rbac.authorization.k8s.io/auth-delegator-hiveadmission created clusterrolebinding.rbac.authorization.k8s.io/hive-frontend created clusterrolebinding.rbac.authorization.k8s.io/hive-operator-rolebinding created clusterrolebinding.rbac.authorization.k8s.io/hiveadmission-hive-hiveadmission created clusterrolebinding.rbac.authorization.k8s.io/manager-rolebinding created deployment.apps/hive-operator created rm -rf overlays/deploy
After a few minutes we can use the oc get pods command to see if our Hive deployment is running:
$ oc get pods -n hive NAME READY STATUS RESTARTS AGE hive-controllers-84889bb864-74qc2 1/1 Running 0 45s hive-operator-86bf7c785f-qw74r 1/1 Running 0 1m hiveadmission-56c86f475d-2nbj7 1/1 Running 0 41s hiveadmission-56c86f475d-bf268 1/1 Running 0 41s
At this point we can move onto the steps required to configure and deploy our OpenShift 4.3 baremetal cluster.
OpenShift 4.3 Baremetal Deployment with Hive:
Currently we have a running Minishift cluster with Hive deployed as a operator inside. Now we can begin the process of setting up the baremetal host that will be used to launch the bootstrap deployment virtual machine for the OpenShift 4.3 that will then deploy the Openshift master nodes on baremetal hosts.
Many of the steps needed to prepare the deployment host are documented in a previous blog of mine here: Deploying OpenShift IPI Baremetal Disconnected (this will be referred as "the blog above" in later context). Follow the requirements in the blog above up to the point of actually deploying the cluster.
One item the blog above does not include is ensuring that the deploy user being used to do the deployment can ssh into the provisioning node without a password from the provisioning node. To achieve this login to the provisioning node and run the following commands:
$ ssh-copy-id bschmaus@192.168.0.246 /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/bschmaus/.ssh/id_rsa.pub" The authenticity of host '192.168.0.246 (192.168.0.246)' can't be established. ECDSA key fingerprint is SHA256:xyfms8ClEh+s4n+Vwbgsz9JGzXSN6X8sZDfiCVjpaOU. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
Lets return back to the RHEL 8 Minishift host command prompt where we had setup our oc environment.
Here we will take the install-config.yaml that was generated from the blog above and add the following line adjusting the user and ipaddress for the environment:
libvirtURI: qemu+ssh://bschmaus@192.168.0.246/system
The install-config.yaml should then look something like the following (adjusted based on environment of course) and be saved on the Minishift host:
apiVersion: v1
baseDomain: schmaustech.com
metadata:
name: kni5
networking:
networkType: OpenShiftSDN
machineCIDR: 192.168.0.0/24
compute:
- name: worker
replicas: 0
controlPlane:
name: master
replicas: 3
platform:
baremetal: {}
platform:
baremetal:
libvirtURI: qemu+ssh://bschmaus@192.168.0.246/system
apiVIP: 192.168.0.199
ingressVIP: 192.168.0.197
dnsVIP: 192.168.0.198
bootstrapOSImage: http://172.22.0.1/images/rhcos-43.81.202001142154.0-qemu.x86_64.qcow2.gz?sha256=6dbd3513a824cfe6de157e5d86972c2005b23d71bd8b3a61548f7a1f0a516b68
clusterOSImage: http://172.22.0.1/images/rhcos-43.81.202001142154.0-openstack.x86_64.qcow2.gz?sha256=a1bda656fa0892f7b936fdc6b6a6086bddaed5dafacedcd7a1e811abb78fe3b0
hosts:
- name: master-0
role: master
bmc:
address: ipmi://192.168.0.11:6241
username: admin
password: password
bootMACAddress: 52:54:00:3d:04:ae
hardwareProfile: default
- name: master-1
role: master
bmc:
address: ipmi://192.168.0.11:6261
username: admin
password: password
bootMACAddress: 52:54:00:37:80:89
hardwareProfile: default
- name: master-2
role: master
bmc:
address: ipmi://192.168.0.11:6231
username: admin
password: password
bootMACAddress: 52:54:00:4f:e4:7c
hardwareProfile: default
sshKey: 'INSERT YOUR SSH KEY HERE'
imageContentSources:
- mirrors:
- rhel8-ocp-auto.schmaustech.com:5000/ocp4/openshift4
source: quay.io/openshift-release-dev/ocp-v4.0-art-dev
- mirrors:
- rhel8-ocp-auto.schmaustech.com:5000/ocp4/openshift4
source: registry.svc.ci.openshift.org/ocp/release
pullSecret: ' INSERT YOUR PULL SECRET HERE'
additionalTrustBundle: |
-----BEGIN CERTIFICATE-----
MIIF9zCCA9+gAwIBAgIUeFymZ6Et3peUhTizIH5aeHXh7ikwDQYJKoZIhvcNAQEL
BQAwgYoxCzAJBgNVBAYTAlVTMRIwEAYDVQQIDAlNaW5uZXNvdGExFjAUBgNVBAcM
DUJyb29rbHluIFBhcmsxEDAOBgNVBAoMB1JlZCBIYXQxFDASBgNVBAsMC0VuZ2lu
ZWVyaW5nMScwJQYDVQQDDB5yaGVsOC1vY3AtYXV0by5zY2htYXVzdGVjaC5jb20w
HhcNMjAwMTI1MTQzOTQ0WhcNMjEwMTI0MTQzOTQ0WjCBijELMAkGA1UEBhMCVVMx
EjAQBgNVBAgMCU1pbm5lc290YTEWMBQGA1UEBwwNQnJvb2tseW4gUGFyazEQMA4G
A1UECgwHUmVkIEhhdDEUMBIGA1UECwwLRW5naW5lZXJpbmcxJzAlBgNVBAMMHnJo
ZWw4LW9jcC1hdXRvLnNjaG1hdXN0ZWNoLmNvbTCCAiIwDQYJKoZIhvcNAQEBBQAD
ggIPADCCAgoCggIBAL28gRfNtpezIE7JpuMoSreaCSnvAf0+9s92WVgn0Q5Ke3Iu
OV997lsSJP4oHfSbR2J0ApsAsoPbp2gzOwHvrjcDff4xsBQBY3Juy6PKgBKPbgUY
jp4b+BEnoI8/OCRzvVHCHWPRus+N4FF2nfv0T2qjiNEQ6vITbZ/kJu/WofVDVKzO
Ro5nbppTuZ5psMa6SR8/5KO2IT+ULa6+Pwpwj7Vw5fj1NpBNlgJtGZUm/07t1BH/
U6VNtRfoJbGFvBC8IJoy7hIn74RI7xFvICXvaJ6VE0OIwNsOUvwN/Gn9jp2I+tBg
K9tleq2boVahql3Or18bai8AqjuCsYOOaPsxjfnKju5/C13rTBkr9KTTrHlf+c9N
+kFkUqnqtEymSa2Fsn0HbJfp8f6mHtaszFv6lxBElkfeMLvvF+P7MWxTJOMTaB83
1lQjlCJNVg8KQphd31V5A4FwlLGvUeM3siph2YMQTTxF/maEjppfPWPu0eZiUbxE
11Zp2u87gtO9MCABjpt1xiMCvn2R2MxUUfD0tL1P6wws2bcq2Cbo0Z0I55DEjAG/
wRjVFa7s/qQYG9EP6HTIHKWhrAI2peOfDbn9ADb+YMnIik4Vsmk4omo9yhDqGbTO
Dl5FrfShLUzeWeou5CkPxxed0JKsvaejh4+45A42H4//V9dIfOPhzMJhUqSVAgMB
AAGjUzBRMB0GA1UdDgQWBBQdyoUTFGBK4OaPG+yxkRldGcNk3jAfBgNVHSMEGDAW
gBQdyoUTFGBK4OaPG+yxkRldGcNk3jAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3
DQEBCwUAA4ICAQCc9Fj3bm+wbHlXEPDNWBM1Oe2J3sN11N4z6DaozAeEjCnJb4SE
/dluktdEkF325qgSr+Wxwi0W/mdUXPXQv/bjlcH+SMMicp7gw6tYPze6RaEk1REk
oSuxAyHyiU081o+lbVQP/ec5CzMgHqcjwBD4eId1ijbAPGrXO+3mVGBsrsqbBNF/
HCDiKw3SnGC+z8Kcl20Qn08FvioNNdGCWQmH2e3gKK7mz4KtUtq2qoduGsw/12QC
6teSoyUxlXtPpB5mpNHzTSxkyn1yiX7edPdadnNtDGr6FSDFvGLtHusgITwQ4Dt3
FQyjCcRQ8/WllYJbE8fb+7wsx+xc7pTSHj7I73giYtig/tFmE6us/9TNCt9mSIzZ
c29BHmVtaCnmtTlPgFzJYXAYBc3XmA+Zfu6u4G5e7b+IkrOD2w4BK4D2zJaHnhzO
ol+rnN7Ff6hZSjgZg2fPDq3L6AMvqpbKcFe1FzCy1FsHcYoW1hKWySD6KOJ2GtfQ
lcTfxMR+c4bG5y6KFgQ9fJOJYLoCo3y8gaFDPQ1Xycie02zWqdH42dstyyPug/VZ
eIcrl3yOSdEzN5wx4Cu0Y8CrSCnf5ZnCuPU3DcD2JpSZRcKaJVX7ITuW2iNrlgkO
flzKt5T35OKENQAvs7GxbEEJvFMruHnHp73SivVBqBnuZv31Q7oLzgQnRw==
-----END CERTIFICATE-----
Now lets take the contents of the metal3-config.yaml created from the blog above and fashion them into a file called installer-manifests.yaml saving it on the Minishift host as well. The contents will look similar to the following (adjusted based on environment of course):
apiVersion: v1
data:
99_metal3-config.yaml: |
kind: ConfigMap
apiVersion: v1
metadata:
name: metal3-config
namespace: openshift-machine-api
data:
http_port: "6180"
provisioning_interface: "ens3"
provisioning_ip: "172.22.0.3/24"
dhcp_range: "172.22.0.10,172.22.0.100"
deploy_kernel_url: "http://172.22.0.3:6180/images/ironic-python-agent.kernel"
deploy_ramdisk_url: "http://172.22.0.3:6180/images/ironic-python-agent.initramfs"
ironic_endpoint: "http://172.22.0.3:6385/v1/"
ironic_inspector_endpoint: "http://172.22.0.3:5050/v1/"
cache_url: "http://192.168.0.246/images"
rhcos_image_url: "http://172.22.0.1/images/rhcos-43.81.202001142154.0-openstack.x86_64.qcow2.gz?sha256=2da72a1eaee005458014cfbab93f77c459b8c63ff110f0d4cbf4c13e7001de68"
kind: ConfigMap
metadata:
name: kni5-install-manifests
Finally lets construct the clusterdeploy.yaml that will be used to initiate the deployment via Hive of our cluster. Again make sure to adjust the pull secret, ssh private key and known host fields here to suit environment:
apiVersion: v1
items:
- apiVersion: v1
kind: Secret
metadata:
creationTimestamp: null
name: kni5-pull-secret
stringData:
.dockerconfigjson: 'PULL SECRETS HERE'
type: kubernetes.io/dockerconfigjson
- apiVersion: v1
kind: Secret
metadata:
creationTimestamp: null
name: kni5-ssh-private-key
stringData:
ssh-privatekey: |
-----BEGIN RSA PRIVATE KEY-----
-----END RSA PRIVATE KEY-----
type: Opaque
- apiVersion: hive.openshift.io/v1
kind: ClusterDeployment
metadata:
creationTimestamp: null
name: kni5
annotations:
hive.openshift.io/try-install-once: "true"
hive.openshift.io/ssh-known-host: "KNOWN HOST STRING HERE"
spec:
baseDomain: e2e.bos.redhat.com
clusterName: kni5
controlPlaneConfig:
servingCertificates: {}
images:
installerImagePullPolicy: Always
installed: false
platform:
bareMetal: {}
provisioning:
releaseImage: quay.io/openshift-release-dev/ocp-release-nightly@sha256:73007e97e00d526426c116b1a35d8103bd00a887656d065b7979ab2e8de44b17
installConfigSecretRef:
name: kni5-install-config
sshPrivateKeySecretRef:
name: kni5-ssh-private-key
manifestsConfigMapRef:
name: kni5-install-manifests
pullSecretRef:
name: kni5-pull-secret
status:
clusterVersionStatus:
availableUpdates: null
desired:
force: false
image: ""
version: ""
observedGeneration: 0
versionHash: ""
- apiVersion: hive.openshift.io/v1
kind: MachinePool
metadata:
creationTimestamp: null
name: kni5-worker
spec:
clusterDeploymentRef:
name: kni5
name: worker
platform:
bareMetal: {}
replicas: 0
status: {}
kind: List
metadata: {}
Now that we have our three configuration files: clusterdeploy.yaml, installer-manifests.yaml and install-config.yaml, we can begin to deploy OpensShift 4.3 on baremetal via Hive.
First we need to create a new project on the Minishift cluster:
$ oc new-project ocpbaredeploy
Now using project "ocpbaredeploy" on server "https://192.168.42.173:8443".
You can add applications to this project with the 'new-app' command. For example, try:
oc new-app centos/ruby-25-centos7~https://github.com/sclorg/ruby-ex.git
to build a new example application in Ruby.
Next we will configure a secret, in my case called kni5-install-config, from the install-config.yaml file:
$ oc create secret generic kni5-install-config --from-file=install-config.yaml secret/kni5-install-config created
In addition to the secret we need to also create the installer-manifests from file:
$ oc create -f installer-manifests.yaml configmap/kni5-install-manifests created
And finally we can create the cluster deployment itself which will actually launch the OCP 4.3 baremetal deployment:
$ oc create -f clusterdeploy.yaml secret/kni5-pull-secret created secret/kni5-ssh-private-key created clusterdeployment.hive.openshift.io/kni5 created
If we do a oc get pods we can see the status of our provision container:
$ oc get pods NAME READY STATUS RESTARTS AGE kni5-0-6d6wc-provision-rnxqf 0/3 ContainerCreating 0 1s kni5-imageset-7vh7c 0/2 Completed 0 30s
We can follow along with the deployment by watching the logs of the provision container with the following oc logs -f command:
$ oc logs -f kni5-0-6d6wc-provision-rnxqf -c hive time="2020-01-28T16:42:10Z" level=debug msg="checking for SSH private key" installID=gm4l6fj7 time="2020-01-28T16:42:10Z" level=debug msg="no SSH_AUTH_SOCK defined. starting ssh-agent" installID=gm4l6fj7 Identity added: /tmp/ssh-privatekey (ansible-generated on rhel8-ocp-auto) time="2020-01-28T16:42:10Z" level=info msg="waiting for files to be available: [/output/openshift-install /output/oc]" installID=gm4l6fj7 time="2020-01-28T16:42:10Z" level=info msg="found file" installID=gm4l6fj7 path=/output/openshift-install time="2020-01-28T16:42:10Z" level=info msg="found file" installID=gm4l6fj7 path=/output/oc time="2020-01-28T16:42:10Z" level=info msg="all files found, ready to proceed" installID=gm4l6fj7 time="2020-01-28T16:42:10Z" level=info msg="copying install-config.yaml" installID=gm4l6fj7 time="2020-01-28T16:42:10Z" level=info msg="copied /installconfig/install-config.yaml to /output/install-config.yaml" installID=gm4l6fj7 time="2020-01-28T16:42:10Z" level=info msg="Adding user ID to passwd file: 1000170000" installID=gm4l6fj7 time="2020-01-28T16:42:10Z" level=info msg="Wrote passwd line: default:x:1000170000:0:default user:/home/hive:/sbin/nologin\n" installID=gm4l6fj7 time="2020-01-28T16:42:10Z" level=info msg="Wrote known host to /home/hive/.ssh/known_hosts: rhel8-ocp-auto.schmaustech.com,192.168.0.246 ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAvnpsB2rUlrxgNQGdqSB10bjPybWFdSX7jqZta5FxOZEYPEVwC5QD/FlSYEjM4GUteNRI1vcEFstBkQs0T4WJU=" installID=gm4l6fj7 time="2020-01-28T16:42:10Z" level=info msg="cleaning up from past install attempts" installID=gm4l6fj7 time="2020-01-28T16:42:10Z" level=debug msg="object does not exist" installID=gm4l6fj7 object=ocpbaredeploy/kni5-0-6d6wc-admin-kubeconfig time="2020-01-28T16:42:10Z" level=debug msg="object does not exist" installID=gm4l6fj7 object=ocpbaredeploy/kni5-0-6d6wc-admin-password time="2020-01-28T16:42:10Z" level=warning msg="skipping cleanup as no infra ID set" installID=gm4l6fj7 time="2020-01-28T16:42:10Z" level=info msg="generating assets" installID=gm4l6fj7 time="2020-01-28T16:42:10Z" level=info msg="running openshift-install create manifests" installID=gm4l6fj7 time="2020-01-28T16:42:10Z" level=info msg="running openshift-install binary" args="[create manifests]" installID=gm4l6fj7 time="2020-01-28T16:42:10Z" level=info msg="waiting for files to be available: [/output/.openshift_install.log]" installID=gm4l6fj7 time="2020-01-28T16:42:10Z" level=info msg="found file" installID=gm4l6fj7 path=/output/.openshift_install.log time="2020-01-28T16:42:10Z" level=info msg="all files found, ready to proceed" installID=gm4l6fj7 time="2020-01-28T16:42:10Z" level=debug msg="OpenShift Installer v4.3.0" time="2020-01-28T16:42:10Z" level=debug msg="Built from commit e39341c5c2b9e6fb2c7ddfa2d3220419aa0e2072" time="2020-01-28T16:42:10Z" level=debug msg="Fetching Master Machines..." time="2020-01-28T16:42:10Z" level=debug msg="Loading Master Machines..." time="2020-01-28T16:42:10Z" level=debug msg=" Loading Cluster ID..." time="2020-01-28T16:42:10Z" level=debug msg=" Loading Install Config..." time="2020-01-28T16:42:10Z" level=debug msg=" Loading SSH Key..." time="2020-01-28T16:42:10Z" level=debug msg=" Loading Base Domain..." time="2020-01-28T16:42:10Z" level=debug msg=" Loading Platform..." time="2020-01-28T16:42:10Z" level=debug msg=" Loading Cluster Name..." time="2020-01-28T16:42:10Z" level=debug msg=" Loading Base Domain..." time="2020-01-28T16:42:10Z" level=debug msg=" Loading Pull Secret..." time="2020-01-28T16:42:10Z" level=debug msg=" Loading Platform..." time="2020-01-28T16:42:10Z" level=debug msg=" Using Install Config loaded from target directory" time="2020-01-28T16:42:10Z" level=debug msg=" Loading Platform Credentials Check..." time="2020-01-28T16:42:10Z" level=debug msg=" Loading Install Config..." time="2020-01-28T16:42:10Z" level=debug msg=" Loading Install Config..." time="2020-01-28T16:42:10Z" level=debug msg=" Loading Image..." time="2020-01-28T16:42:10Z" level=debug msg=" Loading Install Config..." time="2020-01-28T16:42:10Z" level=debug msg=" Loading Master Ignition Config..." time="2020-01-28T16:42:10Z" level=debug msg=" Loading Install Config..." time="2020-01-28T16:42:10Z" level=debug msg=" Loading Root CA..." time="2020-01-28T16:42:10Z" level=debug msg=" Fetching Cluster ID..." time="2020-01-28T16:42:10Z" level=debug msg=" Fetching Install Config..." time="2020-01-28T16:42:10Z" level=debug msg=" Reusing previously-fetched Install Config" time="2020-01-28T16:42:10Z" level=debug msg=" Generating Cluster ID..." time="2020-01-28T16:42:10Z" level=debug msg=" Fetching Platform Credentials Check..." time="2020-01-28T16:42:10Z" level=debug msg=" Fetching Install Config..." time="2020-01-28T16:42:10Z" level=debug msg=" Reusing previously-fetched Install Config" time="2020-01-28T16:42:10Z" level=debug msg=" Generating Platform Credentials Check..." time="2020-01-28T16:42:10Z" level=debug msg=" Fetching Install Config..." time="2020-01-28T16:42:10Z" level=debug msg=" Reusing previously-fetched Install Config" time="2020-01-28T16:42:10Z" level=debug msg=" Fetching Image..." time="2020-01-28T16:42:10Z" level=debug msg=" Fetching Install Config..." time="2020-01-28T16:42:10Z" level=debug msg=" Reusing previously-fetched Install Config" time="2020-01-28T16:42:10Z" level=debug msg=" Generating Image..." time="2020-01-28T16:42:10Z" level=debug msg=" Fetching Master Ignition Config..." time="2020-01-28T16:42:10Z" level=debug msg=" Fetching Install Config..." time="2020-01-28T16:42:10Z" level=debug msg=" Reusing previously-fetched Install Config" time="2020-01-28T16:42:10Z" level=debug msg=" Fetching Root CA..." time="2020-01-28T16:42:10Z" level=debug msg=" Generating Root CA..." time="2020-01-28T16:42:10Z" level=debug msg=" Generating Master Ignition Config..." time="2020-01-28T16:42:10Z" level=debug msg="Generating Master Machines..." time="2020-01-28T16:42:10Z" level=info msg="Consuming Install Config from target directory"
Once the install completes we should see this at the end of the logging:
time="2020-01-28T17:42:52Z" level=debug msg="Still waiting for the cluster to initialize: Working towards 4.3.0-0.nightly-2019-12-06-094536: 100% complete, waiting on authentication" time="2020-01-28T17:42:52Z" level=debug msg="Still waiting for the cluster to initialize: Working towards 4.3.0-0.nightly-2019-12-06-094536: 100% complete, waiting on authentication" time="2020-01-28T17:42:52Z" level=debug msg="Still waiting for the cluster to initialize: Working towards 4.3.0-0.nightly-2019-12-06-094536: 100% complete, waiting on authentication" time="2020-01-28T17:44:50Z" level=debug msg="Cluster is initialized" time="2020-01-28T17:44:50Z" level=debug msg="Cluster is initialized" time="2020-01-28T17:44:50Z" level=info msg="Waiting up to 10m0s for the openshift-console route to be created..." time="2020-01-28T17:44:50Z" level=debug msg="Cluster is initialized" time="2020-01-28T17:44:50Z" level=info msg="Waiting up to 10m0s for the openshift-console route to be created..." time="2020-01-28T17:44:50Z" level=info msg="Waiting up to 10m0s for the openshift-console route to be created..." time="2020-01-28T17:44:50Z" level=debug msg="Route found in openshift-console namespace: console" time="2020-01-28T17:44:50Z" level=debug msg="Route found in openshift-console namespace: downloads" time="2020-01-28T17:44:50Z" level=debug msg="OpenShift console route is created" time="2020-01-28T17:44:50Z" level=debug msg="Route found in openshift-console namespace: console" time="2020-01-28T17:44:50Z" level=debug msg="Route found in openshift-console namespace: console" time="2020-01-28T17:44:50Z" level=debug msg="Route found in openshift-console namespace: downloads" time="2020-01-28T17:44:50Z" level=debug msg="OpenShift console route is created" time="2020-01-28T17:44:50Z" level=debug msg="Route found in openshift-console namespace: downloads" time="2020-01-28T17:44:50Z" level=debug msg="OpenShift console route is created" time="2020-01-28T17:44:50Z" level=info msg="Install complete!" time="2020-01-28T17:44:50Z" level=info msg="To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/output/auth/kubeconfig'" time="2020-01-28T17:44:50Z" level=info msg="Access the OpenShift web-console here: https://console-openshift-console.apps.kni5.schmaustech.com" REDACTED LINE OF OUTPUT time="2020-01-28T17:44:50Z" level=info msg="Install complete!" time="2020-01-28T17:44:50Z" level=info msg="To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/output/auth/kubeconfig'" time="2020-01-28T17:44:50Z" level=info msg="Access the OpenShift web-console here: https://console-openshift-console.apps.kni5.schmaustech.com" REDACTED LINE OF OUTPUT time="2020-01-28T17:44:50Z" level=info msg="Install complete!" time="2020-01-28T17:44:50Z" level=info msg="To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/output/auth/kubeconfig'" time="2020-01-28T17:44:50Z" level=info msg="Access the OpenShift web-console here: https://console-openshift-console.apps.kni5.schmaustech.com" REDACTED LINE OF OUTPUT time="2020-01-28T17:44:51Z" level=info msg="command completed successfully" installID=gm4l6fj7 time="2020-01-28T17:44:51Z" level=info msg="saving installer output" installID=gm4l6fj7 time="2020-01-28T17:44:51Z" level=debug msg="installer console log: level=info msg=\"Obtaining RHCOS image file from 'https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.3/43.81.201911221453.0/x86_64/rhcos-43.81.201911221453.0-qemu.x86_64.qcow2.gz?sha256=073fb598762e56c7852e743f4e2f1187f7783f9bc741aa993e4cc36a70325ab2'\"\nlevel=info msg=\"Consuming Worker Ignition Config from target directory\"\nlevel=info msg=\"Consuming Bootstrap Ignition Config from target directory\"\nlevel=info msg=\"Consuming Master Ignition Config from target directory\"\nlevel=info msg=\"Creating infrastructure resources...\"\nlevel=info msg=\"Waiting up to 30m0s for the Kubernetes API at https://api.kni5.schmaustech.com:6443...\"\nlevel=info msg=\"API v1.16.2 up\"\nlevel=info msg=\"Waiting up to 30m0s for bootstrapping to complete...\"\nlevel=info msg=\"Destroying the bootstrap resources...\"\nlevel=info msg=\"Waiting up to 30m0s for the cluster at https://api.kni5.schmaustech.com:6443 to initialize...\"\nlevel=info msg=\"Waiting up to 10m0s for the openshift-console route to be created...\"\nlevel=info msg=\"Install complete!\"\nlevel=info msg=\"To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/output/auth/kubeconfig'\"\nlevel=info msg=\"Access the OpenShift web-console here: https://console-openshift-console.apps.kni5.schmaustech.com\"\nREDACTED LINE OF OUTPUT\n" installID=gm4l6fj7 time="2020-01-28T17:44:51Z" level=info msg="install completed successfully" installID=gm4l6fj7
To validate we have a fully operational cluster we will need to obtain the kubeconfig. This is stored in the secrets of hive. Below we will grep for all secrets related to the cluster kni5:
$ oc get secrets|grep kni5 kni5-0-6d6wc-admin-kubeconfig Opaque 2 1h kni5-0-6d6wc-admin-password Opaque 2 1h kni5-install-config Opaque 1 1h kni5-merged-pull-secret kubernetes.io/dockerconfigjson 1 1h kni5-pull-secret kubernetes.io/dockerconfigjson 1 1h kni5-ssh-private-key Opaque 1 1h
We are specifically interested in the secret with kubeconfig in the naming convention and we can use the command below to extract out the kubeconfig and place it into a file:
oc get secret kni5-0-6d6wc-admin-kubeconfig -o jsonpath='{ .data.kubeconfig }' | base64 --decode > kni5.kubeconfig
After extraction we can export the kni5 kubeconfig to the KUBECONFIG variable:
export KUBECONFIG=./kni5.kubeconfig
And finally we can do an oc get nodes to see the nodes:
$ oc get nodes NAME STATUS ROLES AGE VERSION master-0.kni5.schmaustech.com Ready master,worker 74m v1.16.2 master-1.kni5.schmaustech.com Ready master,worker 74m v1.16.2 master-2.kni5.schmaustech.com Ready master,worker 74m v1.16.2
And a get all pods to see the running containers:
$ oc get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE openshift-apiserver-operator openshift-apiserver-operator-6467df49bb-k9scl 1/1 Running 2 75m openshift-apiserver apiserver-d8rbk 1/1 Running 0 67m openshift-apiserver apiserver-ngg7c 1/1 Running 0 68m openshift-apiserver apiserver-sqv5l 1/1 Running 0 68m openshift-authentication-operator authentication-operator-c4788d544-r2cns 1/1 Running 0 66m openshift-authentication oauth-openshift-664ff64575-n7hbf 1/1 Running 0 63m openshift-authentication oauth-openshift-664ff64575-qp9r9 1/1 Running 0 63m openshift-cloud-credential-operator cloud-credential-operator-cfccb976c-5h9hb 1/1 Running 0 75m openshift-cluster-machine-approver machine-approver-58bf678f78-tf75d 2/2 Running 0 75m openshift-cluster-node-tuning-operator cluster-node-tuning-operator-84dcb88fdb-rv6w6 1/1 Running 0 66m openshift-cluster-node-tuning-operator tuned-6pqvv 1/1 Running 0 66m openshift-cluster-node-tuning-operator tuned-gcpsl 1/1 Running 0 66m openshift-cluster-node-tuning-operator tuned-hlf5j 1/1 Running 0 66m openshift-cluster-samples-operator cluster-samples-operator-6b477c8f46-pl7x4 2/2 Running 0 62m openshift-cluster-storage-operator cluster-storage-operator-85cd5b4567-zqbdz 1/1 Running 0 64m openshift-cluster-version cluster-version-operator-6747896cf7-68mz6 1/1 Running 0 75m openshift-console-operator console-operator-7f6b8d755d-b4hk4 1/1 Running 0 64m openshift-console console-869c7d68fb-jbz7j 1/1 Running 0 60m openshift-console console-869c7d68fb-jfd8p 1/1 Running 0 63m openshift-console downloads-79cb8f7c7f-49xt4 1/1 Running 0 64m openshift-console downloads-79cb8f7c7f-kcfnh 1/1 Running 0 64m openshift-controller-manager-operator openshift-controller-manager-operator-6bd6b5c9b7-4djr8 1/1 Running 2 75m openshift-controller-manager controller-manager-hq5pw 1/1 Running 0 66m openshift-controller-manager controller-manager-vjdlc 1/1 Running 0 66m openshift-controller-manager controller-manager-wb8m2 1/1 Running 0 68m openshift-dns-operator dns-operator-768898c96f-cpb4m 2/2 Running 0 75m openshift-dns dns-default-8rs4t 2/2 Running 0 71m openshift-dns dns-default-bt8jr 2/2 Running 0 71m openshift-dns dns-default-t2bx7 2/2 Running 0 71m openshift-etcd etcd-member-master-0.kni5.schmaustech.com 2/2 Running 0 74m openshift-etcd etcd-member-master-1.kni5.schmaustech.com 2/2 Running 0 75m openshift-etcd etcd-member-master-2.kni5.schmaustech.com 2/2 Running 0 75m openshift-image-registry cluster-image-registry-operator-69cfd6b89-ldv6d 2/2 Running 0 64m openshift-ingress-operator ingress-operator-846944b6cd-442mh 2/2 Running 0 64m openshift-ingress router-default-668c8bc58c-5p65n 1/1 Running 0 63m openshift-ingress router-default-668c8bc58c-7lbl7 1/1 Running 0 62m openshift-insights insights-operator-5f8867cf5b-2b4c6 1/1 Running 3 75m openshift-kni-infra coredns-master-0.kni5.schmaustech.com 1/1 Running 0 74m openshift-kni-infra coredns-master-1.kni5.schmaustech.com 1/1 Running 0 75m openshift-kni-infra coredns-master-2.kni5.schmaustech.com 1/1 Running 0 74m openshift-kni-infra haproxy-master-0.kni5.schmaustech.com 2/2 Running 0 74m openshift-kni-infra haproxy-master-1.kni5.schmaustech.com 2/2 Running 0 75m openshift-kni-infra haproxy-master-2.kni5.schmaustech.com 2/2 Running 0 75m openshift-kni-infra keepalived-master-0.kni5.schmaustech.com 2/2 Running 0 74m openshift-kni-infra keepalived-master-1.kni5.schmaustech.com 2/2 Running 0 75m openshift-kni-infra keepalived-master-2.kni5.schmaustech.com 2/2 Running 0 73m openshift-kni-infra mdns-publisher-master-0.kni5.schmaustech.com 1/1 Running 0 74m openshift-kni-infra mdns-publisher-master-1.kni5.schmaustech.com 1/1 Running 0 75m openshift-kni-infra mdns-publisher-master-2.kni5.schmaustech.com 1/1 Running 0 74m openshift-kube-apiserver-operator kube-apiserver-operator-5866656944-6j7jd 1/1 Running 2 75m openshift-kube-apiserver installer-2-master-0.kni5.schmaustech.com 0/1 Completed 0 68m openshift-kube-apiserver installer-2-master-1.kni5.schmaustech.com 0/1 Completed 0 64m openshift-kube-apiserver installer-2-master-2.kni5.schmaustech.com 0/1 Completed 0 69m openshift-kube-apiserver installer-3-master-1.kni5.schmaustech.com 0/1 Completed 0 64m openshift-kube-apiserver installer-5-master-0.kni5.schmaustech.com 0/1 Completed 0 57m openshift-kube-apiserver installer-5-master-1.kni5.schmaustech.com 0/1 Completed 0 62m openshift-kube-apiserver installer-5-master-2.kni5.schmaustech.com 0/1 Completed 0 60m openshift-kube-apiserver kube-apiserver-master-0.kni5.schmaustech.com 3/3 Running 0 57m openshift-kube-apiserver kube-apiserver-master-1.kni5.schmaustech.com 3/3 Running 0 62m openshift-kube-apiserver kube-apiserver-master-2.kni5.schmaustech.com 3/3 Running 0 60m openshift-kube-apiserver revision-pruner-2-master-0.kni5.schmaustech.com 0/1 Completed 0 64m openshift-kube-apiserver revision-pruner-2-master-1.kni5.schmaustech.com 0/1 Completed 0 64m openshift-kube-apiserver revision-pruner-2-master-2.kni5.schmaustech.com 0/1 Completed 0 64m openshift-kube-apiserver revision-pruner-3-master-1.kni5.schmaustech.com 0/1 Completed 0 62m openshift-kube-apiserver revision-pruner-5-master-0.kni5.schmaustech.com 0/1 Completed 0 56m openshift-kube-apiserver revision-pruner-5-master-1.kni5.schmaustech.com 0/1 Completed 0 60m openshift-kube-apiserver revision-pruner-5-master-2.kni5.schmaustech.com 0/1 Completed 0 58m openshift-kube-controller-manager-operator kube-controller-manager-operator-7967d57b6c-c2wlk 1/1 Running 2 75m openshift-kube-controller-manager installer-4-master-2.kni5.schmaustech.com 0/1 Completed 0 69m openshift-kube-controller-manager installer-5-master-0.kni5.schmaustech.com 0/1 Completed 0 68m openshift-kube-controller-manager installer-5-master-1.kni5.schmaustech.com 0/1 Completed 0 67m openshift-kube-controller-manager installer-5-master-2.kni5.schmaustech.com 0/1 Completed 0 65m openshift-kube-controller-manager installer-6-master-0.kni5.schmaustech.com 0/1 Completed 0 61m openshift-kube-controller-manager installer-6-master-1.kni5.schmaustech.com 0/1 Completed 0 60m openshift-kube-controller-manager installer-6-master-2.kni5.schmaustech.com 0/1 OOMKilled 0 63m openshift-kube-controller-manager installer-7-master-0.kni5.schmaustech.com 0/1 Completed 0 57m openshift-kube-controller-manager installer-7-master-1.kni5.schmaustech.com 0/1 Completed 0 59m openshift-kube-controller-manager installer-7-master-2.kni5.schmaustech.com 0/1 Completed 0 59m openshift-kube-controller-manager kube-controller-manager-master-0.kni5.schmaustech.com 3/3 Running 0 57m openshift-kube-controller-manager kube-controller-manager-master-1.kni5.schmaustech.com 3/3 Running 0 59m openshift-kube-controller-manager kube-controller-manager-master-2.kni5.schmaustech.com 3/3 Running 0 58m openshift-kube-controller-manager revision-pruner-4-master-2.kni5.schmaustech.com 0/1 Completed 0 68m openshift-kube-controller-manager revision-pruner-5-master-0.kni5.schmaustech.com 0/1 Completed 0 68m openshift-kube-controller-manager revision-pruner-5-master-1.kni5.schmaustech.com 0/1 Completed 0 65m openshift-kube-controller-manager revision-pruner-5-master-2.kni5.schmaustech.com 0/1 Completed 0 64m openshift-kube-controller-manager revision-pruner-6-master-0.kni5.schmaustech.com 0/1 Completed 0 60m openshift-kube-controller-manager revision-pruner-6-master-1.kni5.schmaustech.com 0/1 Completed 0 59m openshift-kube-controller-manager revision-pruner-6-master-2.kni5.schmaustech.com 0/1 Completed 0 61m openshift-kube-controller-manager revision-pruner-7-master-0.kni5.schmaustech.com 0/1 Completed 0 56m openshift-kube-controller-manager revision-pruner-7-master-1.kni5.schmaustech.com 0/1 Completed 0 59m openshift-kube-controller-manager revision-pruner-7-master-2.kni5.schmaustech.com 0/1 Completed 0 58m openshift-kube-scheduler-operator openshift-kube-scheduler-operator-6bbb9c89c4-8gfql 1/1 Running 2 75m openshift-kube-scheduler installer-2-master-1.kni5.schmaustech.com 0/1 Completed 0 71m openshift-kube-scheduler installer-3-master-2.kni5.schmaustech.com 0/1 Completed 0 71m openshift-kube-scheduler installer-4-master-0.kni5.schmaustech.com 0/1 Completed 0 70m openshift-kube-scheduler installer-4-master-1.kni5.schmaustech.com 0/1 Completed 0 69m openshift-kube-scheduler installer-5-master-0.kni5.schmaustech.com 0/1 Completed 0 65m openshift-kube-scheduler installer-5-master-1.kni5.schmaustech.com 0/1 Completed 0 68m openshift-kube-scheduler installer-5-master-2.kni5.schmaustech.com 0/1 Completed 0 67m openshift-kube-scheduler installer-6-master-0.kni5.schmaustech.com 0/1 Completed 0 60m openshift-kube-scheduler installer-6-master-1.kni5.schmaustech.com 0/1 Completed 0 63m openshift-kube-scheduler installer-6-master-2.kni5.schmaustech.com 0/1 Completed 0 61m openshift-kube-scheduler openshift-kube-scheduler-master-0.kni5.schmaustech.com 1/1 Running 0 60m openshift-kube-scheduler openshift-kube-scheduler-master-1.kni5.schmaustech.com 1/1 Running 0 63m openshift-kube-scheduler openshift-kube-scheduler-master-2.kni5.schmaustech.com 1/1 Running 1 61m openshift-kube-scheduler revision-pruner-2-master-1.kni5.schmaustech.com 0/1 Completed 0 71m openshift-kube-scheduler revision-pruner-3-master-2.kni5.schmaustech.com 0/1 Completed 0 70m openshift-kube-scheduler revision-pruner-4-master-0.kni5.schmaustech.com 0/1 Completed 0 69m openshift-kube-scheduler revision-pruner-4-master-1.kni5.schmaustech.com 0/1 Completed 0 69m openshift-kube-scheduler revision-pruner-5-master-0.kni5.schmaustech.com 0/1 Completed 0 64m openshift-kube-scheduler revision-pruner-5-master-1.kni5.schmaustech.com 0/1 Completed 0 67m openshift-kube-scheduler revision-pruner-5-master-2.kni5.schmaustech.com 0/1 Completed 0 65m openshift-kube-scheduler revision-pruner-6-master-0.kni5.schmaustech.com 0/1 Completed 0 59m openshift-kube-scheduler revision-pruner-6-master-1.kni5.schmaustech.com 0/1 Completed 0 61m openshift-kube-scheduler revision-pruner-6-master-2.kni5.schmaustech.com 0/1 Completed 0 60m openshift-machine-api cluster-autoscaler-operator-5967b8b469-r8r7r 2/2 Running 0 62m openshift-machine-api machine-api-controllers-59876d48b9-5pc8n 4/4 Running 0 69m openshift-machine-api machine-api-operator-7dd8fbc5c-bd4h6 2/2 Running 0 74m openshift-machine-api metal3-6d87476688-x5chm 8/8 Running 0 69m openshift-machine-config-operator etcd-quorum-guard-d6ff6bd8f-247hv 1/1 Running 0 70m openshift-machine-config-operator etcd-quorum-guard-d6ff6bd8f-27vd6 1/1 Running 0 70m openshift-machine-config-operator etcd-quorum-guard-d6ff6bd8f-d6jql 1/1 Running 0 70m openshift-machine-config-operator machine-config-controller-98b8ff5d-m7764 1/1 Running 1 71m openshift-machine-config-operator machine-config-daemon-lkw5r 2/2 Running 0 72m openshift-machine-config-operator machine-config-daemon-nlbbn 2/2 Running 0 72m openshift-machine-config-operator machine-config-daemon-sxnpb 2/2 Running 0 72m openshift-machine-config-operator machine-config-operator-5c4c599bc7-fmgn2 1/1 Running 0 75m openshift-machine-config-operator machine-config-server-bx9q2 1/1 Running 0 70m openshift-machine-config-operator machine-config-server-pfhgn 1/1 Running 0 70m openshift-machine-config-operator machine-config-server-vztjh 1/1 Running 0 70m openshift-marketplace certified-operators-699d47c664-sp9z9 1/1 Running 0 63m openshift-marketplace community-operators-66d78d8cd-hs2td 1/1 Running 0 63m openshift-marketplace marketplace-operator-6487d7d4c7-lwzvd 1/1 Running 0 64m openshift-marketplace redhat-operators-6b6f87448f-dwbbr 1/1 Running 0 63m openshift-monitoring alertmanager-main-0 3/3 Running 0 61m openshift-monitoring alertmanager-main-1 3/3 Running 0 61m openshift-monitoring alertmanager-main-2 3/3 Running 0 61m openshift-monitoring cluster-monitoring-operator-6bd6f6485b-qgtwx 1/1 Running 0 64m openshift-monitoring grafana-7d5458c694-k2q4j 2/2 Running 0 61m openshift-monitoring kube-state-metrics-544fbcbfbb-nk2v7 3/3 Running 0 63m openshift-monitoring node-exporter-9qbz9 2/2 Running 0 63m openshift-monitoring node-exporter-gzd5b 2/2 Running 0 63m openshift-monitoring node-exporter-s8lm6 2/2 Running 0 63m openshift-monitoring openshift-state-metrics-cfbc56bbc-9zprs 3/3 Running 0 63m openshift-monitoring prometheus-adapter-5686f9c8bb-8tv9l 1/1 Running 0 61m openshift-monitoring prometheus-adapter-5686f9c8bb-d54hf 1/1 Running 0 61m openshift-monitoring prometheus-k8s-0 7/7 Running 1 58m openshift-monitoring prometheus-k8s-1 7/7 Running 1 59m openshift-monitoring prometheus-operator-d7c899b8c-qq7qw 1/1 Running 0 61m openshift-monitoring telemeter-client-778b4db874-d9c6t 3/3 Running 0 61m openshift-monitoring thanos-querier-5984bb45dc-6g2xs 4/4 Running 0 61m openshift-monitoring thanos-querier-5984bb45dc-vlttb 4/4 Running 0 61m openshift-multus multus-276kq 1/1 Running 0 74m openshift-multus multus-admission-controller-5k87f 1/1 Running 1 72m openshift-multus multus-admission-controller-m7tg8 1/1 Running 0 72m openshift-multus multus-admission-controller-mxj4j 1/1 Running 1 72m openshift-multus multus-qnfm2 1/1 Running 0 74m openshift-multus multus-xw572 1/1 Running 0 74m openshift-network-operator network-operator-69d75fd9b6-msztp 1/1 Running 0 75m openshift-operator-lifecycle-manager catalog-operator-767bc6d8c5-nlbt4 1/1 Running 0 75m openshift-operator-lifecycle-manager olm-operator-85b66cf96d-r4tqw 1/1 Running 0 75m openshift-operator-lifecycle-manager packageserver-69dcc957f8-p5rp6 1/1 Running 0 70m openshift-operator-lifecycle-manager packageserver-69dcc957f8-p9vhq 1/1 Running 1 70m openshift-sdn ovs-59mq4 1/1 Running 0 74m openshift-sdn ovs-k8gs8 1/1 Running 0 74m openshift-sdn ovs-tbl69 1/1 Running 0 74m openshift-sdn sdn-4s4pv 1/1 Running 0 74m openshift-sdn sdn-controller-l4g5k 1/1 Running 0 74m openshift-sdn sdn-controller-lkjxc 1/1 Running 0 74m openshift-sdn sdn-controller-skrkp 1/1 Running 0 74m openshift-sdn sdn-f4j6f 1/1 Running 0 74m openshift-sdn sdn-wpp5n 1/1 Running 0 74m openshift-service-ca-operator service-ca-operator-7556fc7b7-9hq5t 1/1 Running 1 75m openshift-service-ca apiservice-cabundle-injector-c865fd686-pp66j 1/1 Running 1 71m openshift-service-ca configmap-cabundle-injector-796f8b9994-ntnwp 1/1 Running 1 71m openshift-service-ca service-serving-cert-signer-5f65674db4-x7ltk 1/1 Running 1 71m openshift-service-catalog-apiserver-operator openshift-service-catalog-apiserver-operator-84689c4d85-45kcd 1/1 Running 0 66m openshift-service-catalog-controller-manager-operator openshift-service-catalog-controller-manager-operator-67fd4fkzk 1/1 Running 0 66m
If everything worked out you should now have a OpenShift 4.3 Baremetal deployment that was deployed via Hive. However I should note that Hive is rapidly evolving the method used here could be changing in the future.
Sunday, November 17, 2019
Deploying OpenShift IPI Baremetal Disconnected
The following blog is a write up of the steps I used to generate a OpenShift IPI disconnected baremetal install. In this configuration we first mirror down the images to a local repository. Then deploy the Openshift cluster onto virtual machines that are mimicking a baremetal environment with vBMC being used as the IPMI interface into the virtual machines. Please note that while this was demonstrated in a virtual setting these steps should also work with physical hardware.
Lab Setup:
Physical node specification:
Processors: i7 - 8 vcpus Memory: 32gb Disk: 512gb M2 SSD Single NIC with 2 vlans (external and provisioning) tagged in Nested virtualization should be enabled
Virtual machine Node specification:
Processor: Passthrough from physical node - 4 vcpus Memory: 16gb Disk: 60gb raw image Two NICs ens3 (provisioning) ens4 (external) - no tags - dhcp interfaces IPMI: Provided via centralized Virtual BMC controller
Lab Diagram:
Predefined DNS records:
*.apps.kni5 IN A 192.168.0.197 ns1.kni5 IN A 192.168.0.198 api.kni5 IN A 192.168.0.199 master-0.kni5 IN A 192.168.0.200 master-1.kni5 IN A 192.168.0.201 master-2.kni5 IN A 192.168.0.202
Cluster and user definitions:
Preparing Provisioning Node:
The first step in preparing the provisioning node is to install, kickstart or image the provisioning node with RHEL8 and register the host. I leverage an ISO image local on NUC-1 with an embedded kickstart file that way I can rebuild my provisioning node on the fly and start off with a fresh install.
Once the provisioning node is installed make sure the following packages are installed:
Next lets make sure the username used for installation has passwordless sudo access. This is more about convenience give a lot of the commands require root and or sudo access:
Unfortunately at this time selinux does need to be set to permissive so we will do that now:
For the bootstrap node to boot on this virtual machine we need to ensure a default storage pool exists. On a RHEL8 installation this pool does not seem to exist out of the box so lets create it:
The Openshift installer expects there to be a baremetal and provisioning interface on the provisioning node so lets configure them with the following:
At this time we should also pull the latest oc binary and openshift-baremetal-install binary. The oc command will be used to mirror the repository and also validate the cluster is up once installation is complete:
Create Initial Install-Config.yaml and Local Image Repository:
Now that we have prepared the provisioning host, we need to first create our initial install-config.yaml file. The file should look similar to the sample below but adjusted for your environment (Note: RELEASEVERSION should be typed as in example as we will change that in later step):
Lets also create a pull-secret.json file that has the OpenShift pull-secret credentials so we can access the Openshift repository (Note: PULL-SECRET-JSON needs to be the actual pull-secret one gets from OpenShift):
Now that we have the initial install-config.yaml that we will use to deploy the OpenShift cluster lets change gears and configure the local image repository on the provisioning node. The steps below outline what is needed to configure the repository:
Test that the repository is working with the following test:
Update Install-Config.yaml Pull Secret & Certificate:
Now we need to update the pull-secret in our install-config.yaml file to reference the local repository credentials we used above. To do this we first need to capture the base64 output for the user/password we configured for the local mirror which was dummy and dummy:
With the output above place it into a string like the sample below making sure to update the repository hostname as well to match the environment. Save the contents to a file called local_pull_secret:
Now lets inject that local_pull_secret into our install-config.yaml file:
Since we also created a cert above and our install-config.yaml file will need to connect to the local repository lets go ahead and add that cert to the install-config.yaml:
Once the install-config.yaml file is updated with both the local repository pull-secret and domain cert our install-config.yaml will look something like the following:
Mirror Images:
Now that we have configured all the services and configuration files we need for our OpenShift IPI deployment we can now begin the process of mirroring the images to the local respository:
Adjust ImageContentSources in Install-Config.yaml:
Recall we original defined the install-config.yaml with a line that had RELEASEVERSION in all caps. Since we declared the OCP_RELEASE variable in the previous steps, we are now ready to update the install-config.yaml and replace RELEASEVERSION with the proper version defined for our local repository. The following steps initiate that change:
Deploying the OpenShift Cluster:
Finally after all of the steps above, we can begin the actual deployment using the commands below:
If all the steps were followed, the cluster should successfully deploy with the exception of Metal3 container as there is an issue with this that needs to be addressed.
We can check if the cluster nodes are online with the following:
We can also view all the pods to validate things are online as well:
Cluster Name: kni5 Domain Name: schmaustech.com Username for installation: bschmaus
Preparing Provisioning Node:
The first step in preparing the provisioning node is to install, kickstart or image the provisioning node with RHEL8 and register the host. I leverage an ISO image local on NUC-1 with an embedded kickstart file that way I can rebuild my provisioning node on the fly and start off with a fresh install.
Once the provisioning node is installed make sure the following packages are installed:
kexec-tools @development (package group) git usbredir golang libXv virt-install libvirt libvirt-devel libselinux-utils qemu-kvm mkisofs
Next lets make sure the username used for installation has passwordless sudo access. This is more about convenience give a lot of the commands require root and or sudo access:
# cat << EOF > /etc/sudoers.d/openshift Defaults:bschmaus !requiretty bschmaus ALL = (root) NOPASSWD:ALL EOF # chmod 600 /etc/sudoers.d/openshift
Unfortunately at this time selinux does need to be set to permissive so we will do that now:
# sudo setenforce permissive # sudo sed -i "s/=enforcing/=permissive/g" /etc/selinux/config
For the bootstrap node to boot on this virtual machine we need to ensure a default storage pool exists. On a RHEL8 installation this pool does not seem to exist out of the box so lets create it:
# sudo virsh pool-define-as --name default --type dir --target /var/lib/libvirt/images # sudo virsh pool-start default # sudo virsh pool-autostart default # sudo usermod --append --groups libvirt bschmaus
The Openshift installer expects there to be a baremetal and provisioning interface on the provisioning node so lets configure them with the following:
# export PROV_CONN=ens3
# export MAIN_CONN=ens4
# sudo nmcli connection add ifname provisioning type bridge con-name provisioning
# sudo nmcli con add type bridge-slave ifname "$PROV_CONN" master provisioning
# sudo nmcli connection add ifname baremetal type bridge con-name baremetal
# sudo nmcli con add type bridge-slave ifname "$MAIN_CONN" master baremetal
# sudo nmcli con down "System $MAIN_CONN"; sudo pkill dhclient; sudo dhclient baremetal
# sudo nmcli connection modify provisioning ipv4.addresses 172.22.0.1/24 ipv4.method manual
# sudo nmcli con down provisioning
# sudo nmcli con up provisioning
# sudo ip a show $PROV_CONN;ip a show $MAIN_CONN; ip a show provisioning; ip a show baremetal
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master provisioning state UP group default qlen 1000
link/ether 52:54:00:9e:01:ec brd ff:ff:ff:ff:ff:ff
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master baremetal state UP group default qlen 1000
link/ether 52:54:00:07:98:b1 brd ff:ff:ff:ff:ff:ff
8: provisioning: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 52:54:00:9e:01:ec brd ff:ff:ff:ff:ff:ff
inet 172.22.0.1/24 brd 172.22.0.255 scope global noprefixroute provisioning
valid_lft forever preferred_lft forever
inet6 fe80::b189:6c77:d795:57dc/64 scope link noprefixroute
valid_lft forever preferred_lft forever
7: baremetal: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 52:54:00:07:98:b1 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.246/24 brd 192.168.0.255 scope global dynamic noprefixroute baremetal
valid_lft 366sec preferred_lft 366sec
inet6 fe80::495e:1100:2ad3:851e/64 scope link noprefixroute
valid_lft forever preferred_lft forever
At this time we should also pull the latest oc binary and openshift-baremetal-install binary. The oc command will be used to mirror the repository and also validate the cluster is up once installation is complete:
# export VERSION=$(curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/latest/release.txt | grep 'Name:' | awk -F: '{print $2}' | xargs)
# export RELEASE_IMAGE=$(curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/latest/release.txt | grep 'Pull From: quay.io' | awk -F ' ' '{print $3}' | xargs)
# export pullsecret_file=/home/bschmaus/pull-secret.json
# export cmd=openshift-baremetal-install
# export extract_dir=$(pwd)
# curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/latest/openshift-client-linux-$VERSION.tar.gz | tar zxvf - oc
# sudo cp /home/bschmaus/oc /usr/local/bin/oc
# /usr/local/bin/oc adm release extract --registry-config "${pullsecret_file}" --command=$cmd --to "${extract_dir}" ${RELEASE_IMAGE}
# sudo cp /home/bschmaus/openshift-baremetal-install /usr/local/bin/openshift-baremetal-install
Create Initial Install-Config.yaml and Local Image Repository:
Now that we have prepared the provisioning host, we need to first create our initial install-config.yaml file. The file should look similar to the sample below but adjusted for your environment (Note: RELEASEVERSION should be typed as in example as we will change that in later step):
apiVersion: v1
baseDomain: schmaustech.com
metadata:
name: kni5
networking:
machineCIDR: 192.168.0.0/24
compute:
- name: worker
replicas: 0
controlPlane:
name: master
replicas: 3
platform:
baremetal: {}
platform:
baremetal:
apiVIP: 192.168.0.199
ingressVIP: 192.168.0.197
dnsVIP: 192.168.0.198
hosts:
- name: master-0
role: master
bmc:
address: ipmi://192.168.0.11:6241
username: admin
password: password
bootMACAddress: 52:54:00:3d:04:ae
hardwareProfile: default
- name: master-1
role: master
bmc:
address: ipmi://192.168.0.11:6242
username: admin
password: password
bootMACAddress: 52:54:00:0f:91:f3
hardwareProfile: default
- name: master-2
role: master
bmc:
address: ipmi://192.168.0.11:6243
username: admin
password: password
bootMACAddress: 52:54:00:ee:d2:f2
hardwareProfile: default
sshKey: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDP+5QkRCiuhsYItXj7DzLcOIs2RbCgpMzDtPlt/hfLnDkLGozYIFapMp+o4l+6ornbZ3L+hYE0T8SyvyYVWfm1XpPcVgUIW6qp7yfEyTSRhpGnoY74PD33FIf6BtU2HoFLWjQcE6OrQOF0wijI3fgL0jSzvAxvYoXU/huMx/kI2jBcWEq5cADRfvpeYXhVEJLrIIOepoAZE1syaPT7jQEoLDfvxrDZPKObCOI2vzLiAQXI7gK1uc9YDb6IEA/4Ik4eV2R1+VCgKhgk5RUqn69+8a1o783g1tChKuLwA4K9lyEAbFBwlHMctfNOLeC1w+bYpDXH/3GydcYfq79/18dVd+xEUlzzC+2/qycWG36C1MxUZa2fXvSRWLnpkLcxtIes4MikFeIr3jkJlFUzITigzvFrKa2IKaJzQ53WsE++LVnKJfcFNLtWfdEOZMowG/KtgzSSac/iVEJRM2YTIJsQsqhhI4PTrqVlUy/NwcXOFfUF/NkF2deeUZ21Cdn+bKZDKtFu2x+ujyAWZKNq570YaFT3a4TrL6WmE9kdHnJOXYR61Tiq/1fU+y0fv1d0f1cYr4+mNRCGIZoQOgJraF7/YluLB23INkJgtbah/0t1xzSsQ59gzFhRlLkW9gQDekj2tOGJmZIuYCnTXGiqXHnri2yAPexgRiaFjoM3GCpsWw== bschmaus@lap1.schmaustech.com'
imageContentSources: - mirrors: - rhel8-ocp-auto.schmaustech.com:5000/ocp4/openshift4 source: registry.svc.ci.openshift.org/ocp/RELEASEVERSION - mirrors: - rhel8-ocp-auto.schmaustech.com:5000/ocp4/openshift4 source: registry.svc.ci.openshift.org/ocp/release pullSecret: 'PULL SECRET HERE'
Lets also create a pull-secret.json file that has the OpenShift pull-secret credentials so we can access the Openshift repository (Note: PULL-SECRET-JSON needs to be the actual pull-secret one gets from OpenShift):
# echo 'PULL-SECRET-JSON' > /home/bschmaus/pull-secret-json
Now that we have the initial install-config.yaml that we will use to deploy the OpenShift cluster lets change gears and configure the local image repository on the provisioning node. The steps below outline what is needed to configure the repository:
# sudo yum -y install podman httpd httpd-tools
# sudo mkdir -p /opt/registry/{auth,certs,data}
# sudo openssl req -newkey rsa:4096 -nodes -sha256 -keyout /opt/registry/certs/domain.key -x509 -days 365 -out /opt/registry/certs/domain.crt -subj "/C=US/ST=Minnesota/L=Brooklyn Park/O=Red Hat/OU=Engineering/CN=rhel8-ocp-auto.schmaustech.com"
# sudo cp /opt/registry/certs/domain.crt /etc/pki/ca-trust/source/anchors/
# sudo update-ca-trust extract
# sudo htpasswd -bBc /opt/registry/auth/htpasswd dummy dummy
# sudo firewall-cmd --add-port=5000/tcp --zone=libvirt --permanent
# sudo firewall-cmd --add-port=5000/tcp --zone=public --permanent
# sudo firewall-cmd --add-service=http --permanent
# sudo firewall-cmd --reload
# sudo podman create --name poc-registry -p 5000:5000 -v /opt/registry/data:/var/lib/registry:z -v /opt/registry/auth:/auth:z -e "REGISTRY_AUTH=htpasswd" -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry" -e "REGISTRY_HTTP_SECRET=ALongRandomSecretForRegistry" -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd -v /opt/registry/certs:/certs:z -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key docker.io/library/registry:2
# sudo podman start poc-registry
Test that the repository is working with the following test:
# curl -u dummy:dummy -k https://rhel8-ocp-auto.schmaustech.com:5000/v2/_catalog
Update Install-Config.yaml Pull Secret & Certificate:
Now we need to update the pull-secret in our install-config.yaml file to reference the local repository credentials we used above. To do this we first need to capture the base64 output for the user/password we configured for the local mirror which was dummy and dummy:
# echo -n 'dummy:dummy' | base64 -w0 ZHVtbXk6ZHVtbXk=
With the output above place it into a string like the sample below making sure to update the repository hostname as well to match the environment. Save the contents to a file called local_pull_secret:
# cat << 'EOF' > /home/bschmaus/local_pull_secret
pullSecret: '{ "auths": { "rhel8-ocp-auto.schmaustech.com:5000": {"auth": "ZHVtbXk6ZHVtbXk=","email": "bschmaus@redhat.com"} } }'
EOF
Now lets inject that local_pull_secret into our install-config.yaml file:
# sed -i '/^pullSecret/d' /home/bschmaus/install-config.yaml # cat /home/bschmaus/local_pull_secret >> /home/bschmaus/install-config.yaml
Since we also created a cert above and our install-config.yaml file will need to connect to the local repository lets go ahead and add that cert to the install-config.yaml:
# sudo cp /opt/registry/certs/domain.crt /home/bschmaus/domain.crt # sed -i -e 's/^/ /' /home/bschmaus/domain.crt # echo "additionalTrustBundle: |" >> /home/bschmaus/install-config.yaml # cat /home/bschmaus/domain.crt >> /home/bschmaus/install-config.yaml
Once the install-config.yaml file is updated with both the local repository pull-secret and domain cert our install-config.yaml will look something like the following:
apiVersion: v1
baseDomain: schmaustech.com
metadata:
name: kni5
networking:
machineCIDR: 192.168.0.0/24
compute:
- name: worker
replicas: 0
controlPlane:
name: master
replicas: 3
platform:
baremetal: {}
platform:
baremetal:
apiVIP: 192.168.0.199
ingressVIP: 192.168.0.197
dnsVIP: 192.168.0.198
hosts:
- name: master-0
role: master
bmc:
address: ipmi://192.168.0.11:6241
username: admin
password: password
bootMACAddress: 52:54:00:3d:04:ae
hardwareProfile: default
- name: master-1
role: master
bmc:
address: ipmi://192.168.0.11:6242
username: admin
password: password
bootMACAddress: 52:54:00:0f:91:f3
hardwareProfile: default
- name: master-2
role: master
bmc:
address: ipmi://192.168.0.11:6243
username: admin
password: password
bootMACAddress: 52:54:00:ee:d2:f2
hardwareProfile: default
sshKey: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDP+5QkRCiuhsYItXj7DzLcOIs2RbCgpMzDtPlt/hfLnDkLGozYIFapMp+o4l+6ornbZ3L+hYE0T8SyvyYVWfm1XpPcVgUIW6qp7yfEyTSRhpGnoY74PD33FIf6BtU2HoFLWjQcE6OrQOF0wijI3fgL0jSzvAxvYoXU/huMx/kI2jBcWEq5cADRfvpeYXhVEJLrIIOepoAZE1syaPT7jQEoLDfvxrDZPKObCOI2vzLiAQXI7gK1uc9YDb6IEA/4Ik4eV2R1+VCgKhgk5RUqn69+8a1o783g1tChKuLwA4K9lyEAbFBwlHMctfNOLeC1w+bYpDXH/3GydcYfq79/18dVd+xEUlzzC+2/qycWG36C1MxUZa2fXvSRWLnpkLcxtIes4MikFeIr3jkJlFUzITigzvFrKa2IKaJzQ53WsE++LVnKJfcFNLtWfdEOZMowG/KtgzSSac/iVEJRM2YTIJsQsqhhI4PTrqVlUy/NwcXOFfUF/NkF2deeUZ21Cdn+bKZDKtFu2x+ujyAWZKNq570YaFT3a4TrL6WmE9kdHnJOXYR61Tiq/1fU+y0fv1d0f1cYr4+mNRCGIZoQOgJraF7/YluLB23INkJgtbah/0t1xzSsQ59gzFhRlLkW9gQDekj2tOGJmZIuYCnTXGiqXHnri2yAPexgRiaFjoM3GCpsWw== bschmaus@bschmaus.remote.csb'
imageContentSources:
- mirrors:
- rhel8-ocp-auto.schmaustech.com:5000/ocp4/openshift4
source: registry.svc.ci.openshift.org/ocp/RELEASEVERSION
- mirrors:
- rhel8-ocp-auto.schmaustech.com:5000/ocp4/openshift4
source: registry.svc.ci.openshift.org/ocp/release
pullSecret: '{ "auths": { "rhel8-ocp-auto.schmaustech.com:5000": {"auth": "ZHVtbXk6ZHVtbXk=","email": "bschmaus@redhat.com"} } }'
additionalTrustBundle: |
-----BEGIN CERTIFICATE-----
MIIF9zCCA9+gAwIBAgIUJhBYhR40iyQOEWifRhKAjwupm4gwDQYJKoZIhvcNAQEL
BQAwgYoxCzAJBgNVBAYTAlVTMRIwEAYDVQQIDAlNaW5uZXNvdGExFjAUBgNVBAcM
DUJyb29rbHluIFBhcmsxEDAOBgNVBAoMB1JlZCBIYXQxFDASBgNVBAsMC0VuZ2lu
ZWVyaW5nMScwJQYDVQQDDB5yaGVsOC1vY3AtYXV0by5zY2htYXVzdGVjaC5jb20w
HhcNMTkxMTE3MjEwNjEzWhcNMjAxMTE2MjEwNjEzWjCBijELMAkGA1UEBhMCVVMx
EjAQBgNVBAgMCU1pbm5lc290YTEWMBQGA1UEBwwNQnJvb2tseW4gUGFyazEQMA4G
A1UECgwHUmVkIEhhdDEUMBIGA1UECwwLRW5naW5lZXJpbmcxJzAlBgNVBAMMHnJo
ZWw4LW9jcC1hdXRvLnNjaG1hdXN0ZWNoLmNvbTCCAiIwDQYJKoZIhvcNAQEBBQAD
ggIPADCCAgoCggIBAMiiLRYITGwaOXext9cTOpemfwBjfKx0eREEpqww9nz5qRPn
a0NK9Q5zgpieQvGWAN8y9c6wK0aUS4SRcFSx8RVbjP2L4p3ii0UsH6xP6JqHtfIg
ynbqco6CyfSq7k+GwmoMLFx0Tki3Ta6syVQ9pN/YctTPIeIBMOXkSeOxNkn8dtdG
oiCXS2OB0cW+wf3INp48Cc2zbbc+QmMw/LQlJIZcrP/C/Luh+fGiOt5XbrFD2Ain
FYkKHOEzGucuEHDC7f/wLqLkwUf6iN4aB0fVU7kdYn1C1aZJhmlY5tSzR3eRvNsL
QgFPfkHI647OTbDA8R4VNRRgkza+dEgEvLUbAyeQ7JztcFfnncWlDuvZdj6MCstR
MND4lw3Ig0AK9PYXb5ui5g57ms7tInmqVJKb/9Xp9DKSWEUEOe+TQBeZ6spWmQjg
GcPwuX5g+9RHh4iYmClxclDrZ6k6o6NQY9ldXQeI6kNVHacvg2S7iaWXFrshDtk4
ss5Lvlag4/+yLJUIiWeIyG8yBkDlH7d+OV8F9rhf86J7tgbcA+pxms0v11Ot0I4n
GibtZDmU0SPpsdAQzv798jhRiwJJMjh7Gw3EnnwxC/3GIK0LqqcKxMd3VmLRQYtb
VFKYJyK6LBml/iONAmCIcaLMc7CcoaalM7pmWAdyis3FFdIsYDFOQP7pWPHlAgMB
AAGjUzBRMB0GA1UdDgQWBBSVX7d9U2nc5LgiF7CFYABdTvEn8DAfBgNVHSMEGDAW
gBSVX7d9U2nc5LgiF7CFYABdTvEn8DAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3
DQEBCwUAA4ICAQCOzQ5oPbfAAinjhGRXuBPjlkm0bfKxW3sd+ExFOopVRYlCepzg
CQ3iiMkD2ygI6Ga1B8NhQ1kHkvZ9vISAG3R1Ew65Od/qoZ0vuCUlo9fG761Yp1i6
w/hR1MmcE0PL2EkyVz+GwNYjq445YD0aI3m3El9VPKoJPB6VHCBmII0qcCCTpag2
Uq77vzk4M8HYV/1lqRKH601pkaZOUul7KuS2a4GzNUNr90Fps1I0s45CE9NpyS0s
6cDd3RITlTHoNRBTeXWy5I5lWTqeJx5lwiGc+lkXQYCY/sVOtATwTSp6EqzvS08r
Q/4sZDxmcmGwig+afBa8Of71ndhzd5MxJHyOrhsfsGfwH6ThB8SaCliyvC/160MT
3LuXLn6OzGLcoLMcOhlJhhy7H7DnJeellVlSc/FHVr17IKInBd7viF4Sw5NGjH/d
q5peosB2tDkaGgOtgIMsuA7aYrilV+3ZZ3nx1Yipwju+9hU4ncCcO16OGC/bgRym
Bg6W8b9HZ+v1dvmh7aYHKDdZCXcNX6W/bWVC/rBpo4Cq+0jJso77CQYj95EdooSQ
kqc6bj8BpHfxU6o6nZ1Aqtfw17yPeqh6sfByn1yfLuhPBUGuU0mZQAmFGkwSP+HH
ZqLMKxvpJe5ufLz97O8gjCh38XV6mt3VUfVEn4Yrx5M2RtEMeuEtbYMCvw==
-----END CERTIFICATE-----
Mirror Images:
Now that we have configured all the services and configuration files we need for our OpenShift IPI deployment we can now begin the process of mirroring the images to the local respository:
# LATEST_CI_IMAGE=$(curl https://openshift-release.svc.ci.openshift.org/api/v1/releasestream/4.3.0-0.ci/latest | grep -o 'registry.svc.ci.openshift.org[^"]\+')
# export OPENSHIFT_RELEASE_IMAGE="${OPENSHIFT_RELEASE_IMAGE:-$LATEST_CI_IMAGE}"
# export GOPATH=/home/bschmaus/go
# export OCP_RELEASE=`echo $LATEST_CI_IMAGE|cut -d: -f2`
# export UPSTREAM_REPO=$LATEST_CI_IMAGE
# export LOCAL_REG='rhel8-ocp-auto.schmaustech.com:5000'
# export LOCAL_REPO='ocp4/openshift4'
# export LOCAL_SECRET_JSON="${HOME}/pull-secret.json"
# export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=${LOCAL_REG}/${LOCAL_REPO}:${OCP_RELEASE}
# /usr/local/bin/oc adm release mirror -a $LOCAL_SECRET_JSON --from=$UPSTREAM_REPO --to-release-image=$LOCAL_REG/$LOCAL_REPO:$OCP_RELEASE --to=$LOCAL_REG/$LOCAL_REPO
Adjust ImageContentSources in Install-Config.yaml:
Recall we original defined the install-config.yaml with a line that had RELEASEVERSION in all caps. Since we declared the OCP_RELEASE variable in the previous steps, we are now ready to update the install-config.yaml and replace RELEASEVERSION with the proper version defined for our local repository. The following steps initiate that change:
# NEW_RELEASE=`echo $OCP_RELEASE|sed s/.0-0.ci//g` # sed -i s/RELEASEVERSION/$NEW_RELEASE/g /home/bschmaus/install-config.yaml
Deploying the OpenShift Cluster:
Finally after all of the steps above, we can begin the actual deployment using the commands below:
# mkdir /home/bschmaus/ocp # cp /home/bschmaus/install-config.yaml /home/bschmaus/ocp # /usr/local/bin/openshift-baremetal-install --dir /home/bschmaus/ocp --log-level debug create cluster
If all the steps were followed, the cluster should successfully deploy with the exception of Metal3 container as there is an issue with this that needs to be addressed.
We can check if the cluster nodes are online with the following:
# export KUBECONFIG=/home/bschmaus/ocp/auth/kubeconfig # oc get nodes NAME STATUS ROLES AGE VERSION master-0.kni5.schmaustech.com Ready master,worker 10h v1.16.2 master-1.kni5.schmaustech.com Ready master,worker 10h v1.16.2 master-2.kni5.schmaustech.com Ready master,worker 10h v1.16.2
We can also view all the pods to validate things are online as well:
# oc get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE openshift-apiserver-operator openshift-apiserver-operator-589544b58f-hpzdj 1/1 Running 2 10h openshift-apiserver apiserver-gq5wv 1/1 Running 0 9h openshift-apiserver apiserver-gsds9 1/1 Running 0 9h openshift-apiserver apiserver-kqw6d 1/1 Running 0 9h openshift-authentication-operator authentication-operator-58d65b5d94-7s225 1/1 Running 0 9h openshift-authentication oauth-openshift-58c95b9459-jvnx5 1/1 Running 0 9h openshift-authentication oauth-openshift-58c95b9459-z9cbp 1/1 Running 0 9h openshift-cloud-credential-operator cloud-credential-operator-8c9748878-55n7k 1/1 Running 3 10h openshift-cluster-machine-approver machine-approver-6485cf466b-m6r9m 2/2 Running 0 10h openshift-cluster-node-tuning-operator cluster-node-tuning-operator-7668d5c85c-lt8vh 1/1 Running 0 9h openshift-cluster-node-tuning-operator tuned-fssr9 1/1 Running 0 9h openshift-cluster-node-tuning-operator tuned-nzq4q 1/1 Running 0 9h openshift-cluster-node-tuning-operator tuned-znl95 1/1 Running 0 9h openshift-cluster-samples-operator cluster-samples-operator-66fd64c57b-swnvx 2/2 Running 0 9h openshift-cluster-storage-operator cluster-storage-operator-698c8fc449-hzbqp 1/1 Running 0 9h openshift-cluster-version cluster-version-operator-7449dc5b9c-2kcb8 1/1 Running 0 10h openshift-console-operator console-operator-67bdf96b5b-lgzj4 1/1 Running 0 9h openshift-console console-6df4667b8c-4bw9l 1/1 Running 0 9h openshift-console console-6df4667b8c-m2pl7 1/1 Running 1 9h openshift-console downloads-65fdcc888-29t6m 1/1 Running 0 9h openshift-console downloads-65fdcc888-vh6tj 1/1 Running 0 9h openshift-controller-manager-operator openshift-controller-manager-operator-69bb4c6545-m9hf4 1/1 Running 2 10h openshift-controller-manager controller-manager-drrvv 1/1 Running 0 9h openshift-controller-manager controller-manager-fck8g 1/1 Running 0 9h openshift-controller-manager controller-manager-gnj6b 1/1 Running 0 9h openshift-dns-operator dns-operator-54d6dbb59b-wrjtl 1/1 Running 0 10h openshift-dns dns-default-dl2lq 2/2 Running 0 9h openshift-dns dns-default-vs8xd 2/2 Running 0 9h openshift-dns dns-default-wn4px 2/2 Running 0 9h openshift-etcd etcd-member-master-0.kni5.schmaustech.com 2/2 Running 0 10h openshift-etcd etcd-member-master-1.kni5.schmaustech.com 2/2 Running 0 10h openshift-etcd etcd-member-master-2.kni5.schmaustech.com 2/2 Running 0 10h openshift-image-registry cluster-image-registry-operator-788f556d9d-l9hrh 2/2 Running 0 9h openshift-ingress-operator ingress-operator-6f8d45d96f-4kw7x 1/1 Running 0 9h openshift-ingress router-default-5675955655-4wqw4 1/1 Running 0 9h openshift-ingress router-default-5675955655-rvjmq 1/1 Running 0 9h openshift-insights insights-operator-69b4497995-ltggd 1/1 Running 3 10h openshift-kni-infra coredns-master-0.kni5.schmaustech.com 1/1 Running 0 10h openshift-kni-infra coredns-master-1.kni5.schmaustech.com 1/1 Running 0 10h openshift-kni-infra coredns-master-2.kni5.schmaustech.com 1/1 Running 0 10h openshift-kni-infra haproxy-master-0.kni5.schmaustech.com 2/2 Running 2 10h openshift-kni-infra haproxy-master-1.kni5.schmaustech.com 2/2 Running 2 10h openshift-kni-infra haproxy-master-2.kni5.schmaustech.com 2/2 Running 2 10h openshift-kni-infra keepalived-master-0.kni5.schmaustech.com 2/2 Running 0 10h openshift-kni-infra keepalived-master-1.kni5.schmaustech.com 2/2 Running 0 10h openshift-kni-infra keepalived-master-2.kni5.schmaustech.com 2/2 Running 0 10h openshift-kni-infra mdns-publisher-master-0.kni5.schmaustech.com 1/1 Running 0 10h openshift-kni-infra mdns-publisher-master-1.kni5.schmaustech.com 1/1 Running 0 10h openshift-kni-infra mdns-publisher-master-2.kni5.schmaustech.com 1/1 Running 0 10h openshift-kube-apiserver-operator kube-apiserver-operator-79cc8666bd-sbcbq 1/1 Running 2 10h openshift-kube-apiserver installer-2-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver installer-2-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver installer-2-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver installer-3-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver installer-5-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver installer-5-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver installer-5-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver kube-apiserver-master-0.kni5.schmaustech.com 3/3 Running 0 9h openshift-kube-apiserver kube-apiserver-master-1.kni5.schmaustech.com 3/3 Running 0 9h openshift-kube-apiserver kube-apiserver-master-2.kni5.schmaustech.com 3/3 Running 1 9h openshift-kube-apiserver revision-pruner-2-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver revision-pruner-2-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver revision-pruner-2-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver revision-pruner-3-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver revision-pruner-5-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver revision-pruner-5-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver revision-pruner-5-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager-operator kube-controller-manager-operator-d46bf7586-ctjcd 1/1 Running 2 10h openshift-kube-controller-manager installer-2-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager installer-3-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager installer-3-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager installer-3-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager installer-4-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager installer-4-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager installer-4-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager installer-5-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager installer-5-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager installer-5-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager kube-controller-manager-master-0.kni5.schmaustech.com 3/3 Running 0 9h openshift-kube-controller-manager kube-controller-manager-master-1.kni5.schmaustech.com 3/3 Running 1 9h openshift-kube-controller-manager kube-controller-manager-master-2.kni5.schmaustech.com 3/3 Running 1 9h openshift-kube-controller-manager revision-pruner-2-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager revision-pruner-3-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager revision-pruner-3-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager revision-pruner-3-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager revision-pruner-4-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager revision-pruner-4-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager revision-pruner-4-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager revision-pruner-5-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager revision-pruner-5-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager revision-pruner-5-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler-operator openshift-kube-scheduler-operator-6ff9678df8-8xfnv 1/1 Running 2 10h openshift-kube-scheduler installer-2-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler installer-3-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler installer-5-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler installer-5-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler installer-5-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler installer-6-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler installer-6-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler installer-6-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler openshift-kube-scheduler-master-0.kni5.schmaustech.com 1/1 Running 1 9h openshift-kube-scheduler openshift-kube-scheduler-master-1.kni5.schmaustech.com 1/1 Running 0 9h openshift-kube-scheduler openshift-kube-scheduler-master-2.kni5.schmaustech.com 1/1 Running 1 9h openshift-kube-scheduler revision-pruner-2-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler revision-pruner-3-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler revision-pruner-5-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler revision-pruner-5-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler revision-pruner-5-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler revision-pruner-6-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler revision-pruner-6-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler revision-pruner-6-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-machine-api cluster-autoscaler-operator-7f977b7c45-975p5 2/2 Running 0 9h openshift-machine-api machine-api-controllers-556b5ffc85-nvm9k 4/4 Running 0 9h openshift-machine-api machine-api-operator-8589cc9889-wp8k6 2/2 Running 1 10h openshift-machine-api metal3-68b9cf8bf-kmfw2 0/8 Init:CreateContainerConfigError 0 9h openshift-machine-config-operator etcd-quorum-guard-6f6574bc9b-czhjz 1/1 Running 0 9h openshift-machine-config-operator etcd-quorum-guard-6f6574bc9b-gd5ms 1/1 Running 0 9h openshift-machine-config-operator etcd-quorum-guard-6f6574bc9b-zzjnb 1/1 Running 0 9h openshift-machine-config-operator machine-config-controller-9d8c59488-dm74p 1/1 Running 1 9h openshift-machine-config-operator machine-config-daemon-224vq 2/2 Running 0 9h openshift-machine-config-operator machine-config-daemon-hxhbc 2/2 Running 0 9h openshift-machine-config-operator machine-config-daemon-ngxbt 2/2 Running 0 9h openshift-machine-config-operator machine-config-operator-6cbbd79995-6b277 1/1 Running 1 10h openshift-machine-config-operator machine-config-server-lswt8 1/1 Running 0 9h openshift-machine-config-operator machine-config-server-mrvln 1/1 Running 0 9h openshift-machine-config-operator machine-config-server-nshlv 1/1 Running 0 9h openshift-marketplace certified-operators-7c44559446-w4bl2 1/1 Running 0 9h openshift-marketplace community-operators-64bfb7b678-7gs7q 1/1 Running 0 9h openshift-marketplace marketplace-operator-7776f6c9ff-mxls8 1/1 Running 0 9h openshift-marketplace redhat-operators-864d755755-49vh2 1/1 Running 0 9h openshift-monitoring alertmanager-main-0 3/3 Running 0 9h openshift-monitoring alertmanager-main-1 3/3 Running 0 9h openshift-monitoring alertmanager-main-2 3/3 Running 0 9h openshift-monitoring cluster-monitoring-operator-7c66dc45b4-8x6kl 1/1 Running 0 9h openshift-monitoring grafana-668586776b-dnfj7 2/2 Running 0 9h openshift-monitoring kube-state-metrics-75df8cfbdf-bgfjv 3/3 Running 0 9h openshift-monitoring node-exporter-7b4s9 2/2 Running 0 9h openshift-monitoring node-exporter-n459c 2/2 Running 0 9h openshift-monitoring node-exporter-vvfh2 2/2 Running 0 9h openshift-monitoring openshift-state-metrics-5b995c8497-2sv4m 3/3 Running 0 9h openshift-monitoring prometheus-adapter-bb778b866-cccw5 1/1 Running 0 9h openshift-monitoring prometheus-adapter-bb778b866-jxlwf 1/1 Running 0 9h openshift-monitoring prometheus-k8s-0 7/7 Running 1 9h openshift-monitoring prometheus-k8s-1 7/7 Running 1 9h openshift-monitoring prometheus-operator-6c4f54f97c-242ft 1/1 Running 0 9h openshift-monitoring thanos-querier-bf4f5dd76-bn8rb 4/4 Running 0 9h openshift-monitoring thanos-querier-bf4f5dd76-lcs69 4/4 Running 0 9h openshift-multus multus-admission-controller-899jw 1/1 Running 1 9h openshift-multus multus-admission-controller-bwgcb 1/1 Running 1 9h openshift-multus multus-admission-controller-s4rdl 1/1 Running 1 9h openshift-multus multus-mzfjv 1/1 Running 0 10h openshift-multus multus-qxq9z 1/1 Running 0 10h openshift-multus multus-zfs6k 1/1 Running 0 10h openshift-network-operator network-operator-55b786448c-fkt2l 1/1 Running 0 10h openshift-operator-lifecycle-manager catalog-operator-75b65486c-6xhwk 1/1 Running 0 10h openshift-operator-lifecycle-manager olm-operator-76cfbdc87f-cr4qm 1/1 Running 0 10h openshift-operator-lifecycle-manager packageserver-579759bb6d-44d84 1/1 Running 1 9h openshift-operator-lifecycle-manager packageserver-579759bb6d-x7srm 1/1 Running 1 9h openshift-sdn ovs-8pmrz 1/1 Running 0 10h openshift-sdn ovs-npwxk 1/1 Running 0 10h openshift-sdn ovs-svmwk 1/1 Running 0 10h openshift-sdn sdn-controller-d8rd8 1/1 Running 0 10h openshift-sdn sdn-controller-pjqxt 1/1 Running 0 10h openshift-sdn sdn-controller-zr2kr 1/1 Running 0 10h openshift-sdn sdn-q4w8d 1/1 Running 0 10h openshift-sdn sdn-v72dr 1/1 Running 0 10h openshift-sdn sdn-zjtc2 1/1 Running 0 10h openshift-service-ca-operator service-ca-operator-748f8bff-r4g94 1/1 Running 1 10h openshift-service-ca apiservice-cabundle-injector-549875965f-2dlq9 1/1 Running 1 9h openshift-service-ca configmap-cabundle-injector-6c49fc5d79-k4mbg 1/1 Running 1 9h openshift-service-ca service-serving-cert-signer-5fc7c8df6f-rlp6v 1/1 Running 1 9h openshift-service-catalog-apiserver-operator openshift-service-catalog-apiserver-operator-c7c9d4494-rqmv5 1/1 Running 0 9h openshift-service-catalog-controller-manager-operator openshift-service-catalog-controller-manager-operator-587bhqzmm 1/1 Running 0 9h
Subscribe to:
Posts (Atom)