The following blog is a write up of the steps I used to generate a OpenShift IPI disconnected baremetal install. In this configuration we first mirror down the images to a local repository. Then deploy the Openshift cluster onto virtual machines that are mimicking a baremetal environment with vBMC being used as the IPMI interface into the virtual machines. Please note that while this was demonstrated in a virtual setting these steps should also work with physical hardware.
Lab Setup:
Physical node specification:
Processors: i7 - 8 vcpus Memory: 32gb Disk: 512gb M2 SSD Single NIC with 2 vlans (external and provisioning) tagged in Nested virtualization should be enabled
Virtual machine Node specification:
Processor: Passthrough from physical node - 4 vcpus Memory: 16gb Disk: 60gb raw image Two NICs ens3 (provisioning) ens4 (external) - no tags - dhcp interfaces IPMI: Provided via centralized Virtual BMC controller
Lab Diagram:
Predefined DNS records:
*.apps.kni5 IN A 192.168.0.197 ns1.kni5 IN A 192.168.0.198 api.kni5 IN A 192.168.0.199 master-0.kni5 IN A 192.168.0.200 master-1.kni5 IN A 192.168.0.201 master-2.kni5 IN A 192.168.0.202
Cluster and user definitions:
Preparing Provisioning Node:
The first step in preparing the provisioning node is to install, kickstart or image the provisioning node with RHEL8 and register the host. I leverage an ISO image local on NUC-1 with an embedded kickstart file that way I can rebuild my provisioning node on the fly and start off with a fresh install.
Once the provisioning node is installed make sure the following packages are installed:
Next lets make sure the username used for installation has passwordless sudo access. This is more about convenience give a lot of the commands require root and or sudo access:
Unfortunately at this time selinux does need to be set to permissive so we will do that now:
For the bootstrap node to boot on this virtual machine we need to ensure a default storage pool exists. On a RHEL8 installation this pool does not seem to exist out of the box so lets create it:
The Openshift installer expects there to be a baremetal and provisioning interface on the provisioning node so lets configure them with the following:
At this time we should also pull the latest oc binary and openshift-baremetal-install binary. The oc command will be used to mirror the repository and also validate the cluster is up once installation is complete:
Create Initial Install-Config.yaml and Local Image Repository:
Now that we have prepared the provisioning host, we need to first create our initial install-config.yaml file. The file should look similar to the sample below but adjusted for your environment (Note: RELEASEVERSION should be typed as in example as we will change that in later step):
Lets also create a pull-secret.json file that has the OpenShift pull-secret credentials so we can access the Openshift repository (Note: PULL-SECRET-JSON needs to be the actual pull-secret one gets from OpenShift):
Now that we have the initial install-config.yaml that we will use to deploy the OpenShift cluster lets change gears and configure the local image repository on the provisioning node. The steps below outline what is needed to configure the repository:
Test that the repository is working with the following test:
Update Install-Config.yaml Pull Secret & Certificate:
Now we need to update the pull-secret in our install-config.yaml file to reference the local repository credentials we used above. To do this we first need to capture the base64 output for the user/password we configured for the local mirror which was dummy and dummy:
With the output above place it into a string like the sample below making sure to update the repository hostname as well to match the environment. Save the contents to a file called local_pull_secret:
Now lets inject that local_pull_secret into our install-config.yaml file:
Since we also created a cert above and our install-config.yaml file will need to connect to the local repository lets go ahead and add that cert to the install-config.yaml:
Once the install-config.yaml file is updated with both the local repository pull-secret and domain cert our install-config.yaml will look something like the following:
Mirror Images:
Now that we have configured all the services and configuration files we need for our OpenShift IPI deployment we can now begin the process of mirroring the images to the local respository:
Adjust ImageContentSources in Install-Config.yaml:
Recall we original defined the install-config.yaml with a line that had RELEASEVERSION in all caps. Since we declared the OCP_RELEASE variable in the previous steps, we are now ready to update the install-config.yaml and replace RELEASEVERSION with the proper version defined for our local repository. The following steps initiate that change:
Deploying the OpenShift Cluster:
Finally after all of the steps above, we can begin the actual deployment using the commands below:
If all the steps were followed, the cluster should successfully deploy with the exception of Metal3 container as there is an issue with this that needs to be addressed.
We can check if the cluster nodes are online with the following:
We can also view all the pods to validate things are online as well:
Cluster Name: kni5 Domain Name: schmaustech.com Username for installation: bschmaus
Preparing Provisioning Node:
The first step in preparing the provisioning node is to install, kickstart or image the provisioning node with RHEL8 and register the host. I leverage an ISO image local on NUC-1 with an embedded kickstart file that way I can rebuild my provisioning node on the fly and start off with a fresh install.
Once the provisioning node is installed make sure the following packages are installed:
kexec-tools @development (package group) git usbredir golang libXv virt-install libvirt libvirt-devel libselinux-utils qemu-kvm mkisofs
Next lets make sure the username used for installation has passwordless sudo access. This is more about convenience give a lot of the commands require root and or sudo access:
# cat << EOF > /etc/sudoers.d/openshift Defaults:bschmaus !requiretty bschmaus ALL = (root) NOPASSWD:ALL EOF # chmod 600 /etc/sudoers.d/openshift
Unfortunately at this time selinux does need to be set to permissive so we will do that now:
# sudo setenforce permissive # sudo sed -i "s/=enforcing/=permissive/g" /etc/selinux/config
For the bootstrap node to boot on this virtual machine we need to ensure a default storage pool exists. On a RHEL8 installation this pool does not seem to exist out of the box so lets create it:
# sudo virsh pool-define-as --name default --type dir --target /var/lib/libvirt/images # sudo virsh pool-start default # sudo virsh pool-autostart default # sudo usermod --append --groups libvirt bschmaus
The Openshift installer expects there to be a baremetal and provisioning interface on the provisioning node so lets configure them with the following:
# export PROV_CONN=ens3
# export MAIN_CONN=ens4
# sudo nmcli connection add ifname provisioning type bridge con-name provisioning
# sudo nmcli con add type bridge-slave ifname "$PROV_CONN" master provisioning
# sudo nmcli connection add ifname baremetal type bridge con-name baremetal
# sudo nmcli con add type bridge-slave ifname "$MAIN_CONN" master baremetal
# sudo nmcli con down "System $MAIN_CONN"; sudo pkill dhclient; sudo dhclient baremetal
# sudo nmcli connection modify provisioning ipv4.addresses 172.22.0.1/24 ipv4.method manual
# sudo nmcli con down provisioning
# sudo nmcli con up provisioning
# sudo ip a show $PROV_CONN;ip a show $MAIN_CONN; ip a show provisioning; ip a show baremetal
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master provisioning state UP group default qlen 1000
link/ether 52:54:00:9e:01:ec brd ff:ff:ff:ff:ff:ff
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master baremetal state UP group default qlen 1000
link/ether 52:54:00:07:98:b1 brd ff:ff:ff:ff:ff:ff
8: provisioning: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 52:54:00:9e:01:ec brd ff:ff:ff:ff:ff:ff
inet 172.22.0.1/24 brd 172.22.0.255 scope global noprefixroute provisioning
valid_lft forever preferred_lft forever
inet6 fe80::b189:6c77:d795:57dc/64 scope link noprefixroute
valid_lft forever preferred_lft forever
7: baremetal: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 52:54:00:07:98:b1 brd ff:ff:ff:ff:ff:ff
inet 192.168.0.246/24 brd 192.168.0.255 scope global dynamic noprefixroute baremetal
valid_lft 366sec preferred_lft 366sec
inet6 fe80::495e:1100:2ad3:851e/64 scope link noprefixroute
valid_lft forever preferred_lft forever
At this time we should also pull the latest oc binary and openshift-baremetal-install binary. The oc command will be used to mirror the repository and also validate the cluster is up once installation is complete:
# export VERSION=$(curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/latest/release.txt | grep 'Name:' | awk -F: '{print $2}' | xargs)
# export RELEASE_IMAGE=$(curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/latest/release.txt | grep 'Pull From: quay.io' | awk -F ' ' '{print $3}' | xargs)
# export pullsecret_file=/home/bschmaus/pull-secret.json
# export cmd=openshift-baremetal-install
# export extract_dir=$(pwd)
# curl -s https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/latest/openshift-client-linux-$VERSION.tar.gz | tar zxvf - oc
# sudo cp /home/bschmaus/oc /usr/local/bin/oc
# /usr/local/bin/oc adm release extract --registry-config "${pullsecret_file}" --command=$cmd --to "${extract_dir}" ${RELEASE_IMAGE}
# sudo cp /home/bschmaus/openshift-baremetal-install /usr/local/bin/openshift-baremetal-install
Create Initial Install-Config.yaml and Local Image Repository:
Now that we have prepared the provisioning host, we need to first create our initial install-config.yaml file. The file should look similar to the sample below but adjusted for your environment (Note: RELEASEVERSION should be typed as in example as we will change that in later step):
apiVersion: v1
baseDomain: schmaustech.com
metadata:
name: kni5
networking:
machineCIDR: 192.168.0.0/24
compute:
- name: worker
replicas: 0
controlPlane:
name: master
replicas: 3
platform:
baremetal: {}
platform:
baremetal:
apiVIP: 192.168.0.199
ingressVIP: 192.168.0.197
dnsVIP: 192.168.0.198
hosts:
- name: master-0
role: master
bmc:
address: ipmi://192.168.0.11:6241
username: admin
password: password
bootMACAddress: 52:54:00:3d:04:ae
hardwareProfile: default
- name: master-1
role: master
bmc:
address: ipmi://192.168.0.11:6242
username: admin
password: password
bootMACAddress: 52:54:00:0f:91:f3
hardwareProfile: default
- name: master-2
role: master
bmc:
address: ipmi://192.168.0.11:6243
username: admin
password: password
bootMACAddress: 52:54:00:ee:d2:f2
hardwareProfile: default
sshKey: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDP+5QkRCiuhsYItXj7DzLcOIs2RbCgpMzDtPlt/hfLnDkLGozYIFapMp+o4l+6ornbZ3L+hYE0T8SyvyYVWfm1XpPcVgUIW6qp7yfEyTSRhpGnoY74PD33FIf6BtU2HoFLWjQcE6OrQOF0wijI3fgL0jSzvAxvYoXU/huMx/kI2jBcWEq5cADRfvpeYXhVEJLrIIOepoAZE1syaPT7jQEoLDfvxrDZPKObCOI2vzLiAQXI7gK1uc9YDb6IEA/4Ik4eV2R1+VCgKhgk5RUqn69+8a1o783g1tChKuLwA4K9lyEAbFBwlHMctfNOLeC1w+bYpDXH/3GydcYfq79/18dVd+xEUlzzC+2/qycWG36C1MxUZa2fXvSRWLnpkLcxtIes4MikFeIr3jkJlFUzITigzvFrKa2IKaJzQ53WsE++LVnKJfcFNLtWfdEOZMowG/KtgzSSac/iVEJRM2YTIJsQsqhhI4PTrqVlUy/NwcXOFfUF/NkF2deeUZ21Cdn+bKZDKtFu2x+ujyAWZKNq570YaFT3a4TrL6WmE9kdHnJOXYR61Tiq/1fU+y0fv1d0f1cYr4+mNRCGIZoQOgJraF7/YluLB23INkJgtbah/0t1xzSsQ59gzFhRlLkW9gQDekj2tOGJmZIuYCnTXGiqXHnri2yAPexgRiaFjoM3GCpsWw== bschmaus@lap1.schmaustech.com'
imageContentSources: - mirrors: - rhel8-ocp-auto.schmaustech.com:5000/ocp4/openshift4 source: registry.svc.ci.openshift.org/ocp/RELEASEVERSION - mirrors: - rhel8-ocp-auto.schmaustech.com:5000/ocp4/openshift4 source: registry.svc.ci.openshift.org/ocp/release pullSecret: 'PULL SECRET HERE'
Lets also create a pull-secret.json file that has the OpenShift pull-secret credentials so we can access the Openshift repository (Note: PULL-SECRET-JSON needs to be the actual pull-secret one gets from OpenShift):
# echo 'PULL-SECRET-JSON' > /home/bschmaus/pull-secret-json
Now that we have the initial install-config.yaml that we will use to deploy the OpenShift cluster lets change gears and configure the local image repository on the provisioning node. The steps below outline what is needed to configure the repository:
# sudo yum -y install podman httpd httpd-tools
# sudo mkdir -p /opt/registry/{auth,certs,data}
# sudo openssl req -newkey rsa:4096 -nodes -sha256 -keyout /opt/registry/certs/domain.key -x509 -days 365 -out /opt/registry/certs/domain.crt -subj "/C=US/ST=Minnesota/L=Brooklyn Park/O=Red Hat/OU=Engineering/CN=rhel8-ocp-auto.schmaustech.com"
# sudo cp /opt/registry/certs/domain.crt /etc/pki/ca-trust/source/anchors/
# sudo update-ca-trust extract
# sudo htpasswd -bBc /opt/registry/auth/htpasswd dummy dummy
# sudo firewall-cmd --add-port=5000/tcp --zone=libvirt --permanent
# sudo firewall-cmd --add-port=5000/tcp --zone=public --permanent
# sudo firewall-cmd --add-service=http --permanent
# sudo firewall-cmd --reload
# sudo podman create --name poc-registry -p 5000:5000 -v /opt/registry/data:/var/lib/registry:z -v /opt/registry/auth:/auth:z -e "REGISTRY_AUTH=htpasswd" -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry" -e "REGISTRY_HTTP_SECRET=ALongRandomSecretForRegistry" -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd -v /opt/registry/certs:/certs:z -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key docker.io/library/registry:2
# sudo podman start poc-registry
Test that the repository is working with the following test:
# curl -u dummy:dummy -k https://rhel8-ocp-auto.schmaustech.com:5000/v2/_catalog
Update Install-Config.yaml Pull Secret & Certificate:
Now we need to update the pull-secret in our install-config.yaml file to reference the local repository credentials we used above. To do this we first need to capture the base64 output for the user/password we configured for the local mirror which was dummy and dummy:
# echo -n 'dummy:dummy' | base64 -w0 ZHVtbXk6ZHVtbXk=
With the output above place it into a string like the sample below making sure to update the repository hostname as well to match the environment. Save the contents to a file called local_pull_secret:
# cat << 'EOF' > /home/bschmaus/local_pull_secret
pullSecret: '{ "auths": { "rhel8-ocp-auto.schmaustech.com:5000": {"auth": "ZHVtbXk6ZHVtbXk=","email": "bschmaus@redhat.com"} } }'
EOF
Now lets inject that local_pull_secret into our install-config.yaml file:
# sed -i '/^pullSecret/d' /home/bschmaus/install-config.yaml # cat /home/bschmaus/local_pull_secret >> /home/bschmaus/install-config.yaml
Since we also created a cert above and our install-config.yaml file will need to connect to the local repository lets go ahead and add that cert to the install-config.yaml:
# sudo cp /opt/registry/certs/domain.crt /home/bschmaus/domain.crt # sed -i -e 's/^/ /' /home/bschmaus/domain.crt # echo "additionalTrustBundle: |" >> /home/bschmaus/install-config.yaml # cat /home/bschmaus/domain.crt >> /home/bschmaus/install-config.yaml
Once the install-config.yaml file is updated with both the local repository pull-secret and domain cert our install-config.yaml will look something like the following:
apiVersion: v1
baseDomain: schmaustech.com
metadata:
name: kni5
networking:
machineCIDR: 192.168.0.0/24
compute:
- name: worker
replicas: 0
controlPlane:
name: master
replicas: 3
platform:
baremetal: {}
platform:
baremetal:
apiVIP: 192.168.0.199
ingressVIP: 192.168.0.197
dnsVIP: 192.168.0.198
hosts:
- name: master-0
role: master
bmc:
address: ipmi://192.168.0.11:6241
username: admin
password: password
bootMACAddress: 52:54:00:3d:04:ae
hardwareProfile: default
- name: master-1
role: master
bmc:
address: ipmi://192.168.0.11:6242
username: admin
password: password
bootMACAddress: 52:54:00:0f:91:f3
hardwareProfile: default
- name: master-2
role: master
bmc:
address: ipmi://192.168.0.11:6243
username: admin
password: password
bootMACAddress: 52:54:00:ee:d2:f2
hardwareProfile: default
sshKey: 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDP+5QkRCiuhsYItXj7DzLcOIs2RbCgpMzDtPlt/hfLnDkLGozYIFapMp+o4l+6ornbZ3L+hYE0T8SyvyYVWfm1XpPcVgUIW6qp7yfEyTSRhpGnoY74PD33FIf6BtU2HoFLWjQcE6OrQOF0wijI3fgL0jSzvAxvYoXU/huMx/kI2jBcWEq5cADRfvpeYXhVEJLrIIOepoAZE1syaPT7jQEoLDfvxrDZPKObCOI2vzLiAQXI7gK1uc9YDb6IEA/4Ik4eV2R1+VCgKhgk5RUqn69+8a1o783g1tChKuLwA4K9lyEAbFBwlHMctfNOLeC1w+bYpDXH/3GydcYfq79/18dVd+xEUlzzC+2/qycWG36C1MxUZa2fXvSRWLnpkLcxtIes4MikFeIr3jkJlFUzITigzvFrKa2IKaJzQ53WsE++LVnKJfcFNLtWfdEOZMowG/KtgzSSac/iVEJRM2YTIJsQsqhhI4PTrqVlUy/NwcXOFfUF/NkF2deeUZ21Cdn+bKZDKtFu2x+ujyAWZKNq570YaFT3a4TrL6WmE9kdHnJOXYR61Tiq/1fU+y0fv1d0f1cYr4+mNRCGIZoQOgJraF7/YluLB23INkJgtbah/0t1xzSsQ59gzFhRlLkW9gQDekj2tOGJmZIuYCnTXGiqXHnri2yAPexgRiaFjoM3GCpsWw== bschmaus@bschmaus.remote.csb'
imageContentSources:
- mirrors:
- rhel8-ocp-auto.schmaustech.com:5000/ocp4/openshift4
source: registry.svc.ci.openshift.org/ocp/RELEASEVERSION
- mirrors:
- rhel8-ocp-auto.schmaustech.com:5000/ocp4/openshift4
source: registry.svc.ci.openshift.org/ocp/release
pullSecret: '{ "auths": { "rhel8-ocp-auto.schmaustech.com:5000": {"auth": "ZHVtbXk6ZHVtbXk=","email": "bschmaus@redhat.com"} } }'
additionalTrustBundle: |
-----BEGIN CERTIFICATE-----
MIIF9zCCA9+gAwIBAgIUJhBYhR40iyQOEWifRhKAjwupm4gwDQYJKoZIhvcNAQEL
BQAwgYoxCzAJBgNVBAYTAlVTMRIwEAYDVQQIDAlNaW5uZXNvdGExFjAUBgNVBAcM
DUJyb29rbHluIFBhcmsxEDAOBgNVBAoMB1JlZCBIYXQxFDASBgNVBAsMC0VuZ2lu
ZWVyaW5nMScwJQYDVQQDDB5yaGVsOC1vY3AtYXV0by5zY2htYXVzdGVjaC5jb20w
HhcNMTkxMTE3MjEwNjEzWhcNMjAxMTE2MjEwNjEzWjCBijELMAkGA1UEBhMCVVMx
EjAQBgNVBAgMCU1pbm5lc290YTEWMBQGA1UEBwwNQnJvb2tseW4gUGFyazEQMA4G
A1UECgwHUmVkIEhhdDEUMBIGA1UECwwLRW5naW5lZXJpbmcxJzAlBgNVBAMMHnJo
ZWw4LW9jcC1hdXRvLnNjaG1hdXN0ZWNoLmNvbTCCAiIwDQYJKoZIhvcNAQEBBQAD
ggIPADCCAgoCggIBAMiiLRYITGwaOXext9cTOpemfwBjfKx0eREEpqww9nz5qRPn
a0NK9Q5zgpieQvGWAN8y9c6wK0aUS4SRcFSx8RVbjP2L4p3ii0UsH6xP6JqHtfIg
ynbqco6CyfSq7k+GwmoMLFx0Tki3Ta6syVQ9pN/YctTPIeIBMOXkSeOxNkn8dtdG
oiCXS2OB0cW+wf3INp48Cc2zbbc+QmMw/LQlJIZcrP/C/Luh+fGiOt5XbrFD2Ain
FYkKHOEzGucuEHDC7f/wLqLkwUf6iN4aB0fVU7kdYn1C1aZJhmlY5tSzR3eRvNsL
QgFPfkHI647OTbDA8R4VNRRgkza+dEgEvLUbAyeQ7JztcFfnncWlDuvZdj6MCstR
MND4lw3Ig0AK9PYXb5ui5g57ms7tInmqVJKb/9Xp9DKSWEUEOe+TQBeZ6spWmQjg
GcPwuX5g+9RHh4iYmClxclDrZ6k6o6NQY9ldXQeI6kNVHacvg2S7iaWXFrshDtk4
ss5Lvlag4/+yLJUIiWeIyG8yBkDlH7d+OV8F9rhf86J7tgbcA+pxms0v11Ot0I4n
GibtZDmU0SPpsdAQzv798jhRiwJJMjh7Gw3EnnwxC/3GIK0LqqcKxMd3VmLRQYtb
VFKYJyK6LBml/iONAmCIcaLMc7CcoaalM7pmWAdyis3FFdIsYDFOQP7pWPHlAgMB
AAGjUzBRMB0GA1UdDgQWBBSVX7d9U2nc5LgiF7CFYABdTvEn8DAfBgNVHSMEGDAW
gBSVX7d9U2nc5LgiF7CFYABdTvEn8DAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3
DQEBCwUAA4ICAQCOzQ5oPbfAAinjhGRXuBPjlkm0bfKxW3sd+ExFOopVRYlCepzg
CQ3iiMkD2ygI6Ga1B8NhQ1kHkvZ9vISAG3R1Ew65Od/qoZ0vuCUlo9fG761Yp1i6
w/hR1MmcE0PL2EkyVz+GwNYjq445YD0aI3m3El9VPKoJPB6VHCBmII0qcCCTpag2
Uq77vzk4M8HYV/1lqRKH601pkaZOUul7KuS2a4GzNUNr90Fps1I0s45CE9NpyS0s
6cDd3RITlTHoNRBTeXWy5I5lWTqeJx5lwiGc+lkXQYCY/sVOtATwTSp6EqzvS08r
Q/4sZDxmcmGwig+afBa8Of71ndhzd5MxJHyOrhsfsGfwH6ThB8SaCliyvC/160MT
3LuXLn6OzGLcoLMcOhlJhhy7H7DnJeellVlSc/FHVr17IKInBd7viF4Sw5NGjH/d
q5peosB2tDkaGgOtgIMsuA7aYrilV+3ZZ3nx1Yipwju+9hU4ncCcO16OGC/bgRym
Bg6W8b9HZ+v1dvmh7aYHKDdZCXcNX6W/bWVC/rBpo4Cq+0jJso77CQYj95EdooSQ
kqc6bj8BpHfxU6o6nZ1Aqtfw17yPeqh6sfByn1yfLuhPBUGuU0mZQAmFGkwSP+HH
ZqLMKxvpJe5ufLz97O8gjCh38XV6mt3VUfVEn4Yrx5M2RtEMeuEtbYMCvw==
-----END CERTIFICATE-----
Mirror Images:
Now that we have configured all the services and configuration files we need for our OpenShift IPI deployment we can now begin the process of mirroring the images to the local respository:
# LATEST_CI_IMAGE=$(curl https://openshift-release.svc.ci.openshift.org/api/v1/releasestream/4.3.0-0.ci/latest | grep -o 'registry.svc.ci.openshift.org[^"]\+')
# export OPENSHIFT_RELEASE_IMAGE="${OPENSHIFT_RELEASE_IMAGE:-$LATEST_CI_IMAGE}"
# export GOPATH=/home/bschmaus/go
# export OCP_RELEASE=`echo $LATEST_CI_IMAGE|cut -d: -f2`
# export UPSTREAM_REPO=$LATEST_CI_IMAGE
# export LOCAL_REG='rhel8-ocp-auto.schmaustech.com:5000'
# export LOCAL_REPO='ocp4/openshift4'
# export LOCAL_SECRET_JSON="${HOME}/pull-secret.json"
# export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=${LOCAL_REG}/${LOCAL_REPO}:${OCP_RELEASE}
# /usr/local/bin/oc adm release mirror -a $LOCAL_SECRET_JSON --from=$UPSTREAM_REPO --to-release-image=$LOCAL_REG/$LOCAL_REPO:$OCP_RELEASE --to=$LOCAL_REG/$LOCAL_REPO
Adjust ImageContentSources in Install-Config.yaml:
Recall we original defined the install-config.yaml with a line that had RELEASEVERSION in all caps. Since we declared the OCP_RELEASE variable in the previous steps, we are now ready to update the install-config.yaml and replace RELEASEVERSION with the proper version defined for our local repository. The following steps initiate that change:
# NEW_RELEASE=`echo $OCP_RELEASE|sed s/.0-0.ci//g` # sed -i s/RELEASEVERSION/$NEW_RELEASE/g /home/bschmaus/install-config.yaml
Deploying the OpenShift Cluster:
Finally after all of the steps above, we can begin the actual deployment using the commands below:
# mkdir /home/bschmaus/ocp # cp /home/bschmaus/install-config.yaml /home/bschmaus/ocp # /usr/local/bin/openshift-baremetal-install --dir /home/bschmaus/ocp --log-level debug create cluster
If all the steps were followed, the cluster should successfully deploy with the exception of Metal3 container as there is an issue with this that needs to be addressed.
We can check if the cluster nodes are online with the following:
# export KUBECONFIG=/home/bschmaus/ocp/auth/kubeconfig # oc get nodes NAME STATUS ROLES AGE VERSION master-0.kni5.schmaustech.com Ready master,worker 10h v1.16.2 master-1.kni5.schmaustech.com Ready master,worker 10h v1.16.2 master-2.kni5.schmaustech.com Ready master,worker 10h v1.16.2
We can also view all the pods to validate things are online as well:
# oc get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE openshift-apiserver-operator openshift-apiserver-operator-589544b58f-hpzdj 1/1 Running 2 10h openshift-apiserver apiserver-gq5wv 1/1 Running 0 9h openshift-apiserver apiserver-gsds9 1/1 Running 0 9h openshift-apiserver apiserver-kqw6d 1/1 Running 0 9h openshift-authentication-operator authentication-operator-58d65b5d94-7s225 1/1 Running 0 9h openshift-authentication oauth-openshift-58c95b9459-jvnx5 1/1 Running 0 9h openshift-authentication oauth-openshift-58c95b9459-z9cbp 1/1 Running 0 9h openshift-cloud-credential-operator cloud-credential-operator-8c9748878-55n7k 1/1 Running 3 10h openshift-cluster-machine-approver machine-approver-6485cf466b-m6r9m 2/2 Running 0 10h openshift-cluster-node-tuning-operator cluster-node-tuning-operator-7668d5c85c-lt8vh 1/1 Running 0 9h openshift-cluster-node-tuning-operator tuned-fssr9 1/1 Running 0 9h openshift-cluster-node-tuning-operator tuned-nzq4q 1/1 Running 0 9h openshift-cluster-node-tuning-operator tuned-znl95 1/1 Running 0 9h openshift-cluster-samples-operator cluster-samples-operator-66fd64c57b-swnvx 2/2 Running 0 9h openshift-cluster-storage-operator cluster-storage-operator-698c8fc449-hzbqp 1/1 Running 0 9h openshift-cluster-version cluster-version-operator-7449dc5b9c-2kcb8 1/1 Running 0 10h openshift-console-operator console-operator-67bdf96b5b-lgzj4 1/1 Running 0 9h openshift-console console-6df4667b8c-4bw9l 1/1 Running 0 9h openshift-console console-6df4667b8c-m2pl7 1/1 Running 1 9h openshift-console downloads-65fdcc888-29t6m 1/1 Running 0 9h openshift-console downloads-65fdcc888-vh6tj 1/1 Running 0 9h openshift-controller-manager-operator openshift-controller-manager-operator-69bb4c6545-m9hf4 1/1 Running 2 10h openshift-controller-manager controller-manager-drrvv 1/1 Running 0 9h openshift-controller-manager controller-manager-fck8g 1/1 Running 0 9h openshift-controller-manager controller-manager-gnj6b 1/1 Running 0 9h openshift-dns-operator dns-operator-54d6dbb59b-wrjtl 1/1 Running 0 10h openshift-dns dns-default-dl2lq 2/2 Running 0 9h openshift-dns dns-default-vs8xd 2/2 Running 0 9h openshift-dns dns-default-wn4px 2/2 Running 0 9h openshift-etcd etcd-member-master-0.kni5.schmaustech.com 2/2 Running 0 10h openshift-etcd etcd-member-master-1.kni5.schmaustech.com 2/2 Running 0 10h openshift-etcd etcd-member-master-2.kni5.schmaustech.com 2/2 Running 0 10h openshift-image-registry cluster-image-registry-operator-788f556d9d-l9hrh 2/2 Running 0 9h openshift-ingress-operator ingress-operator-6f8d45d96f-4kw7x 1/1 Running 0 9h openshift-ingress router-default-5675955655-4wqw4 1/1 Running 0 9h openshift-ingress router-default-5675955655-rvjmq 1/1 Running 0 9h openshift-insights insights-operator-69b4497995-ltggd 1/1 Running 3 10h openshift-kni-infra coredns-master-0.kni5.schmaustech.com 1/1 Running 0 10h openshift-kni-infra coredns-master-1.kni5.schmaustech.com 1/1 Running 0 10h openshift-kni-infra coredns-master-2.kni5.schmaustech.com 1/1 Running 0 10h openshift-kni-infra haproxy-master-0.kni5.schmaustech.com 2/2 Running 2 10h openshift-kni-infra haproxy-master-1.kni5.schmaustech.com 2/2 Running 2 10h openshift-kni-infra haproxy-master-2.kni5.schmaustech.com 2/2 Running 2 10h openshift-kni-infra keepalived-master-0.kni5.schmaustech.com 2/2 Running 0 10h openshift-kni-infra keepalived-master-1.kni5.schmaustech.com 2/2 Running 0 10h openshift-kni-infra keepalived-master-2.kni5.schmaustech.com 2/2 Running 0 10h openshift-kni-infra mdns-publisher-master-0.kni5.schmaustech.com 1/1 Running 0 10h openshift-kni-infra mdns-publisher-master-1.kni5.schmaustech.com 1/1 Running 0 10h openshift-kni-infra mdns-publisher-master-2.kni5.schmaustech.com 1/1 Running 0 10h openshift-kube-apiserver-operator kube-apiserver-operator-79cc8666bd-sbcbq 1/1 Running 2 10h openshift-kube-apiserver installer-2-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver installer-2-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver installer-2-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver installer-3-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver installer-5-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver installer-5-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver installer-5-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver kube-apiserver-master-0.kni5.schmaustech.com 3/3 Running 0 9h openshift-kube-apiserver kube-apiserver-master-1.kni5.schmaustech.com 3/3 Running 0 9h openshift-kube-apiserver kube-apiserver-master-2.kni5.schmaustech.com 3/3 Running 1 9h openshift-kube-apiserver revision-pruner-2-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver revision-pruner-2-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver revision-pruner-2-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver revision-pruner-3-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver revision-pruner-5-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver revision-pruner-5-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-apiserver revision-pruner-5-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager-operator kube-controller-manager-operator-d46bf7586-ctjcd 1/1 Running 2 10h openshift-kube-controller-manager installer-2-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager installer-3-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager installer-3-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager installer-3-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager installer-4-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager installer-4-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager installer-4-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager installer-5-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager installer-5-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager installer-5-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager kube-controller-manager-master-0.kni5.schmaustech.com 3/3 Running 0 9h openshift-kube-controller-manager kube-controller-manager-master-1.kni5.schmaustech.com 3/3 Running 1 9h openshift-kube-controller-manager kube-controller-manager-master-2.kni5.schmaustech.com 3/3 Running 1 9h openshift-kube-controller-manager revision-pruner-2-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager revision-pruner-3-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager revision-pruner-3-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager revision-pruner-3-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager revision-pruner-4-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager revision-pruner-4-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager revision-pruner-4-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager revision-pruner-5-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager revision-pruner-5-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-controller-manager revision-pruner-5-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler-operator openshift-kube-scheduler-operator-6ff9678df8-8xfnv 1/1 Running 2 10h openshift-kube-scheduler installer-2-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler installer-3-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler installer-5-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler installer-5-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler installer-5-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler installer-6-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler installer-6-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler installer-6-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler openshift-kube-scheduler-master-0.kni5.schmaustech.com 1/1 Running 1 9h openshift-kube-scheduler openshift-kube-scheduler-master-1.kni5.schmaustech.com 1/1 Running 0 9h openshift-kube-scheduler openshift-kube-scheduler-master-2.kni5.schmaustech.com 1/1 Running 1 9h openshift-kube-scheduler revision-pruner-2-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler revision-pruner-3-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler revision-pruner-5-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler revision-pruner-5-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler revision-pruner-5-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler revision-pruner-6-master-0.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler revision-pruner-6-master-1.kni5.schmaustech.com 0/1 Completed 0 9h openshift-kube-scheduler revision-pruner-6-master-2.kni5.schmaustech.com 0/1 Completed 0 9h openshift-machine-api cluster-autoscaler-operator-7f977b7c45-975p5 2/2 Running 0 9h openshift-machine-api machine-api-controllers-556b5ffc85-nvm9k 4/4 Running 0 9h openshift-machine-api machine-api-operator-8589cc9889-wp8k6 2/2 Running 1 10h openshift-machine-api metal3-68b9cf8bf-kmfw2 0/8 Init:CreateContainerConfigError 0 9h openshift-machine-config-operator etcd-quorum-guard-6f6574bc9b-czhjz 1/1 Running 0 9h openshift-machine-config-operator etcd-quorum-guard-6f6574bc9b-gd5ms 1/1 Running 0 9h openshift-machine-config-operator etcd-quorum-guard-6f6574bc9b-zzjnb 1/1 Running 0 9h openshift-machine-config-operator machine-config-controller-9d8c59488-dm74p 1/1 Running 1 9h openshift-machine-config-operator machine-config-daemon-224vq 2/2 Running 0 9h openshift-machine-config-operator machine-config-daemon-hxhbc 2/2 Running 0 9h openshift-machine-config-operator machine-config-daemon-ngxbt 2/2 Running 0 9h openshift-machine-config-operator machine-config-operator-6cbbd79995-6b277 1/1 Running 1 10h openshift-machine-config-operator machine-config-server-lswt8 1/1 Running 0 9h openshift-machine-config-operator machine-config-server-mrvln 1/1 Running 0 9h openshift-machine-config-operator machine-config-server-nshlv 1/1 Running 0 9h openshift-marketplace certified-operators-7c44559446-w4bl2 1/1 Running 0 9h openshift-marketplace community-operators-64bfb7b678-7gs7q 1/1 Running 0 9h openshift-marketplace marketplace-operator-7776f6c9ff-mxls8 1/1 Running 0 9h openshift-marketplace redhat-operators-864d755755-49vh2 1/1 Running 0 9h openshift-monitoring alertmanager-main-0 3/3 Running 0 9h openshift-monitoring alertmanager-main-1 3/3 Running 0 9h openshift-monitoring alertmanager-main-2 3/3 Running 0 9h openshift-monitoring cluster-monitoring-operator-7c66dc45b4-8x6kl 1/1 Running 0 9h openshift-monitoring grafana-668586776b-dnfj7 2/2 Running 0 9h openshift-monitoring kube-state-metrics-75df8cfbdf-bgfjv 3/3 Running 0 9h openshift-monitoring node-exporter-7b4s9 2/2 Running 0 9h openshift-monitoring node-exporter-n459c 2/2 Running 0 9h openshift-monitoring node-exporter-vvfh2 2/2 Running 0 9h openshift-monitoring openshift-state-metrics-5b995c8497-2sv4m 3/3 Running 0 9h openshift-monitoring prometheus-adapter-bb778b866-cccw5 1/1 Running 0 9h openshift-monitoring prometheus-adapter-bb778b866-jxlwf 1/1 Running 0 9h openshift-monitoring prometheus-k8s-0 7/7 Running 1 9h openshift-monitoring prometheus-k8s-1 7/7 Running 1 9h openshift-monitoring prometheus-operator-6c4f54f97c-242ft 1/1 Running 0 9h openshift-monitoring thanos-querier-bf4f5dd76-bn8rb 4/4 Running 0 9h openshift-monitoring thanos-querier-bf4f5dd76-lcs69 4/4 Running 0 9h openshift-multus multus-admission-controller-899jw 1/1 Running 1 9h openshift-multus multus-admission-controller-bwgcb 1/1 Running 1 9h openshift-multus multus-admission-controller-s4rdl 1/1 Running 1 9h openshift-multus multus-mzfjv 1/1 Running 0 10h openshift-multus multus-qxq9z 1/1 Running 0 10h openshift-multus multus-zfs6k 1/1 Running 0 10h openshift-network-operator network-operator-55b786448c-fkt2l 1/1 Running 0 10h openshift-operator-lifecycle-manager catalog-operator-75b65486c-6xhwk 1/1 Running 0 10h openshift-operator-lifecycle-manager olm-operator-76cfbdc87f-cr4qm 1/1 Running 0 10h openshift-operator-lifecycle-manager packageserver-579759bb6d-44d84 1/1 Running 1 9h openshift-operator-lifecycle-manager packageserver-579759bb6d-x7srm 1/1 Running 1 9h openshift-sdn ovs-8pmrz 1/1 Running 0 10h openshift-sdn ovs-npwxk 1/1 Running 0 10h openshift-sdn ovs-svmwk 1/1 Running 0 10h openshift-sdn sdn-controller-d8rd8 1/1 Running 0 10h openshift-sdn sdn-controller-pjqxt 1/1 Running 0 10h openshift-sdn sdn-controller-zr2kr 1/1 Running 0 10h openshift-sdn sdn-q4w8d 1/1 Running 0 10h openshift-sdn sdn-v72dr 1/1 Running 0 10h openshift-sdn sdn-zjtc2 1/1 Running 0 10h openshift-service-ca-operator service-ca-operator-748f8bff-r4g94 1/1 Running 1 10h openshift-service-ca apiservice-cabundle-injector-549875965f-2dlq9 1/1 Running 1 9h openshift-service-ca configmap-cabundle-injector-6c49fc5d79-k4mbg 1/1 Running 1 9h openshift-service-ca service-serving-cert-signer-5fc7c8df6f-rlp6v 1/1 Running 1 9h openshift-service-catalog-apiserver-operator openshift-service-catalog-apiserver-operator-c7c9d4494-rqmv5 1/1 Running 0 9h openshift-service-catalog-controller-manager-operator openshift-service-catalog-controller-manager-operator-587bhqzmm 1/1 Running 0 9h
