When dealing with disconnected spoke clusters that are being deployed by Red Hat Advanced Cluster Management we have to be aware that any operators that we want to install into our disconnected spoke clusters also need to be mirrored into our local Red Hat Quay registry for them to be accessible by the spoke cluster. In this blog we will mirror down the Red Hat Advanced Cluster Manager operator image components because we need the agent images that normally would get started on a spoke cluster so the spoke cluster can properly join the Red Hat Advanced Cluster Manager hub and report in its metrics and status. Note this procedure could be modified to pull in any of the operators that are normally visible in OpenShift's OperatorHub.
Before we get started we need to ensure we have the following tools available to use: grpcurl, opm and podman. To install grpcurl we need to retrieve the proper release binary from the following github repository and extract it:
$ wget -q -O - "https://github.com/fullstorydev/grpcurl/releases/download/v1.8.6/grpcurl_1.8.6_linux_x86_64.tar.gz" | sudo tar -C /usr/local/bin/ -xvz LICENSE grpcurl [bschmaus@provisioning ~]$ which grpcurl /usr/local/bin/grpcurl
Next we need to pull the latest opm binary from the OpenShift mirror (in this case 4.10) and extract it:
$ curl https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/latest-4.10/opm-linux.tar.gz | sudo tar -C /usr/local/bin/ -xvz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 23.7M 0 135k 0 0 151k 0 0:02:40 --:--:-- 0:02:40 151kopm
100 23.7M 100 23.7M 0 0 7685k 0 0:00:03 0:00:03 --:--:-- 7685k
$ which opm
/usr/local/bin/opm
And finally we can use dnf install podman to install podman if it is not already there:
$ sudo dnf install podman Updating Subscription Management repositories. Last metadata expiration check: 1:32:12 ago on Fri 06 May 2022 11:22:36 AM CDT. Package podman-1:3.4.2-9.module+el8.5.0+13852+150547f7.x86_64 is already installed. Dependencies resolved. Nothing to do. Complete! $ which podman /usr/bin/podman
Now that we have our tools ready we can begin the process of mirroring the Red Hat Advanced Cluster Management operator and its corresponding images. The first step is to login to the source registry and the target registry. In my case the source registry is registry.redhat.io and the target registry is my local Quay poc-registry-quay-quay-poc.apps.kni20.schmaustech.com:
$ podman login registry.redhat.io Username: schmaustech Password: Login Succeeded! $ podman login poc-registry-quay-quay-poc.apps.kni20.schmaustech.com --tls-verify=false Username: openshift Password: Login Succeeded!
Next lets determine the list of packages we want to include in our pruned index of operators. We already know we just want Red Hat Advanced Cluster Management but this will provide the background context on how to get all the available operators in the event one would like to mirror more. We first need to start a source image index pod so we can extract out the list by executing the following:
$ podman run -p50051:50051 -it registry.redhat.io/redhat/redhat-operator-index:v4.10 WARN[0000] DEPRECATION NOTICE: Sqlite-based catalogs and their related subcommands are deprecated. Support for them will be removed in a future release. Please migrate your catalog workflows to the new file-based catalog format. WARN[0000] unable to set termination log path error="open /dev/termination-log: permission denied" INFO[0000] Keeping server open for infinite seconds database=/database/index.db port=50051 INFO[0000] serving registry database=/database/index.db port=50051
In another terminal window on the same host where the above podman command was run use grpcurl to extract out a list of operator packages and redirect it into packages.out file:
$ grpcurl -plaintext localhost:50051 api.Registry/ListPackages > packages.out
The packages.out file that was generate will contain a listing of all the different operators that could be potentially mirrored. If we go ahead and grep out cluster from the packages.out file we can see a few of those listing. The specific one we are interested in is at the top of the list: advanced-cluster-management.
$ grep cluster packages.out "name": "advanced-cluster-management" "name": "cluster-kube-descheduler-operator" "name": "cluster-logging" "name": "clusterresourceoverride" "name": "odf-multicluster-orchestrator" "name": "odr-cluster-operator"
Now that we have the exact name of the operator as listed in the operator index from registry.redhat.io we can now use that along with opm to generate our own operator registry index for our Red Hat Quay registry. To do this we will use the opm index prune command and specify the listing of packages we want. The output will create a local index:
$ opm index prune --from-index "registry.redhat.io/redhat/redhat-operator-index:v4.10" --packages 'advanced-cluster-management' --tag poc-registry-quay-quay-poc.apps.kni20.schmaustech.com/rhacm2/olm-index/redhat-oprator-index:v4.10 WARN[0000] DEPRECATION NOTICE: Sqlite-based catalogs and their related subcommands are deprecated. Support for them will be removed in a future release. Please migrate your catalog workflows to the new file-based catalog format. INFO[0000] pruning the index packages="[advanced-cluster-management]" INFO[0000] Pulling previous image registry.redhat.io/redhat/redhat-operator-index:v4.10 to get metadata packages="[advanced-cluster-management]" INFO[0000] running /usr/bin/podman pull registry.redhat.io/redhat/redhat-operator-index:v4.10 packages="[advanced-cluster-management]" INFO[0022] running /usr/bin/podman pull registry.redhat.io/redhat/redhat-operator-index:v4.10 packages="[advanced-cluster-management]" INFO[0024] Getting label data from previous image packages="[advanced-cluster-management]" INFO[0024] running podman inspect packages="[advanced-cluster-management]" INFO[0024] running podman create packages="[advanced-cluster-management]" INFO[0024] running podman cp packages="[advanced-cluster-management]" INFO[0029] running podman rm packages="[advanced-cluster-management]" INFO[0030] deleting packages pkg=3scale-operator INFO[0030] packages: [3scale-operator] pkg=3scale-operator INFO[0031] deleting packages pkg=amq-broker-rhel8 INFO[0031] packages: [amq-broker-rhel8] pkg=amq-broker-rhel8 INFO[0031] deleting packages pkg=amq-online INFO[0031] packages: [amq-online] pkg=amq-online INFO[0031] deleting packages pkg=amq-streams INFO[0031] packages: [amq-streams] pkg=amq-streams INFO[0031] deleting packages pkg=amq7-interconnect-operator INFO[0031] packages: [amq7-interconnect-operator] pkg=amq7-interconnect-operator INFO[0031] deleting packages pkg=ansible-automation-platform-operator INFO[0031] packages: [ansible-automation-platform-operator] pkg=ansible-automation-platform-operator INFO[0032] deleting packages pkg=ansible-cloud-addons-operator INFO[0032] packages: [ansible-cloud-addons-operator] pkg=ansible-cloud-addons-operator INFO[0032] deleting packages pkg=apicast-operator INFO[0032] packages: [apicast-operator] pkg=apicast-operator INFO[0032] deleting packages pkg=aws-efs-csi-driver-operator INFO[0032] packages: [aws-efs-csi-driver-operator] pkg=aws-efs-csi-driver-operator INFO[0032] deleting packages pkg=businessautomation-operator INFO[0032] packages: [businessautomation-operator] pkg=businessautomation-operator INFO[0032] deleting packages pkg=cincinnati-operator INFO[0032] packages: [cincinnati-operator] pkg=cincinnati-operator INFO[0032] deleting packages pkg=cluster-kube-descheduler-operator INFO[0032] packages: [cluster-kube-descheduler-operator] pkg=cluster-kube-descheduler-operator INFO[0032] deleting packages pkg=cluster-logging INFO[0032] packages: [cluster-logging] pkg=cluster-logging INFO[0032] deleting packages pkg=clusterresourceoverride INFO[0032] packages: [clusterresourceoverride] pkg=clusterresourceoverride INFO[0032] deleting packages pkg=codeready-workspaces INFO[0032] packages: [codeready-workspaces] pkg=codeready-workspaces INFO[0032] deleting packages pkg=codeready-workspaces2 INFO[0032] packages: [codeready-workspaces2] pkg=codeready-workspaces2 INFO[0032] deleting packages pkg=compliance-operator INFO[0032] packages: [compliance-operator] pkg=compliance-operator INFO[0033] deleting packages pkg=container-security-operator INFO[0033] packages: [container-security-operator] pkg=container-security-operator INFO[0033] deleting packages pkg=costmanagement-metrics-operator INFO[0033] packages: [costmanagement-metrics-operator] pkg=costmanagement-metrics-operator INFO[0033] deleting packages pkg=cryostat-operator INFO[0033] packages: [cryostat-operator] pkg=cryostat-operator INFO[0033] deleting packages pkg=datagrid INFO[0033] packages: [datagrid] pkg=datagrid INFO[0033] deleting packages pkg=devworkspace-operator INFO[0033] packages: [devworkspace-operator] pkg=devworkspace-operator INFO[0033] deleting packages pkg=dpu-network-operator INFO[0033] packages: [dpu-network-operator] pkg=dpu-network-operator INFO[0033] deleting packages pkg=eap INFO[0033] packages: [eap] pkg=eap INFO[0033] deleting packages pkg=elasticsearch-operator INFO[0033] packages: [elasticsearch-operator] pkg=elasticsearch-operator INFO[0033] deleting packages pkg=external-dns-operator INFO[0033] packages: [external-dns-operator] pkg=external-dns-operator INFO[0033] deleting packages pkg=file-integrity-operator INFO[0033] packages: [file-integrity-operator] pkg=file-integrity-operator INFO[0033] deleting packages pkg=fuse-apicurito INFO[0033] packages: [fuse-apicurito] pkg=fuse-apicurito INFO[0033] deleting packages pkg=fuse-console INFO[0033] packages: [fuse-console] pkg=fuse-console INFO[0033] deleting packages pkg=fuse-online INFO[0033] packages: [fuse-online] pkg=fuse-online INFO[0033] deleting packages pkg=gatekeeper-operator-product INFO[0033] packages: [gatekeeper-operator-product] pkg=gatekeeper-operator-product INFO[0033] deleting packages pkg=idp-mgmt-operator-product INFO[0033] packages: [idp-mgmt-operator-product] pkg=idp-mgmt-operator-product INFO[0033] deleting packages pkg=integration-operator INFO[0033] packages: [integration-operator] pkg=integration-operator INFO[0033] deleting packages pkg=jaeger-product INFO[0033] packages: [jaeger-product] pkg=jaeger-product INFO[0033] deleting packages pkg=jws-operator INFO[0033] packages: [jws-operator] pkg=jws-operator INFO[0033] deleting packages pkg=kiali-ossm INFO[0033] packages: [kiali-ossm] pkg=kiali-ossm INFO[0033] deleting packages pkg=klusterlet-product INFO[0033] packages: [klusterlet-product] pkg=klusterlet-product INFO[0033] deleting packages pkg=kubernetes-nmstate-operator INFO[0033] packages: [kubernetes-nmstate-operator] pkg=kubernetes-nmstate-operator INFO[0033] deleting packages pkg=kubevirt-hyperconverged INFO[0033] packages: [kubevirt-hyperconverged] pkg=kubevirt-hyperconverged INFO[0034] deleting packages pkg=local-storage-operator INFO[0034] packages: [local-storage-operator] pkg=local-storage-operator INFO[0034] deleting packages pkg=loki-operator INFO[0034] packages: [loki-operator] pkg=loki-operator INFO[0034] deleting packages pkg=mcg-operator INFO[0034] packages: [mcg-operator] pkg=mcg-operator INFO[0034] deleting packages pkg=metallb-operator INFO[0034] packages: [metallb-operator] pkg=metallb-operator INFO[0034] deleting packages pkg=mtc-operator INFO[0034] packages: [mtc-operator] pkg=mtc-operator INFO[0034] deleting packages pkg=mtv-operator INFO[0034] packages: [mtv-operator] pkg=mtv-operator INFO[0034] deleting packages pkg=nfd INFO[0034] packages: [nfd] pkg=nfd INFO[0034] deleting packages pkg=node-healthcheck-operator INFO[0034] packages: [node-healthcheck-operator] pkg=node-healthcheck-operator INFO[0034] deleting packages pkg=node-maintenance-operator INFO[0034] packages: [node-maintenance-operator] pkg=node-maintenance-operator INFO[0034] deleting packages pkg=numaresources-operator INFO[0034] packages: [numaresources-operator] pkg=numaresources-operator INFO[0034] deleting packages pkg=ocs-operator INFO[0034] packages: [ocs-operator] pkg=ocs-operator INFO[0034] deleting packages pkg=odf-csi-addons-operator INFO[0034] packages: [odf-csi-addons-operator] pkg=odf-csi-addons-operator INFO[0034] deleting packages pkg=odf-lvm-operator INFO[0034] packages: [odf-lvm-operator] pkg=odf-lvm-operator INFO[0034] deleting packages pkg=odf-multicluster-orchestrator INFO[0034] packages: [odf-multicluster-orchestrator] pkg=odf-multicluster-orchestrator INFO[0034] deleting packages pkg=odf-operator INFO[0034] packages: [odf-operator] pkg=odf-operator INFO[0034] deleting packages pkg=odr-cluster-operator INFO[0034] packages: [odr-cluster-operator] pkg=odr-cluster-operator INFO[0034] deleting packages pkg=odr-hub-operator INFO[0034] packages: [odr-hub-operator] pkg=odr-hub-operator INFO[0034] deleting packages pkg=openshift-cert-manager-operator INFO[0034] packages: [openshift-cert-manager-operator] pkg=openshift-cert-manager-operator INFO[0034] deleting packages pkg=openshift-gitops-operator INFO[0034] packages: [openshift-gitops-operator] pkg=openshift-gitops-operator INFO[0034] deleting packages pkg=openshift-pipelines-operator-rh INFO[0034] packages: [openshift-pipelines-operator-rh] pkg=openshift-pipelines-operator-rh INFO[0034] deleting packages pkg=openshift-secondary-scheduler-operator INFO[0034] packages: [openshift-secondary-scheduler-operator] pkg=openshift-secondary-scheduler-operator INFO[0034] deleting packages pkg=openshift-special-resource-operator INFO[0034] packages: [openshift-special-resource-operator] pkg=openshift-special-resource-operator INFO[0034] deleting packages pkg=opentelemetry-product INFO[0034] packages: [opentelemetry-product] pkg=opentelemetry-product INFO[0034] deleting packages pkg=performance-addon-operator INFO[0034] packages: [performance-addon-operator] pkg=performance-addon-operator INFO[0034] deleting packages pkg=poison-pill-manager INFO[0034] packages: [poison-pill-manager] pkg=poison-pill-manager INFO[0034] deleting packages pkg=ptp-operator INFO[0034] packages: [ptp-operator] pkg=ptp-operator INFO[0034] deleting packages pkg=quay-bridge-operator INFO[0034] packages: [quay-bridge-operator] pkg=quay-bridge-operator INFO[0034] deleting packages pkg=quay-operator INFO[0034] packages: [quay-operator] pkg=quay-operator INFO[0034] deleting packages pkg=red-hat-camel-k INFO[0034] packages: [red-hat-camel-k] pkg=red-hat-camel-k INFO[0034] deleting packages pkg=redhat-oadp-operator INFO[0034] packages: [redhat-oadp-operator] pkg=redhat-oadp-operator INFO[0034] deleting packages pkg=rh-service-binding-operator INFO[0034] packages: [rh-service-binding-operator] pkg=rh-service-binding-operator INFO[0034] deleting packages pkg=rhacs-operator INFO[0034] packages: [rhacs-operator] pkg=rhacs-operator INFO[0034] deleting packages pkg=rhpam-kogito-operator INFO[0034] packages: [rhpam-kogito-operator] pkg=rhpam-kogito-operator INFO[0035] deleting packages pkg=rhsso-operator INFO[0035] packages: [rhsso-operator] pkg=rhsso-operator INFO[0035] deleting packages pkg=sandboxed-containers-operator INFO[0035] packages: [sandboxed-containers-operator] pkg=sandboxed-containers-operator INFO[0035] deleting packages pkg=serverless-operator INFO[0035] packages: [serverless-operator] pkg=serverless-operator INFO[0035] deleting packages pkg=service-registry-operator INFO[0035] packages: [service-registry-operator] pkg=service-registry-operator INFO[0035] deleting packages pkg=servicemeshoperator INFO[0035] packages: [servicemeshoperator] pkg=servicemeshoperator INFO[0035] deleting packages pkg=skupper-operator INFO[0035] packages: [skupper-operator] pkg=skupper-operator INFO[0035] deleting packages pkg=sriov-network-operator INFO[0035] packages: [sriov-network-operator] pkg=sriov-network-operator INFO[0035] deleting packages pkg=submariner INFO[0035] packages: [submariner] pkg=submariner INFO[0035] deleting packages pkg=tang-operator INFO[0035] packages: [tang-operator] pkg=tang-operator INFO[0035] deleting packages pkg=vertical-pod-autoscaler INFO[0035] packages: [vertical-pod-autoscaler] pkg=vertical-pod-autoscaler INFO[0035] deleting packages pkg=web-terminal INFO[0035] packages: [web-terminal] pkg=web-terminal INFO[0035] deleting packages pkg=windows-machine-config-operator INFO[0035] packages: [windows-machine-config-operator] pkg=windows-machine-config-operator INFO[0035] Generating dockerfile packages="[advanced-cluster-management]" INFO[0035] writing dockerfile: ./index.Dockerfile2850553610 packages="[advanced-cluster-management]" INFO[0035] running podman build packages="[advanced-cluster-management]" INFO[0035] [podman build --format docker -f ./index.Dockerfile2850553610 -t poc-registry-quay-quay-poc.apps.kni20.schmaustech.com/rhacm2/olm-index/redhat-oprator-index:v4.10 .] packages="[advanced-cluster-management]"
With the index created we can now push it up to our Red Hat Quay registry with the podman push command. It should be noted that in this example we are pushing into the rhacm2 organization and that must exist before attempting the push.
$ podman push poc-registry-quay-quay-poc.apps.kni20.schmaustech.com/rhacm2/olm-index/redhat-oprator-index:v4.10 --tls-verify=false Getting image source signatures Copying blob 0d6867937695 done Copying blob eeaf5a4136cb done Copying blob 9dc1e45bb9ee done Copying blob 457de0330aa6 done Copying blob 324075f0d95e done Copying blob 5b1fa8e3e100 done Copying config f6bfd86300 done Writing manifest to image destination Storing signatures
Once we have pushed the index up we can then use the oc adm catalog mirror command to mirror the images:
$ oc adm catalog mirror poc-registry-quay-quay-poc.apps.kni20.schmaustech.com/rhacm2/olm-index/redhat-oprator-index:v4.10 poc-registry-quay-quay-poc.apps.kni20.schmaustech.com/rhacm2 -a /home/bschmaus/quay-merged-pull-secret.json --insecure
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! DEPRECATION NOTICE:
!! Sqlite-based catalogs are deprecated. Support for them will be removed in a
!! future release. Please migrate your catalog workflows to the new file-based
!! catalog format.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
src image has index label for database path: /database/index.db
using index path mapping: /database/index.db:/tmp/3687994887
wrote database to /tmp/3687994887
using database at: /tmp/3687994887/index.db
poc-registry-quay-quay-poc.apps.kni20.schmaustech.com/
rhacm2/openshift4-ose-configmap-reloader
blobs:
registry.redhat.io/openshift4/ose-configmap-reloader sha256:b77bb434db5a2c43574630adfbe80aa3b36c179ccc20541ae91e2812a3ad9ce2 1.461KiB
registry.redhat.io/openshift4/ose-configmap-reloader sha256:d8dcf8c7f6920565fec6db5c1479312aad177148dadd89e07f838cd7f44fa074 1.474KiB
registry.redhat.io/openshift4/ose-configmap-reloader sha256:7d2d8330490119f01d1087adb98a324b3292f4711436cc4f64a8d9cb081fc345 1.479KiB
(...)
uploading: poc-registry-quay-quay-poc.apps.kni20.schmaustech.com/rhacm2/rhacm2-klusterlet-addon-controller-rhel8 sha256:cf7ba91a4dc5dbc52d2fd9c09127be351a3d5292f1f2edf5171d2fe79f573850 18.93MiB
uploading: poc-registry-quay-quay-poc.apps.kni20.schmaustech.com/rhacm2/rhacm2-klusterlet-addon-controller-rhel8 sha256:83b47d1b1652022425cbee522218c3758b8c59b052d5e845d5f8d897e31609a7 18.93MiB
mounted: poc-registry-quay-quay-poc.apps.kni20.schmaustech.com/rhacm2/rhacm2-klusterlet-addon-controller-rhel8 sha256:2a99c93da16827d9a6254f86f495d2c72c62a916f9c398577577221d35d2c790 37.81MiB
mounted: poc-registry-quay-quay-poc.apps.kni20.schmaustech.com/rhacm2/rhacm2-klusterlet-addon-controller-rhel8 sha256:d46336f50433ab27336fad8f9b251b2f68a66d376c902dfca23a6851acae502c 37.47MiB
mounted: poc-registry-quay-quay-poc.apps.kni20.schmaustech.com/rhacm2/rhacm2-klusterlet-addon-controller-rhel8 sha256:0016483f9a1476d5d57b7a871ed4d3994ba802c643f53f6037d2b28f799f963f 35.94MiB
uploading: poc-registry-quay-quay-poc.apps.kni20.schmaustech.com/rhacm2/rhacm2-klusterlet-addon-controller-rhel8 sha256:59f461f3c96d9c3b9d77d9de5fc4df45e4f6965264b2d047d7474d130ca8f9b4 18.93MiB
mounted: poc-registry-quay-quay-poc.apps.kni20.schmaustech.com/rhacm2/rhacm2-klusterlet-addon-controller-rhel8 sha256:a9e23b64ace00a199db21d302292b434e9d3956d79319d958ecc19603d00c946 37.79MiB
uploading: poc-registry-quay-quay-poc.apps.kni20.schmaustech.com/rhacm2/rhacm2-klusterlet-addon-controller-rhel8 sha256:30ee6976ea1e5387884dd0299fd122b76ec6e1d4fdad01d01302997f2d461edc 18.63MiB
uploading: poc-registry-quay-quay-poc.apps.kni20.schmaustech.com/rhacm2/rhacm2-klusterlet-addon-controller-rhel8 sha256:c4e367c519079053c4297b06640529659c996e21823f2c580e533468b26a2de7 19.07MiB
sha256:acedcbb6483e2b6b51f69900de4f582f48a486114ef6ecaede82f1f549fb4ebf poc-registry-quay-quay-poc.apps.kni20.schmaustech.com/rhacm2/rhacm2-acm-must-gather-rhel8
(...)
sha256:bcf26708e40297fcc5c09657aa540408930e51e7319ec7612fb5529936746cc0 poc-registry-quay-quay-poc.apps.kni20.schmaustech.com/rhacm2/rhacm2-search-api-rhel8:2290f420
sha256:fcfbd48e615e46fe5d33e3059aedbc2a75f4f10489dbe91fa072610dbbe86130 poc-registry-quay-quay-poc.apps.kni20.schmaustech.com/rhacm2/rhacm2-search-api-rhel8:4ad02099
sha256:601f4d74ece9da8888488d303e05220dc9cd9796b07a95cf2001d3a42198b8de poc-registry-quay-quay-poc.apps.kni20.schmaustech.com/rhacm2/rhacm2-search-api-rhel8:2e28486f
sha256:623281445ffba2f86d4d2708e9aaa84b566a461e9bf29f703589cdb27492dcd3 poc-registry-quay-quay-poc.apps.kni20.schmaustech.com/rhacm2/rhacm2-search-api-rhel8:7fa459f6
info: Mirroring completed in 16m46.55s (44.49MB/s)
no digest mapping available for poc-registry-quay-quay-poc.apps.kni20.schmaustech.com/rhacm2/olm-index/redhat-operator-index:v4.10, skip writing to ImageContentSourcePolicy
wrote mirroring manifests to manifests-olm-index/redhat-operator-index-1651068403
deleted dir /tmp/2002258786
This concludes the demonstration of how to mirror specific operators down to ones own instance of Red Hat Quay.
