Practical Example of Red Hat Image Mode: Bootc

Red Hat Image Mode is a new approach to operating system (OS) deployment that lets users build, deploy, and manage Red Hat Enterprise Linux as a bootc container image. It reduces complexity across the enterprise by letting development, operations, and solution providers use the same container-native tools and techniques to manage everything from applications to the underlying OS.

The following blog is the step by step process I took to create a Bootc image mode iso image. The following steps were carried out on a Red Hat Enterprise Linux 9.4 host as root though sudo and a regular user could also be used.

The first requirement was getting the container tools and so I used the following to get those installed.

# dnf install container-tools
Updating Subscription Management repositories.
Last metadata expiration check: 2:20:17 ago on Fri 26 Jul 2024 07:17:27 AM CDT.
Dependencies resolved.
==============================================================================================================================================================================================================================================
 Package                                                Architecture                                  Version                                                   Repository                                                               Size
==============================================================================================================================================================================================================================================
Installing:
 container-tools                                        noarch                                        1-14.el9                                                  rhel-9-for-x86_64-appstream-rpms                                        8.3 k
Upgrading:
 aardvark-dns                                           x86_64                                        2:1.10.0-3.el9_4                                          rhel-9-for-x86_64-appstream-rpms                                        969 k
 buildah                                                x86_64                                        2:1.33.7-3.el9_4                                          rhel-9-for-x86_64-appstream-rpms                                        9.4 M
 netavark                                               x86_64                                        2:1.10.3-1.el9                                            rhel-9-for-x86_64-appstream-rpms                                        4.0 M
 podman                                                 x86_64                                        4:4.9.4-6.el9_4                                           rhel-9-for-x86_64-appstream-rpms                                         16 M
Installing dependencies:
 cockpit-podman                                         noarch                                        84.1-1.el9                                                rhel-9-for-x86_64-appstream-rpms                                        683 k
 podman-docker                                          noarch                                        4:4.9.4-6.el9_4                                           rhel-9-for-x86_64-appstream-rpms                                        106 k
 podman-remote                                          x86_64                                        4:4.9.4-6.el9_4                                           rhel-9-for-x86_64-appstream-rpms                                         10 M
 python3-podman                                         noarch                                        3:4.9.0-1.el9                                             rhel-9-for-x86_64-appstream-rpms                                        178 k
 python3-pyxdg                                          noarch                                        0.27-3.el9                                                rhel-9-for-x86_64-appstream-rpms                                        108 k
 python3-tomli                                          noarch                                        2.0.1-5.el9                                               rhel-9-for-x86_64-appstream-rpms                                         37 k
 skopeo                                                 x86_64                                        2:1.14.3-0.1.el9                                          rhel-9-for-x86_64-appstream-rpms                                        8.5 M
 toolbox                                                x86_64                                        0.0.99.5-2.el9                                            rhel-9-for-x86_64-appstream-rpms                                        2.5 M
 udica                                                  noarch                                        0.2.8-1.el9                                               rhel-9-for-x86_64-appstream-rpms                                         54 k

Transaction Summary
==============================================================================================================================================================================================================================================
Install  10 Packages
Upgrade   4 Packages

Total download size: 52 M
Is this ok [y/N]: y
Downloading Packages:
(1/14): container-tools-1-14.el9.noarch.rpm                                                                                                                                                                    29 kB/s | 8.3 kB     00:00    
(2/14): python3-tomli-2.0.1-5.el9.noarch.rpm                                                                                                                                                                  126 kB/s |  37 kB     00:00    
(3/14): python3-pyxdg-0.27-3.el9.noarch.rpm                                                                                                                                                                   347 kB/s | 108 kB     00:00    
(4/14): cockpit-podman-84.1-1.el9.noarch.rpm                                                                                                                                                                  3.0 MB/s | 683 kB     00:00    
(5/14): udica-0.2.8-1.el9.noarch.rpm                                                                                                                                                                          616 kB/s |  54 kB     00:00    
(6/14): python3-podman-4.9.0-1.el9.noarch.rpm                                                                                                                                                                 1.3 MB/s | 178 kB     00:00    
(7/14): podman-docker-4.9.4-6.el9_4.noarch.rpm                                                                                                                                                                1.2 MB/s | 106 kB     00:00    
(8/14): toolbox-0.0.99.5-2.el9.x86_64.rpm                                                                                                                                                                     4.6 MB/s | 2.5 MB     00:00    
(9/14): netavark-1.10.3-1.el9.x86_64.rpm                                                                                                                                                                      4.4 MB/s | 4.0 MB     00:00    
(10/14): aardvark-dns-1.10.0-3.el9_4.x86_64.rpm                                                                                                                                                               3.3 MB/s | 969 kB     00:00    
(11/14): skopeo-1.14.3-0.1.el9.x86_64.rpm                                                                                                                                                                     3.9 MB/s | 8.5 MB     00:02    
(12/14): podman-remote-4.9.4-6.el9_4.x86_64.rpm                                                                                                                                                               3.2 MB/s |  10 MB     00:03    
(13/14): buildah-1.33.7-3.el9_4.x86_64.rpm                                                                                                                                                                    3.5 MB/s | 9.4 MB     00:02    
(14/14): podman-4.9.4-6.el9_4.x86_64.rpm                                                                                                                                                                      5.3 MB/s |  16 MB     00:02    
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                         9.6 MB/s |  52 MB     00:05     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                      1/1 
  Upgrading        : aardvark-dns-2:1.10.0-3.el9_4.x86_64                                                                                                                                                                                1/18 
  Upgrading        : netavark-2:1.10.3-1.el9.x86_64                                                                                                                                                                                      2/18 
  Upgrading        : podman-4:4.9.4-6.el9_4.x86_64                                                                                                                                                                                       3/18 
  Installing       : skopeo-2:1.14.3-0.1.el9.x86_64                                                                                                                                                                                      4/18 
  (...)
  Verifying        : podman-4:4.9.4-6.el9_4.x86_64                                                                                                                                                                                      17/18 
  Verifying        : podman-2:4.6.1-8.el9_3.x86_64                                                                                                                                                                                      18/18 
Installed products updated.

Upgraded:
  aardvark-dns-2:1.10.0-3.el9_4.x86_64                           buildah-2:1.33.7-3.el9_4.x86_64                           netavark-2:1.10.3-1.el9.x86_64                           podman-4:4.9.4-6.el9_4.x86_64                          
Installed:
  cockpit-podman-84.1-1.el9.noarch      container-tools-1-14.el9.noarch      podman-docker-4:4.9.4-6.el9_4.noarch      podman-remote-4:4.9.4-6.el9_4.x86_64      python3-podman-3:4.9.0-1.el9.noarch      python3-pyxdg-0.27-3.el9.noarch     
  python3-tomli-2.0.1-5.el9.noarch      skopeo-2:1.14.3-0.1.el9.x86_64       toolbox-0.0.99.5-2.el9.x86_64             udica-0.2.8-1.el9.noarch                 

Complete!

Once the tooling is in place we need to login to register.redhat.io with a Red Hat account.

# podman login registry.redhat.io
Username: myusername
Password: 
Login Succeeded!

Next we need to pull two images for this workflow locally:

• The latest Red Hat Bootc Image Builder image
• The latest Red Hat Bootc image

First let's pull the image builder image.

# sudo podman pull registry.redhat.io/rhel9/bootc-image-builder
Trying to pull registry.redhat.io/rhel9/bootc-image-builder:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob edab65b863ae done   | 
Copying blob ce39a10ee5db done   | 
Copying blob 6e743249fd30 done   | 
Copying config 7e467a06cb done   | 
Writing manifest to image destination
Storing signatures
7e467a06cbc49d0e601ab5acad54afcf16c7fd3187296c74f2780ec3e758977a

Then we can pull the Red Hat Enterprise Linux Bootc image. Note if there is a need to build a custom Bootc image that can also be done as well and is documented here

# podman pull registry.redhat.io/rhel9/rhel-bootc:latest
Trying to pull registry.redhat.io/rhel9/rhel-bootc:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob b696b6658912 done   | 
Copying blob b696b6658912 done   | 
Copying blob e0b929cd893f done   | 
(...)
Copying blob 2942d3f50802 done   | 
Copying blob 8ea0992b56d4 done   | 
Copying blob ad312c5c40cc done   | 
Copying blob bd9ddc54bea9 done   | 
Copying config 482f4c67cc done   | 
Writing manifest to image destination
Storing signatures
482f4c67cc158fa4b8db27c09832d3133bc45b1d989aa6d166ca2ef45f6c7178

Once we have pulled our images let's just review what we have with podman images.

# podman images
REPOSITORY                                    TAG         IMAGE ID      CREATED     SIZE
registry.redhat.io/rhel9/rhel-bootc           latest      482f4c67cc15  4 days ago  1.47 GB
registry.redhat.io/rhel9/bootc-image-builder  latest      7e467a06cbc4  4 days ago  521 MB

Next I created a directory structure under root home directory.

# mkdir ~/bootc
# cd bootc
# mkdir output

I also created a config.toml file which allows us to customize the image. In this example I embedding my user/password, public ssh-key and the groups I should belong to.

# cat <<EOL > config.toml 
[[blueprint.customizations.user]]
name = "myuser"
password = "password"
key = "ssh-rsa publick-key"
groups = ["wheel"]
EOL

Before we begin building let's review where we are and what is in the directory structure.

# pwd
/root/bootc
# ls
config.toml  output

Before we begin the build process we need to workaround an issue here by removing the signatures in our local copy of the RHEL Bootc image.

# skopeo copy --remove-signatures containers-storage:registry.redhat.io/rhel9/rhel-bootc:latest containers-storage:registry.redhat.io/rhel9/rhel-bootc:latest
INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled 
Copying blob f2d952b04649 skipped: already exists  
Copying blob bc1abff2c8d4 skipped: already exists  
Copying blob 491341ca1509 skipped: already exists  
(...)
Copying blob 6c118cde0f5c skipped: already exists  
Copying blob 5f70bf18a086 skipped: already exists  
Copying blob 12787d84fa13 skipped: already exists  
Copying config 482f4c67cc done   | 
Writing manifest to image destination

If everything looks good we can proceed to run the build process which consists of using podman to run the bootc-image-builder while passing in some directories and referencing the starting image we will use to build our ISO. The entire build process happens within a container and the ISO generated is dumped to the output directory we have mapped into the container. The process will take a bit to run and the log output is very long. I have provided the complete log run here with the condensed version below.

# podman run --rm -it --privileged --security-opt label=type:unconfined_t -v /var/lib/containers/storage:/var/lib/containers/storage -v /root/bootc/output:/output -v /root/bootc/config.toml:/config.toml registry.redhat.io/rhel9/bootc-image-builder --type iso --config /config.toml --local registry.redhat.io/rhel9/rhel-bootc:latest
Generating manifest manifest-iso.json
DONE
Building manifest-iso.json
starting -Pipeline source org.osbuild.containers-storage: 51102953cf5005007cf8b3bd76c39a0ae558aa34f311915042cdc0bc4c1fb246
Build
  root: <host>
Pipeline source org.osbuild.curl: 07337b98b3c859adfb37b011d83cf0511884147bf999e7869ffbf9074b529a4f
Build
  root: <host>
(...)
Writing to 'stdio:/run/osbuild/tree/install.iso' completed successfully.

⏱  Duration: 3s
org.osbuild.implantisomd5: bf37713d1bdb752cb80271b8243583e86665b003c69abff0e1730de24398fac1 {
  "filename": "install.iso"
}
['implantisomd5', '/run/osbuild/tree/install.iso']
Inserting md5sum into iso image...
md5 = 11cf490ec817e7904ba35961d5323195
Inserting fragment md5sums into iso image...
fragmd5 = 4f7338c869faf1a2881d46dc4ac1eebf46148d4bf2fa1d59149d74f43195
frags = 20
Setting supported flag to 0

⏱  Duration: 3s
manifest - finished successfully
build:        9133fb8610ab053dae7e281e6a6655dbb912c4530d32e4da75c06b8713a87c80
anaconda-tree:    fcd61d1236a42900977530f12f45fe452f2f0c8bf3c80a7ba60cb45ffe4bf36d
rootfs-image:    0a517e05ab42f947beec8dae4d2da338ca9cc7fe17b1daba013e24b1c60aeadf
efiboot-tree:    61a20c820b40436ce7bd6d1a74c6b97a05f7c8800b678083942e814cf9f7cc0e
bootiso-tree:    53d4713e1d036ca2d06ae3b0b64c8b9a4efb63cbc6fed4f13afe1e984b7af04d
bootiso:      bf37713d1bdb752cb80271b8243583e86665b003c69abff0e1730de24398fac1
Build complete!
Results saved in
.

Once the build process completes we will find an install.iso in the output/bootiso/ directory.

# cd ~/bootc/output/bootiso/
# ls -l
total 2211840
-rw-r--r--. 1 root root 2264924160 Jul 26 10:38 install.iso

The installation iso that was created can then be used to install RedHat Enterprise Linux on another host automatically and rather quickly compared to the old package based method.

Hopefully this provides a simple example to understand the concept of building and consuming Red Hat Image Mode.

Slimming Down RHEL 8 Image

I recently had a project that required the Red Hat Enterprise Linux 8 Binary DVD due to the package requirements.  This normally would not be an issue but there were further requirements that made this a bit more challenging.  Those requirements were that the image also needed to be under 4gb in size and could not be delivered via networking.   This sets up the challenge of trying to devise a way to take the DVD image which is over 6gb and par it down leaving only what is required behind.  The following will describe the process the I used to meets these requirements and while there might be other methods this was the procedure I used.

The first thing we need to do is download the RHEL 8 Binary DVD image to a running system.  Once on the system I created a directory called /rhel, mounted up the image and copied the contents to /rhel:

# mount ./rhel-8.0-x86_64-dvd.iso /mnt/ -o loop
# mkdir /rhel
# shopt -s dotglob
# cp -ai /mnt/* /rhel
# umount /mnt

The next step was to create a list of packages that I needed from an existing RHEL 8 system that was installed for the purpose of the project.   I just used a simple rpm command but formatted it just for the package name without the version.   The reason being is we just want the package name as versions might be different from image and current system due to applied errata on running system:

# rpm -qa --queryformat='%{NAME}\n' | sort -n > rhel8.lst

Now I need to get a list of packages from the RHEL 8 image and in this version there are two directories that hold packages: Appstream and BaseOS.  In my example I only needed to parse out the Appstream packages to gain enough image reduction.  So I will run the following command to grab the list:

# ls -1 /rhel/AppStream/Packages/*|xargs -n 1 basename > appstream.lst

Now that I have my two lists I needed to create some logic to be able to remove packages that I would not need and this required a comparison of the two lists.  The following script should be run from with in the /rhel directory and on successful comparison it will keep the matched packages in /rhel/AppStream/Packages.  If there is no match the package will be removed.

#!/bin/bash
while read package; do
    flag="0"
    while read rhel8; do
        if [[ "$package" =~ ^"$rhel8" ]]
        then
            flag="1"
            echo KEEP: $package $rhel8 $flag
            break
        fi
    done < rhel8.lst
    if [[ "$flag" = "0" ]]
    then
        echo REMOVE $package $rhel8 $flag
        find ./ -name $package -print -exec rm -r -f {} \;
    fi
done < appstream.lst

Now that we have removed the excessive packages from the /rhel directory structure, which if you recall is just the contents of the RHEL 8 image, we can now begin to make a new iso image.  But first since we changed the AppStream packaging we need to update the repository data.  We do this by using createrepo command:

# cd /rhel/AppStream
# createrepo  -g repodata/*comps*.xml . --update
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete

At this point we can go ahead and create a new image using the following command inside /rhel:

# mkisofs -o /tmp/rhel8-slim.iso -b isolinux/isolinux.bin -c isolinux/boot.cat --no-emul-boot --boot-load-size 4 --boot-info-table -J -R -V RHEL-8-0-0-BaseOS-x86_64 .

Once the command runs the new image will be written out to /tmp/rhel8-slim.iso and if we are lucky the size is now greatly diminished:

# ls -lh /tmp/rhel8-slim.iso
-rw-r--r--. 1 root root 2.0G Jul  1 15:17 /tmp/rhel8-slim.iso

And there it is a slimmer 2gb RHEL 8 image that contains only the packages that are needed for this specific project.   Be aware this method could also be used to add custom packages to RHEL 8 as well or even script customization and/or configurations.  The flexibility is limited only to ones imagination.

Stacking OpenShift with Rook and CNV

In previous blogs I was working with Rook/Ceph on Kubernetes and demonstrating how to setup a Ceph cluster and even replace failed OSDs. With that in mind I wanted to shift gears a bit and bring it more into alignment with OpenShift and Container Native Virtualization(CNV).

The following blog will guide us through a simple OpenShift deployment with Rook/Ceph and CNV configured. I will also demonstrate the use of a Rook PVC that provides the back end storage for a CNV deployed virtual instance.

The configuration for this lab is four virtual machines where one node is the master and compute and the other 3 nodes compute.  Each of these nodes has a base install of Red Hat Enterprise Linux 7 on it and the physical host they are on allows for nested virtualization.

Before we start with the installation of various software lets make sure we do a bit of user setup to ensure our install runs smoothly.  The next few steps will need to be done on all nodes to ensure a user origin (this could be any non root user) is created and has sudo rights without use of a password:

# useradd origin
# passwd origin
# echo -e 'Defaults:origin !requiretty\norigin ALL = (root) NOPASSWD:ALL' | tee /etc/sudoers.d/openshift 
# chmod 440 /etc/sudoers.d/openshift

Then we need to perform the the following steps to setup keyless authentication for the origin user from the master node to the rest of the nodes that will make up the cluster:

# ssh-keygen -q -N ""
# vi /home/origin/.ssh/config
Host ocp-master
    Hostname ocp-master.schmaustech.com
    User origin
Host ocp-node1
    Hostname ocp-node1.schmaustech.com
    User origin
Host ocp-node2
    Hostname ocp-node2.schmaustech.com
    User origin
Host ocp-node3
    Hostname ocp-node3.schmaustech.com
    User origin

# chmod 600 /home/origin/.ssh/config
# ssh-copy-id ocp-master
# ssh-copy-id ocp-node1
# ssh-copy-id ocp-node2
# ssh-copy-id ocp-node3

Now we can move on to enabling the necessary repositries on all nodes to ensure we can get access to the right packages we will need for installation:

[origin@ocp-master ~]$ sudo subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-extras-rpms --enable=rhel-7-server-rh-common-rpms --enable=rhel-7-server-ose-3.11-rpms --enable=rhel-7-server-ansible-2.6-rpms --enable=rhel-7-server-cnv-1.4-tech-preview-rpms

Next lets install the initial required packages on all the nodes:

[origin@ocp-master ~]$ sudo yum -y install openshift-ansible docker-1.13.1 kubevirt-ansible kubevirt-virtctl

On the master node lets configure the Ansible hosts file for our OpenShift installation.   The following is the example I used and I simply replaced /etc/ansible/hosts with it.

[OSEv3:children]
masters
nodes
etcd
[OSEv3:vars]
# admin user created in previous section
ansible_ssh_user=origin
ansible_become=true
oreg_url=registry.access.redhat.com/openshift3/ose-${component}:${version}
openshift_deployment_type=openshift-enterprise
#  use HTPasswd for authentication
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}]
# define default sub-domain for Master node
openshift_master_default_subdomain=apps.schmaustech.com
# allow unencrypted connection within cluster
openshift_docker_insecure_registries=172.30.0.0/16
[masters]
ocp-master.schmaustech.com openshift_schedulable=true containerized=false
[etcd]
ocp-master.schmaustech.com
[nodes]
# defined values for [openshift_node_group_name] in the file below
# [/usr/share/ansible/openshift-ansible/roles/openshift_facts/defaults/main.yml]
ocp-master.schmaustech.com openshift_node_group_name='node-config-all-in-one'
ocp-node1.schmaustech.com openshift_node_group_name='node-config-compute'
ocp-node2.schmaustech.com openshift_node_group_name='node-config-compute'
ocp-node3.schmaustech.com openshift_node_group_name='node-config-compute'

With the Ansible host file in place we are ready to run the OpenShift prerequisite playbook:

[origin@ocp-master ~]$ ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml

Once the prerequisite playbook executes sucessfully we can then run the OpenShift deploy cluster playbook:

[origin@ocp-master ~]$ ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml

Lets validate OpenShift is up and running:

[origin@ocp-master ~]$ oc get nodes
NAME         STATUS    ROLES                  AGE       VERSION
ocp-master   Ready     compute,infra,master   15h       v1.11.0+d4cacc0
ocp-node1    Ready     compute                14h       v1.11.0+d4cacc0
ocp-node2    Ready     compute                14h       v1.11.0+d4cacc0
ocp-node3    Ready     compute                14h       v1.11.0+d4cacc0

[origin@ocp-master ~]$ oc get pods --all-namespaces -o wide
NAMESPACE                           NAME                                           READY     STATUS      RESTARTS   AGE       IP              NODE         NOMINATED NODE
default                             docker-registry-1-g4hgd                        1/1       Running     0          14h       10.128.0.4      ocp-master   <none>
default                             registry-console-1-zwhrd                       1/1       Running     0          14h       10.128.0.6      ocp-master   <none>
default                             router-1-v8pkp                                 1/1       Running     0          14h       192.168.3.100   ocp-master   <none>
kube-service-catalog                apiserver-gxjst                                1/1       Running     0          14h       10.128.0.17     ocp-master   <none>
kube-service-catalog                controller-manager-2v6qs                       1/1       Running     3          14h       10.128.0.18     ocp-master   <none>
openshift-ansible-service-broker    asb-1-d8clq                                    1/1       Running     0          14h       10.128.0.21     ocp-master   <none>
openshift-console                   console-566f847459-pk52j                       1/1       Running     0          14h       10.128.0.12     ocp-master   <none>
openshift-monitoring                alertmanager-main-0                            3/3       Running     0          14h       10.128.0.14     ocp-master   <none>
openshift-monitoring                alertmanager-main-1                            3/3       Running     0          14h       10.128.0.15     ocp-master   <none>
openshift-monitoring                alertmanager-main-2                            3/3       Running     0          14h       10.128.0.16     ocp-master   <none>
openshift-monitoring                cluster-monitoring-operator-79d6c544f5-c8rfs   1/1       Running     0          14h       10.128.0.7      ocp-master   <none>
openshift-monitoring                grafana-8497b48bd5-bqzxb                       2/2       Running     0          14h       10.128.0.10     ocp-master   <none>
openshift-monitoring                kube-state-metrics-7d8b57fc8f-ktdq4            3/3       Running     0          14h       10.128.0.19     ocp-master   <none>
openshift-monitoring                node-exporter-5gmbc                            2/2       Running     0          14h       192.168.3.103   ocp-node3    <none>
openshift-monitoring                node-exporter-fxthd                            2/2       Running     0          14h       192.168.3.102   ocp-node2    <none>
openshift-monitoring                node-exporter-gj27b                            2/2       Running     0          14h       192.168.3.101   ocp-node1    <none>
openshift-monitoring                node-exporter-r6vjs                            2/2       Running     0          14h       192.168.3.100   ocp-master   <none>
openshift-monitoring                prometheus-k8s-0                               4/4       Running     1          14h       10.128.0.11     ocp-master   <none>
openshift-monitoring                prometheus-k8s-1                               4/4       Running     1          14h       10.128.0.13     ocp-master   <none>
openshift-monitoring                prometheus-operator-5677fb6f87-4czth           1/1       Running     0          14h       10.128.0.8      ocp-master   <none>
openshift-node                      sync-7rqcb                                     1/1       Running     0          14h       192.168.3.103   ocp-node3    <none>
openshift-node                      sync-829ql                                     1/1       Running     0          14h       192.168.3.101   ocp-node1    <none>
openshift-node                      sync-mwq6v                                     1/1       Running     0          14h       192.168.3.102   ocp-node2    <none>
openshift-node                      sync-vc4hw                                     1/1       Running     0          15h       192.168.3.100   ocp-master   <none>
openshift-sdn                       ovs-n55b8                                      1/1       Running     0          14h       192.168.3.101   ocp-node1    <none>
openshift-sdn                       ovs-nvtgq                                      1/1       Running     0          14h       192.168.3.103   ocp-node3    <none>
openshift-sdn                       ovs-t8dgh                                      1/1       Running     0          14h       192.168.3.102   ocp-node2    <none>
openshift-sdn                       ovs-wgw2v                                      1/1       Running     0          15h       192.168.3.100   ocp-master   <none>
openshift-sdn                       sdn-7r9kn                                      1/1       Running     0          14h       192.168.3.101   ocp-node1    <none>
openshift-sdn                       sdn-89284                                      1/1       Running     0          15h       192.168.3.100   ocp-master   <none>
openshift-sdn                       sdn-hmgjg                                      1/1       Running     0          14h       192.168.3.103   ocp-node3    <none>
openshift-sdn                       sdn-n7lzh                                      1/1       Running     0          14h       192.168.3.102   ocp-node2    <none>
openshift-template-service-broker   apiserver-md5sr                                1/1       Running     0          14h       10.128.0.22     ocp-master   <none>
openshift-web-console               webconsole-674f79b6fc-cjrhw                    1/1       Running     0          14h       10.128.0.9      ocp-master   <none>

With OpenShift up and running we can move onto install Rook/Ceph cluster.  The first step is to clone the Rook Git repo down to the master node and make an adjustment for the kubelet-plugins.  Please note here I am cloning down a colleagues Rook clone and not direct from the Rook project:

[origin@ocp-master ~]$ git clone https://github.com/ksingh7/ocp4-rook.git
[origin@ocp-master ~]$ sed -i.bak s+/etc/kubernetes/kubelet-plugins/volume/exec+/usr/libexec/kubernetes/kubelet-plugins/volume/exec+g /home/origin/ocp4-rook/ceph/operator.yaml

With the repository cloned we can now apply the the security context constraints needed by the Rook pods using the scc.yaml and then launch the Rook operator with operator.yaml:

[origin@ocp-master ~]$ oc create -f /home/origin/ocp4-rook/ceph/scc.yaml
[origin@ocp-master ~]$ oc create -f /home/origin/ocp4-rook/ceph/operator.yaml

Lets validate the Rook operator came up:

[origin@ocp-master ~]$ oc get pods -n rook-ceph-system 
NAME                                 READY     STATUS    RESTARTS   AGE
rook-ceph-agent-77x5n                1/1       Running   0          1h
rook-ceph-agent-cdvqr                1/1       Running   0          1h
rook-ceph-agent-gz7tl                1/1       Running   0          1h
rook-ceph-agent-rsbwh                1/1       Running   0          1h
rook-ceph-operator-b76466dcd-zmscb   1/1       Running   0          1h
rook-discover-6p5ht                  1/1       Running   0          1h
rook-discover-fnrf4                  1/1       Running   0          1h
rook-discover-grr5w                  1/1       Running   0          1h
rook-discover-mllt7                  1/1       Running   0          1h

Once the operator is up we can proceed on deploying the Ceph cluster and once that is up deploy the Ceph toolbox pod:

[origin@ocp-master ~]$ oc create -f /home/origin/ocp4-rook/ceph/cluster.yaml  
[origin@ocp-master ~]$ oc create -f /home/origin/ocp4-rook/ceph/toolbox.yaml

Lets validate the Ceph cluster is up:

[origin@ocp-master ~]$ oc get pods -n rook-ceph
NAME                                     READY     STATUS      RESTARTS   AGE
rook-ceph-mgr-a-785ddd6d6c-d4w56         1/1       Running     0          1h
rook-ceph-mon-a-67855c796b-sdvqm         1/1       Running     0          1h
rook-ceph-mon-b-6d58cd7656-xkrdz         1/1       Running     0          1h
rook-ceph-mon-c-869b8d9d9-m7544          1/1       Running     0          1h
rook-ceph-osd-0-d6cbd5776-987p9          1/1       Running     0          1h
rook-ceph-osd-1-cfddf997-pzq69           1/1       Running     0          1h
rook-ceph-osd-2-79fc94c6d5-krtnj         1/1       Running     0          1h
rook-ceph-osd-3-f9b55c4d6-7jp7c          1/1       Running     0          1h
rook-ceph-osd-prepare-ocp-master-ztmhs   0/2       Completed   0          1h
rook-ceph-osd-prepare-ocp-node1-mgbcd    0/2       Completed   0          1h
rook-ceph-osd-prepare-ocp-node2-98rtw    0/2       Completed   0          1h
rook-ceph-osd-prepare-ocp-node3-ngscg    0/2       Completed   0          1h
rook-ceph-tools                          1/1       Running     0          1h

Lets also validate from the Ceph toolbox that the cluster health is ok:

[origin@ocp-master ~]$ oc -n rook-ceph rsh rook-ceph-tools
sh-4.2# ceph status
  cluster:
    id:     6ddab3e4-1730-412f-89b8-0738708adac8
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum b,a,c
    mgr: a(active)
    osd: 4 osds: 4 up, 4 in
 
  data:
    pools:   1 pools, 100 pgs
    objects: 281  objects, 1.1 GiB
    usage:   51 GiB used, 169 GiB / 220 GiB avail
    pgs:     100 active+clean

Now that we have confirmed the Ceph cluster is deployed lets configure a Ceph storage class and also make it the default storage class for the environment:

[origin@ocp-master ~]$ oc create -f /home/origin/ocp4-rook/ceph/storageclass.yaml
[origin@ocp-master ~]$ oc patch storageclass rook-ceph-block -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

And now if we display the storage class we can see Rook/Ceph is our default:

[origin@ocp-master ~]$ oc get storageclass
NAME                        PROVISIONER          AGE
rook-ceph-block (default)   ceph.rook.io/block   6h

Proceeding with our stack installation lets get CNV installed.  Again with the use of the Ansible playbook we used earlier for OpenShift this is a relatively easy task:

[origin@ocp-master ~]$ oc login -u system:admin
[origin@ocp-master ~]$ cd /usr/share/ansible/kubevirt-ansible
[origin@ocp-master ~]$ ansible-playbook -i /etc/ansible/hosts -e @vars/cnv.yml playbooks/kubevirt.yml -e apb_action=provision

Once the installation completes lets run the following command to ensure the pods for CNV are up:

[origin@ocp-master ~]$ oc get pods --all-namespaces -o wide|egrep "kubevirt|cdi"
cdi                                 cdi-apiserver-7bfd97d585-tqjgt                 1/1       Running     0          6h        10.129.0.11     ocp-node3    
cdi                                 cdi-deployment-6689fcb476-4klcj                1/1       Running     0          6h        10.131.0.12     ocp-node1    
cdi                                 cdi-operator-5889d7588c-wvgl4                  1/1       Running     0          6h        10.130.0.12     ocp-node2    
cdi                                 cdi-uploadproxy-79c9fb9f59-pkskw               1/1       Running     0          6h        10.129.0.13     ocp-node3    
cdi                                 virt-launcher-f29vm-h6mc9                      1/1       Running     0          6h        10.129.0.15     ocp-node3    
kubevirt-web-ui                     console-854d4585c8-hgdhv                       1/1       Running     0          6h        10.129.0.10     ocp-node3    
kubevirt-web-ui                     kubevirt-web-ui-operator-6b4574bb95-bmsw7      1/1       Running     0          6h        10.130.0.11     ocp-node2    
kubevirt                            kubevirt-cpu-node-labeller-fvx9n               1/1       Running     0          6h        10.128.0.29     ocp-master   
kubevirt                            kubevirt-cpu-node-labeller-jr858               1/1       Running     0          6h        10.131.0.13     ocp-node1    
kubevirt                            kubevirt-cpu-node-labeller-tgq5g               1/1       Running     0          6h        10.129.0.14     ocp-node3    
kubevirt                            kubevirt-cpu-node-labeller-xqpbl               1/1       Running     0          6h        10.130.0.13     ocp-node2    
kubevirt                            virt-api-865b95d544-hg58l                      1/1       Running     0          6h        10.129.0.8      ocp-node3    
kubevirt                            virt-api-865b95d544-jrkxh                      1/1       Running     0          6h        10.131.0.10     ocp-node1    
kubevirt                            virt-controller-5c89d4978d-q79lh               1/1       Running     0          6h        10.130.0.8      ocp-node2    
kubevirt                            virt-controller-5c89d4978d-t58l7               1/1       Running     0          6h        10.130.0.10     ocp-node2    
kubevirt                            virt-handler-gblbk                             1/1       Running     0          6h        10.128.0.28     ocp-master   
kubevirt                            virt-handler-jnwx6                             1/1       Running     0          6h        10.130.0.9      ocp-node2    
kubevirt                            virt-handler-r94fb                             1/1       Running     0          6h        10.129.0.9      ocp-node3    
kubevirt                            virt-handler-z7775                             1/1       Running     0          6h        10.131.0.11     ocp-node1    
kubevirt                            virt-operator-68984b585c-265bq                 1/1       Running     0          6h        10.129.0.7      ocp-node3    

Now that CNV is up running lets pull down a Fedora 29 image and upload it into a PVC of the default storageclass which of course is Rook/Ceph:

[origin@ocp-master ~]$ curl -L -o /home/origin/f29.qcow2 http://ftp.usf.edu/pub/fedora/linux/releases/29/Cloud/x86_64/images/Fedora-Cloud-Base-29-1.2.x86_64.qcow2
[origin@ocp-master ~]$ virtctl image-upload --pvc-name=f29vm --pvc-size=5Gi --image-path=/home/origin/f29.qcow2 --uploadproxy-url=https://`oc describe route cdi-uploadproxy-route|grep Endpoints|cut -f2` --insecure

We can execute the following to see that the PVC has been created:

[origin@ocp-master ~]$ oc get pvc
NAME      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
f29vm     Bound     pvc-4815df9e-4987-11e9-a732-525400767d62   5Gi        RWO            rook-ceph-block   6h

Besides the PVC we will also need a virtual machine configuration yaml file.  The one below is an example that will be used in this demonstration:

apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
  creationTimestamp: null
  labels:
    kubevirt-vm: f29vm
  name: f29vm
spec:
  running: true
  template:
    metadata:
      creationTimestamp: null
      labels:
        kubevirt.io/domain: f29vm
    spec:
      domain:
        cpu:
          cores: 2
        devices:
          disks:
          - disk:
              bus: virtio
            name: osdisk
            volumeName: osdisk
          - disk:
              bus: virtio
            name: cloudinitdisk
            volumeName: cloudinitvolume
          interfaces:
          - name: default
            bridge: {}
        resources:
          requests:
            memory: 1024M
      terminationGracePeriodSeconds: 0
      volumes:
      - name: osdisk
        persistentVolumeClaim:
          claimName: f29vm
      - name: cloudinitdisk
        cloudInitNoCloud:
          userData: |-
            #cloud-config
            password: ${PASSWORD}
            disable_root: false
            chpasswd: { expire: False }
            ssh_authorized_keys:
            - "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUs1KbLraX74mBM/ksoGwbsEejfpCVeMzbW7JLJjGXF8G1jyVAE3T0Uf5mO8nbNOfkjAjw24lxSsEScF2wslBzA5MIm+GB6Z+ZzR55FcRlZeouGVrfLmb67mYc2c/F/mq35TruHdRk2G5Y0+6cf8cfDs414+yiVA0heHQvWNfO7kb1z9kIOhyD6OOwdNT5jK/1O0+p6SdP+pEal51BsEf6GRGYLWc9SLIEcqtjoprnundr5UPvmC1l/pkqFQigMehwhthrdXC4GseWiyj9CnBkccxQCKvHjzko/wqsWGQLwDG3pBsHhthvbY0G5+VPB9a8YV58WJhC6nHpUTDA8jpB origin@ocp-master"
      networks:
      - name: default
        pod: {}

At this point we have all the necessary components to launch our containerized virtual machine instance.   The following command does the creation using the yaml file we created in the previous step:

[origin@ocp-master ~]$ oc create -f /home/origin/f29vm.yaml

There are multiple ways to validate the virtual machine has been instantiated.   I like to do the following to confirm the instance is running and has an IP address:

[origin@ocp-master ~]$ oc get vms
NAME      AGE       RUNNING   VOLUME
f29vm     6h        true      
[origin@ocp-master ~]$ oc get vmi
NAME      AGE       PHASE     IP            NODENAME
f29vm     6h        Running   10.129.0.15   ocp-node3

One final step you can do is actually log into the instance assuming a key was set in the yaml file:

[origin@ocp-master ~]$ ssh -i /home/origin/.ssh/id_rsa -o "StrictHostKeyChecking=no" fedora@10.129.0.15
[fedora@f29vm ~]$ cat /etc/fedora-release
Fedora release 29 (Twenty Nine)

Hopefully this demonstrated how easy it is to get OpenShift, Rook and CNV up and running and how one can then leverage the storage of Rook to provide a backend for the virtual instance that gets spun up in CNV.   What is awesome is that I have taken the steps above and put them into a DCI job where I can automatically rerun the deployment using newer version of the code base for testing.   If you are not familiar with DCI I will leave with this tease link to DCI: https://doc.distributed-ci.io/