Slimming Down RHEL 8 Image

I recently had a project that required the Red Hat Enterprise Linux 8 Binary DVD due to the package requirements.  This normally would not be an issue but there were further requirements that made this a bit more challenging.  Those requirements were that the image also needed to be under 4gb in size and could not be delivered via networking.   This sets up the challenge of trying to devise a way to take the DVD image which is over 6gb and par it down leaving only what is required behind.  The following will describe the process the I used to meets these requirements and while there might be other methods this was the procedure I used.

The first thing we need to do is download the RHEL 8 Binary DVD image to a running system.  Once on the system I created a directory called /rhel, mounted up the image and copied the contents to /rhel:

# mount ./rhel-8.0-x86_64-dvd.iso /mnt/ -o loop
# mkdir /rhel
# shopt -s dotglob
# cp -ai /mnt/* /rhel
# umount /mnt

The next step was to create a list of packages that I needed from an existing RHEL 8 system that was installed for the purpose of the project.   I just used a simple rpm command but formatted it just for the package name without the version.   The reason being is we just want the package name as versions might be different from image and current system due to applied errata on running system:

# rpm -qa --queryformat='%{NAME}\n' | sort -n > rhel8.lst

Now I need to get a list of packages from the RHEL 8 image and in this version there are two directories that hold packages: Appstream and BaseOS.  In my example I only needed to parse out the Appstream packages to gain enough image reduction.  So I will run the following command to grab the list:

# ls -1 /rhel/AppStream/Packages/*|xargs -n 1 basename > appstream.lst

Now that I have my two lists I needed to create some logic to be able to remove packages that I would not need and this required a comparison of the two lists.  The following script should be run from with in the /rhel directory and on successful comparison it will keep the matched packages in /rhel/AppStream/Packages.  If there is no match the package will be removed.

#!/bin/bash
while read package; do
    flag="0"
    while read rhel8; do
        if [[ "$package" =~ ^"$rhel8" ]]
        then
            flag="1"
            echo KEEP: $package $rhel8 $flag
            break
        fi
    done < rhel8.lst
    if [[ "$flag" = "0" ]]
    then
        echo REMOVE $package $rhel8 $flag
        find ./ -name $package -print -exec rm -r -f {} \;
    fi
done < appstream.lst

Now that we have removed the excessive packages from the /rhel directory structure, which if you recall is just the contents of the RHEL 8 image, we can now begin to make a new iso image.  But first since we changed the AppStream packaging we need to update the repository data.  We do this by using createrepo command:

# cd /rhel/AppStream
# createrepo  -g repodata/*comps*.xml . --update
Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete

At this point we can go ahead and create a new image using the following command inside /rhel:

# mkisofs -o /tmp/rhel8-slim.iso -b isolinux/isolinux.bin -c isolinux/boot.cat --no-emul-boot --boot-load-size 4 --boot-info-table -J -R -V RHEL-8-0-0-BaseOS-x86_64 .

Once the command runs the new image will be written out to /tmp/rhel8-slim.iso and if we are lucky the size is now greatly diminished:

# ls -lh /tmp/rhel8-slim.iso
-rw-r--r--. 1 root root 2.0G Jul  1 15:17 /tmp/rhel8-slim.iso

And there it is a slimmer 2gb RHEL 8 image that contains only the packages that are needed for this specific project.   Be aware this method could also be used to add custom packages to RHEL 8 as well or even script customization and/or configurations.  The flexibility is limited only to ones imagination.

Centralized vBMC Controller

In my lab I use KVM virtual machines as my "baremetal" machines for testing OpenStack and Openshift.  In both of those cases I need something that provides power management to power off/on the virtual machines during deployment phases.   This is where Virtual BMC (vBMC) comes in as a handy tool to provide that functionality.   However I really don't want to install vBMC on all of the physical hosts that were providing my virtual machines.   Thankfully as this blog will explain there is a way to run vBMC where you can centrally manage all the virtual machines.

First lets pick a host that will be our centralized vBMC controller.   This host could be a physical box or a virtual machine it does not matter.  It does however need to have SSH key authentication to any of the KVM hypervisor hosts that contain virtual machines we wish to control with vBMC.

Once I have my vBMC host I will install the required package via rpm since I did not have a repo that contained the package.  If you have a repo that does container the package I would suggest using yum install instead:

# rpm -ivh python2-virtualbmc-1.4.0-1.el7.noarch.rpm 
Preparing...                          ################################# [100%]
Updating / installing...
   1:python2-virtualbmc-1.4.0-1.el7   ################################# [100%]

Once the package is installed we should be able to run the following command to see the command line usage for vbmc when adding a host.  If you get errors about cliff.app and zmq please install these packages (python2-cliff.noarch & python2-zmq.x86_64):

# vbmc add --help
usage: vbmc add [-h] [--username USERNAME] [--password PASSWORD] [--port PORT]
                [--address ADDRESS] [--libvirt-uri LIBVIRT_URI]
                [--libvirt-sasl-username LIBVIRT_SASL_USERNAME]
                [--libvirt-sasl-password LIBVIRT_SASL_PASSWORD]
                domain_name

Create a new BMC for a virtual machine instance

positional arguments:
  domain_name           The name of the virtual machine

optional arguments:
  -h, --help            show this help message and exit
  --username USERNAME   The BMC username; defaults to "admin"
  --password PASSWORD   The BMC password; defaults to "password"
  --port PORT           Port to listen on; defaults to 623
  --address ADDRESS     The address to bind to (IPv4 and IPv6 are supported);
                        defaults to ::
  --libvirt-uri LIBVIRT_URI
                        The libvirt URI; defaults to "qemu:///system"
  --libvirt-sasl-username LIBVIRT_SASL_USERNAME
                        The libvirt SASL username; defaults to None
  --libvirt-sasl-password LIBVIRT_SASL_PASSWORD
                        The libvirt SASL password; defaults to None

Now lets try adding a virtual machine called kube-master located on a remote hypervisor host:

# vbmc add --username admin --password password --port 6230 --address 192.168.0.10 --libvirt-uri qemu+ssh://root@192.168.0.4/system kube-master

Now lets add a second virtual machine on a different hypervisor and notice I increment the port number in use as this is the unique port number that gets called when using ipmi to actually connection to the specific host we wish to power on/off or get a power status from:

# vbmc add --username admin --password password --port 6231 --address 192.168.0.10 --libvirt-uri qemu+ssh://root@192.168.0.5/system cube-vm1

Now lets start the vbmc process for them and confirm they are up and running:

# vbmc start kube-master
2019-06-17 08:48:05,649.649 6915 INFO VirtualBMC [-] Started vBMC instance for domain kube-master

# vbmc start cube-vm1
2019-06-17 14:49:39,491.491 6915 INFO VirtualBMC [-] Started vBMC instance for domain cube-vm1

# vbmc list

+-------------+---------+--------------+------+
| Domain name | Status  | Address      | Port |
+-------------+---------+--------------+------+
| cube-vm1    | running | 192.168.0.10 | 6231 |
| kube-master | running | 192.168.0.10 | 6230 |
+-------------+---------+--------------+------+

Now that we have added a few virtual machines lets validate that things are working by trying to power the hosts up and get a status. In this example we will check the power status of kube-master and power on if it is off:

# ipmitool -I lanplus -H192.168.0.10 -p6230 -Uadmin -Ppassword chassis status
System Power         : off
Power Overload       : false
Power Interlock      : inactive
Main Power Fault     : false
Power Control Fault  : false
Power Restore Policy : always-off
Last Power Event     : 
Chassis Intrusion    : inactive
Front-Panel Lockout  : inactive
Drive Fault          : false
Cooling/Fan Fault    : false

# ipmitool -I lanplus -H192.168.0.10 -p6230 -Uadmin -Ppassword chassis power on
Chassis Power Control: Up/On

# ipmitool -I lanplus -H192.168.0.10 -p6230 -Uadmin -Ppassword chassis status
System Power         : on
Power Overload       : false
Power Interlock      : inactive
Main Power Fault     : false
Power Control Fault  : false
Power Restore Policy : always-off
Last Power Event     : 
Chassis Intrusion    : inactive
Front-Panel Lockout  : inactive
Drive Fault          : false
Cooling/Fan Fault    : false

In the next example we will see that cube-vm1 is powered on and we should power it off:

# ipmitool -I lanplus -H192.168.0.10 -p6231 -Uadmin -Ppassword chassis status
System Power         : on
Power Overload       : false
Power Interlock      : inactive
Main Power Fault     : false
Power Control Fault  : false
Power Restore Policy : always-off
Last Power Event     : 
Chassis Intrusion    : inactive
Front-Panel Lockout  : inactive
Drive Fault          : false
Cooling/Fan Fault    : false

# ipmitool -I lanplus -H192.168.0.10 -p6231 -Uadmin -Ppassword chassis power off
Chassis Power Control: Down/Off

# ipmitool -I lanplus -H192.168.0.10 -p6231 -Uadmin -Ppassword chassis status
System Power         : off
Power Overload       : false
Power Interlock      : inactive
Main Power Fault     : false
Power Control Fault  : false
Power Restore Policy : always-off
Last Power Event     : 
Chassis Intrusion    : inactive
Front-Panel Lockout  : inactive
Drive Fault          : false
Cooling/Fan Fault    : false

Lets summarize what we just did.  We had a vBMC host that was ipaddress 192.168.0.10 where we installed vBMC and configured two different virtual machines kube-master and cube-vm1 which were on two completely different hypervisor guests, ip address 192.168.0.4 and 192.168.0.5 respectively.  This allowed us to remotely power manage those virtual machines without the need to install any additional software on those hypervisor hosts.

Given this flexibility one could foresee maybe in the future have a centalized vBMC container that could then in turn access any KubeVirt deployed virtual machines that are deployed within that Kubernetes cluster.  I guess its only a matter of time.

Metal-3 Installer Dev Scripts & Macvtap

Recently I was working with the dev-scripts from the OpenShift Metal3 project located here on github.  I had been using CI automation to run test jobs which were working perfectly in my simulated virtual baremetal environment.

However last week my CI broke due to a code change.  Looking through the changes I noticed that they introduced a discovery mechanism that relied on multicast.   Under normal circumstances when using real baremetal and not virtual baremetal this issue would not have been rendered.  But in my environment it quickly became clear that multicast traffic was not being passed.

The problem is that in order to leverage PXE booting for my virtual baremetal nodes I needed to ensure that I had a network interface that was attached to a native vlan on my physical interface since PXE traffic cannot be tagged.  The solution for this issue was to use macvtap for my virtual baremetal machines.   But as I quickly learned macvtap by default does not pass multicast.

I determined this by using tcpdump on my bootstrap node and sure enough I did not see any multicast packets when the master nodes were going through the ironic-introspection:

$ sudo tcpdump -n -i any port 5353 | grep 172.22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes

As a quick test to validate my thinking I went ahead and applied the following on the macvtap interface where the virtual bootstrap node runs:

# ip a|grep macvtap0
macvtap0@eno1: BROADCAST,MULTICAST,UP,LOWER_UP mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 500

# ip link set dev macvtap0 allmulticast on

# ip a|grep macvtap0
macvtap0@eno1: BROADCAST,MULTICAST,ALLMULTI,UP,LOWER_UP mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 500

After setting the allmulticast on the macvtap device I went back to tcpdump again and found now my device was passing the multicast traffic I needed for host discovery:

$ sudo tcpdump -n -i any port 5353 | grep 172.22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
07:03:26.186790 IP 172.22.0.55.mdns > 224.0.0.251.mdns: 0 [3q] SRV (QM)? baremetal-introspection._openstack._tcp.local. TXT (QM)? baremetal-introspection._openstack._tcp.local. A (QM)? baremetal-introspection._openstack._tcp.local. (75)
07:03:26.186807 IP 172.22.0.55.mdns > 224.0.0.251.mdns: 0 [3q] SRV (QM)? baremetal-introspection._openstack._tcp.local. TXT (QM)? baremetal-introspection._openstack._tcp.local. A (QM)? baremetal-introspection._openstack._tcp.local. (75)
07:03:26.186790 IP 172.22.0.55.mdns > 224.0.0.251.mdns: 0 [3q] SRV (QM)? baremetal-introspection._openstack._tcp.local. TXT (QM)? baremetal-introspection._openstack._tcp.local. A (QM)? baremetal-introspection._openstack._tcp.local. (75)
07:03:26.188884 IP 172.22.0.1.mdns > 224.0.0.251.mdns: 0*- [0q] 3/0/1 (Cache flush) SRV baremetal-introspection._openstack._tcp.local.:5050 0 0, (Cache flush) TXT "ipa_debug=1" "ipa_inspection_dhcp_all_interfaces=1" "protocol=http" "ipa_collect_lldp=1", (Cache flush) A 172.22.0.1 (203)
07:03:26.188888 IP 172.22.0.1.mdns > 224.0.0.251.mdns: 0*- [0q] 3/0/1 (Cache flush) SRV baremetal-introspection._openstack._tcp.local.:5050 0 0, (Cache flush) TXT "ipa_debug=1" "ipa_inspection_dhcp_all_interfaces=1" "protocol=http" "ipa_collect_lldp=1", (Cache flush) A 172.22.0.1 (203)
07:03:26.188894 IP 172.22.0.1.mdns > 224.0.0.251.mdns: 0*- [0q] 3/0/1 (Cache flush) SRV baremetal-introspection._openstack._tcp.local.:5050 0 0, (Cache flush) TXT "ipa_debug=1" "ipa_inspection_dhcp_all_interfaces=1" "protocol=http" "ipa_collect_lldp=1", (Cache flush) A 172.22.0.1 (203)
07:03:30.659938 IP 172.22.0.58.mdns > 224.0.0.251.mdns: 0 [3q] SRV (QM)? baremetal._openstack._tcp.local. TXT (QM)? baremetal._openstack._tcp.local. A (QM)? baremetal._openstack._tcp.local. (61)
07:03:30.659951 IP 172.22.0.58.mdns > 224.0.0.251.mdns: 0 [3q] SRV (QM)? baremetal._openstack._tcp.local. TXT (QM)? baremetal._openstack._tcp.local. A (QM)? baremetal._openstack._tcp.local. (61)
07:03:30.659938 IP 172.22.0.58.mdns > 224.0.0.251.mdns: 0 [3q] SRV (QM)? baremetal._openstack._tcp.local. TXT (QM)? baremetal._openstack._tcp.local. A (QM)? baremetal._openstack._tcp.local. (61)
07:03:30.660553 IP 172.22.0.1.mdns > 224.0.0.251.mdns: 0*- [0q] 3/0/1 (Cache flush) SRV baremetal._openstack._tcp.local.:6385 0 0, (Cache flush) TXT "ipa_debug=true" "protocol=http", (Cache flush) A 172.22.0.1 (136)
07:03:30.660556 IP 172.22.0.1.mdns > 224.0.0.251.mdns: 0*- [0q] 3/0/1 (Cache flush) SRV baremetal._openstack._tcp.local.:6385 0 0, (Cache flush) TXT "ipa_debug=true" "protocol=http", (Cache flush) A 172.22.0.1 (136)
07:03:30.660561 IP 172.22.0.1.mdns > 224.0.0.251.mdns: 0*- [0q] 3/0/1 (Cache flush) SRV baremetal._openstack._tcp.local.:6385 0 0, (Cache flush) TXT "ipa_debug=true" "protocol=http", (Cache flush) A 172.22.0.1 (136)
07:03:33.976735 IP 172.22.0.78.mdns > 224.0.0.251.mdns: 0 [3q] SRV (QM)? baremetal._openstack._tcp.local. TXT (QM)? baremetal._openstack._tcp.local. A (QM)? baremetal._openstack._tcp.local. (61)
07:03:33.976749 IP 172.22.0.78.mdns > 224.0.0.251.mdns: 0 [3q] SRV (QM)? baremetal._openstack._tcp.local. TXT (QM)? baremetal._openstack._tcp.local. A (QM)? baremetal._openstack._tcp.local. (61)
07:03:33.976735 IP 172.22.0.78.mdns > 224.0.0.251.mdns: 0 [3q] SRV (QM)? baremetal._openstack._tcp.local. TXT (QM)? baremetal._openstack._tcp.local. A (QM)? baremetal._openstack._tcp.local. (61)
07:03:33.978619 IP 172.22.0.1.mdns > 224.0.0.251.mdns: 0*- [0q] 3/0/1 (Cache flush) SRV baremetal._openstack._tcp.local.:6385 0 0, (Cache flush) TXT "ipa_debug=true" "protocol=http", (Cache flush) A 172.22.0.1 (136)
07:03:33.978622 IP 172.22.0.1.mdns > 224.0.0.251.mdns: 0*- [0q] 3/0/1 (Cache flush) SRV baremetal._openstack._tcp.local.:6385 0 0, (Cache flush) TXT "ipa_debug=true" "protocol=http", (Cache flush) A 172.22.0.1 (136)
07:03:33.978632 IP 172.22.0.1.mdns > 224.0.0.251.mdns: 0*- [0q] 3/0/1 (Cache flush) SRV baremetal._openstack._tcp.local.:6385 0 0, (Cache flush) TXT "ipa_debug=true" "protocol=http", (Cache flush) A 172.22.0.1 (136)
07:03:36.077289 IP 172.22.0.58.mdns > 224.0.0.251.mdns: 0 [3q] SRV (QM)? baremetal-introspection._openstack._tcp.local. TXT (QM)? baremetal-introspection._openstack._tcp.local. A (QM)? baremetal-introspection._openstack._tcp.local. (75)
07:03:36.077294 IP 172.22.0.58.mdns > 224.0.0.251.mdns: 0 [3q] SRV (QM)? baremetal-introspection._openstack._tcp.local. TXT (QM)? baremetal-introspection._openstack._tcp.local. A (QM)? baremetal-introspection._openstack._tcp.local. (75)
07:03:36.077289 IP 172.22.0.58.mdns > 224.0.0.251.mdns: 0 [3q] SRV (QM)? baremetal-introspection._openstack._tcp.local. TXT (QM)? baremetal-introspection._openstack._tcp.local. A (QM)? baremetal-introspection._openstack._tcp.local. (75)
07:03:36.077895 IP 172.22.0.1.mdns > 224.0.0.251.mdns: 0*- [0q] 3/0/1 (Cache flush) SRV baremetal-introspection._openstack._tcp.local.:5050 0 0, (Cache flush) TXT "ipa_debug=1" "ipa_inspection_dhcp_all_interfaces=1" "protocol=http" "ipa_collect_lldp=1", (Cache flush) A 172.22.0.1 (203)
07:03:36.077897 IP 172.22.0.1.mdns > 224.0.0.251.mdns: 0*- [0q] 3/0/1 (Cache flush) SRV baremetal-introspection._openstack._tcp.local.:5050 0 0, (Cache flush) TXT "ipa_debug=1" "ipa_inspection_dhcp_all_interfaces=1" "protocol=http" "ipa_collect_lldp=1", (Cache flush) A 172.22.0.1 (203)
07:03:36.077900 IP 172.22.0.1.mdns > 224.0.0.251.mdns: 0*- [0q] 3/0/1 (Cache flush) SRV baremetal-introspection._openstack._tcp.local.:5050 0 0, (Cache flush) TXT "ipa_debug=1" "ipa_inspection_dhcp_all_interfaces=1" "protocol=http" "ipa_collect_lldp=1", (Cache flush) A 172.22.0.1 (203)
07:03:39.395298 IP 172.22.0.78.mdns > 224.0.0.251.mdns: 0 [3q] SRV (QM)? baremetal-introspection._openstack._tcp.local. TXT (QM)? baremetal-introspection._openstack._tcp.local. A (QM)? baremetal-introspection._openstack._tcp.local. (75)
07:03:39.395305 IP 172.22.0.78.mdns > 224.0.0.251.mdns: 0 [3q] SRV (QM)? baremetal-introspection._openstack._tcp.local. TXT (QM)? baremetal-introspection._openstack._tcp.local. A (QM)? baremetal-introspection._openstack._tcp.local. (75)
07:03:39.395298 IP 172.22.0.78.mdns > 224.0.0.251.mdns: 0 [3q] SRV (QM)? baremetal-introspection._openstack._tcp.local. TXT (QM)? baremetal-introspection._openstack._tcp.local. A (QM)? baremetal-introspection._openstack._tcp.local. (75)
07:03:39.396947 IP 172.22.0.1.mdns > 224.0.0.251.mdns: 0*- [0q] 3/0/1 (Cache flush) SRV baremetal-introspection._openstack._tcp.local.:5050 0 0, (Cache flush) TXT "ipa_debug=1" "ipa_inspection_dhcp_all_interfaces=1" "protocol=http" "ipa_collect_lldp=1", (Cache flush) A 172.22.0.1 (203)
07:03:39.396951 IP 172.22.0.1.mdns > 224.0.0.251.mdns: 0*- [0q] 3/0/1 (Cache flush) SRV baremetal-introspection._openstack._tcp.local.:5050 0 0, (Cache flush) TXT "ipa_debug=1" "ipa_inspection_dhcp_all_interfaces=1" "protocol=http" "ipa_collect_lldp=1", (Cache flush) A 172.22.0.1 (203)
07:03:39.396956 IP 172.22.0.1.mdns > 224.0.0.251.mdns: 0*- [0q] 3/0/1 (Cache flush) SRV baremetal-introspection._openstack._tcp.local.:5050 0 0, (Cache flush) TXT "ipa_debug=1" "ipa_inspection_dhcp_all_interfaces=1" "protocol=http" "ipa_collect_lldp=1", (Cache flush) A 172.22.0.1 (203)

Further my deployment completed successfully given that multicast was being passed.  Despite the success though this is not the end of the story.  The reason is that I needed this change permanent and any reboot of the bootstrap node would cause the macvtap state to go back to the default of multicast disabled.

The solution was to ensure the following is set on the device in the bootstrap nodes kvm XML file configuration:

interface type='direct' trustGuestRxFilters='yes'

Hopefully this helps in any situation when macvtap is being used and multicast traffic is required to pass over the interface.

Deploying CSI Ceph RBD Driver on Kubernetes

The Container Storage Interface (CSI) is a standard for exposing arbitrary block and file storage storage systems to containerized workloads on Container Orchestration Systems (COs) like Kubernetes. Using CSI third-party storage providers can write and deploy plugins exposing storage systems in Kubernetes without ever having to touch the core Kubernetes code.

Ceph CSI plugins are one example that implement an interface between CSI enabled Container Orchestrator (CO) and CEPH cluster. It allows dynamically provisioning CEPH volumes and attaching them to workloads. Current implementation of Ceph CSI plugins was tested in Kubernetes environment (requires Kubernetes 1.13+), but the code does not rely on any Kubernetes specific calls and should be able to run with any CSI enabled CO.

Below is simple demonstration on how to enable Ceph RBD CSI drivers on a Kubernetes cluster.  However before we begin lets ensure that we have the following requirements already in place:

• Kubernetes cluster v1.13+
• allow-privileged flag enabled for both kubelet and API server
• A Rook Ceph deployed cluster

Before we start lets confirm we have a Rook Ceph cluster running in our environment:

# kubectl get pods -n rook-ceph
NAME                                      READY   STATUS      RESTARTS   AGE
rook-ceph-mgr-a-5dbb44d7f8-78mmc          1/1     Running     2          18h
rook-ceph-mon-a-64c8d5644-qpjtf           1/1     Running     0          46h
rook-ceph-mon-b-5678cb65c7-gzcc8          1/1     Running     0          18h
rook-ceph-mon-c-799f887c56-b9fxg          1/1     Running     0          78m
rook-ceph-osd-0-5ff6f7bb5c-bc5rp          1/1     Running     0          46h
rook-ceph-osd-1-5f7c4bb454-ngsfq          1/1     Running     0          18h
rook-ceph-osd-2-7885996ffc-wnjsw          1/1     Running     0          78m
rook-ceph-osd-prepare-kube-master-grrdp   0/2     Completed   0          38m
rook-ceph-osd-prepare-kube-node1-vgdwl    0/2     Completed   0          38m
rook-ceph-osd-prepare-kube-node2-f2gq9    0/2     Completed   0          38m

First lets clone the Ceph CSI repo and change directories into which we will work from:

# git clone https://github.com/ceph/ceph-csi.git
Cloning into 'ceph-csi'...
remote: Enumerating objects: 14, done.
remote: Counting objects: 100% (14/14), done.
remote: Compressing objects: 100% (14/14), done.
remote: Total 50633 (delta 3), reused 4 (delta 0), pack-reused 50619
Receiving objects: 100% (50633/50633), 68.56 MiB | 9.67 MiB/s, done.
Resolving deltas: 100% (27537/27537), done.
# cd ceph-csi/deploy/rbd/kubernetes/

Next lets create the CSI attacher role:

# kubectl create -f csi-attacher-rbac.yaml
serviceaccount/rbd-csi-attacher created
clusterrole.rbac.authorization.k8s.io/rbd-external-attacher-runner created
clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-attacher-role created

Next we will create the CSI RBD attacher plugin:

# kubectl create -f csi-rbdplugin-attacher.yaml
service/csi-rbdplugin-attacher created
statefulset.apps/csi-rbdplugin-attacher created

Follow that up with creating the CSI RBD provisioner plugin:

# kubectl create -f csi-rbdplugin-provisioner.yaml
service/csi-rbdplugin-provisioner created
statefulset.apps/csi-rbdplugin-provisioner created

And finally we will create the CSI daemonset for the RBD plugin:

# kubectl create -f csi-rbdplugin.yaml
daemonset.apps/csi-rbdplugin created

At this point we will need to apply a few more role based access permissions for both the provisioner and attacher:

# kubectl apply -f csi-nodeplugin-rbac.yaml
serviceaccount/rbd-csi-nodeplugin created
clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created
clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created

# kubectl apply -f csi-provisioner-rbac.yaml
serviceaccount/rbd-csi-provisioner created
clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created
role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created
rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created

Now lets confirm are resources are up and operational:

# kubectl get po
NAME                          READY   STATUS    RESTARTS   AGE
csi-rbdplugin-6xlml           2/2     Running   0          36s
csi-rbdplugin-attacher-0      1/1     Running   2          5m56s
csi-rbdplugin-n98ms           2/2     Running   0          36s
csi-rbdplugin-ngrtv           2/2     Running   0          36s
csi-rbdplugin-provisioner-0   3/3     Running   0          23s

If everything looks good from the previous command lets change into the examples working directory and attempt to get the storageclass working against Ceph.  However we will need to gather a few details to ensure it works properly.

cd ceph-csi/examples/rbd

The storage class will require us to know the IP addresses of the Ceph Mons, which RBD pool we will use and of course a Ceph auth key.   I am going to use the Ceph toolbox to get that information.

# kubectl exec -it rook-ceph-tools -n rook-ceph /bin/bash

[root@rook-ceph-tools /]# ceph mon stat
e3: 3 mons at {a=10.0.0.81:6790/0,b=10.0.0.82:6790/0,c=10.0.0.83:6790/0}, election epoch 26, leader 0 a, quorum 0,1,2 a,b,c

[root@rook-ceph-tools /]# ceph osd lspools
1 rbd

[root@rook-ceph-tools /]# ceph auth get-key client.admin|base64
QVFDTDliVmNEb21IRHhBQUxXNGhmRkczTFNtcXM0ZW5VaXlTZEE9PQ==

We can take the MON addresses and client admin key and populate that in our secret.yaml file:

---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: default
data:
  admin: QVFDTDliVmNEb21IRHhBQUxXNGhmRkczTFNtcXM0ZW5VaXlTZEE9PQ==

We can also add the MON addresses and pool name to the storageclass.yaml:

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd
provisioner: rbd.csi.ceph.com
parameters:
   monitors: 10.0.0.81:6790,10.0.0.82:6790,10.0.0.83:6790
   pool: rbd
   imageFormat: "2"
   imageFeatures: layering
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: default
   csi.storage.k8s.io/node-publish-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-publish-secret-namespace: default
   adminid: admin
reclaimPolicy: Delete

Now that we have our files generated lets go ahead and issue the creation and validate:

# kubectl create -f secret.yaml 
secret/csi-rbd-secret created

# kubectl create -f storageclass.yaml 
storageclass.storage.k8s.io/csi-rbd created

# kubectl get storageclass
NAME      PROVISIONER        AGE
csi-rbd   rbd.csi.ceph.com   11s

Now that we have completed confiring the Ceph CSI driver and the storageclass for it lets try to provision some storage and attach it to a demo pod.  The first thing we need to do is create a block PVC so lets populate raw-block-pvc.yaml with the following:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: raw-block-pvc
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Block
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-rbd

Lets go ahead and create the PVC:

# kubectl create -f raw-block-pvc.yaml
persistentvolumeclaim/raw-block-pvc created

# kubectl get pvc
NAME            STATUS    VOLUME                                   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
raw-block-pvc   Bound     pvc-fd66b4d6-757d-11e9-8f9e-2a86e4085a59 1Gi        RWX            csi-rbd        3s

Now lets create an application to consume the PVC by first creating a template that references our PVC:

---
apiVersion: v1
kind: Pod
metadata:
  name: pod-with-raw-block-volume
spec:
  containers:
    - name: fc-container
      image: fedora:26
      command: ["/bin/sh", "-c"]
      args: ["tail -f /dev/null"]
      volumeDevices:
        - name: data
          devicePath: /dev/xvda
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: raw-block-pvc

Now that we have a template we can go ahead and create the application POD and if all goes well it will be up and running:

# kubectl create -f raw-block-pod.yaml
pod/pod-with-raw-block-volume created

# kubectl get pod pod-with-raw-block-volume
kubectl get pod fc-container
NAME                               READY   STATUS    RESTARTS   AGE
pod-with-raw-block-volume          1/1     Running   0          1m

Hopefully this provides an example of how to get the Ceph CSI drivers up and running in Kubernetes.

Deploy Rook/Ceph Cluster on Dedicated Networks

Recently a colleague of mine was trying to get Rook to deploy a Ceph cluster that used dedicated public and private networks to segment the Ceph replication traffic and the client access traffic to the OSDs of the cluster.   In a regular Ceph deployment this is rather trivial but when in the context of Kubernetes it becomes a little more complex given that Rook is deploying the cluster containers.  The following is procedure I applied to ensure my OSDs were listening on the appropriate networks.

Before we get into the steps on how to achieve this configuration lets quick take a look at the setup I used.  First I have a three node Kubernetes configuration (1 master with allowed scheduling and two workers):

# kubectl get nodes
NAME          STATUS   ROLES    AGE     VERSION
kube-master   Ready    master   2d22h   v1.14.0
kube-node1    Ready    worker   2d22h   v1.14.0
kube-node2    Ready    worker   2d22h   v1.14.0

On each of the nodes I have 3 network interfaces: eth0 on 10.0.0.0/24 (Kubernetes public), eth1 on 192.168.100.0/24 (Ceph private/cluster) & eth2 on 192.168.200.0/24 (Ceph public):

# ip a|grep eth[0-2]
2: eth0:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    inet 10.0.0.81/24 brd 10.0.0.255 scope global noprefixroute eth0
3: eth1:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    inet 192.168.100.81/24 brd 192.168.100.255 scope global noprefixroute eth1
4: eth2:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    inet 192.168.200.81/24 brd 192.168.200.255 scope global noprefixroute eth2

Before we begin lets see the current vanilla pods and namespaces on the Kubernetes cluster:

# kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                  READY   STATUS    RESTARTS   AGE   IP           NODE          NOMINATED NODE   READINESS GATES
kube-system   coredns-fb8b8dccf-h6wfn               1/1     Running   0          3d    10.244.1.2   kube-node2               
kube-system   coredns-fb8b8dccf-mv7p5               1/1     Running   0          3d    10.244.0.7   kube-master              
kube-system   etcd-kube-master                      1/1     Running   0          3d    10.0.0.81    kube-master              
kube-system   kube-apiserver-kube-master            1/1     Running   0          3d    10.0.0.81    kube-master              
kube-system   kube-controller-manager-kube-master   1/1     Running   1          3d    10.0.0.81    kube-master              
kube-system   kube-flannel-ds-amd64-szhg9           1/1     Running   0          3d    10.0.0.83    kube-node2               
kube-system   kube-flannel-ds-amd64-t4fxs           1/1     Running   0          3d    10.0.0.82    kube-node1               
kube-system   kube-flannel-ds-amd64-wbsdp           1/1     Running   0          3d    10.0.0.81    kube-master              
kube-system   kube-proxy-sn7j7                      1/1     Running   0          3d    10.0.0.83    kube-node2               
kube-system   kube-proxy-wtzm5                      1/1     Running   0          3d    10.0.0.81    kube-master              
kube-system   kube-proxy-xlwd9                      1/1     Running   0          3d    10.0.0.82    kube-node1               
kube-system   kube-scheduler-kube-master            1/1     Running   1          3d    10.0.0.81    kube-master              

# kubectl get ns
NAME              STATUS   AGE
default           Active   3d
kube-node-lease   Active   3d
kube-public       Active   3d
kube-system       Active   3d

Before can deploy the cluster we need to create a configmap for the rook-ceph namespace.  This namespace is normally created when the cluster is deployed however we want specific configuration items to be incorporated into the cluster upon deployment and so to do this we will create the rook-ceph namespace and apply a configmap that we create to that namespace.

First create a configmap file that looks like the following and notice I am referencing my Ceph cluster networks.  I will save this file with an arbitrary name like config-override.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: rook-config-override
  namespace: rook-ceph
data:
  config: |
    [global]
    public network =  192.168.200.0/24
    cluster network = 192.168.100.0/24
    public addr = ""
    cluster addr = ""

Next I will create the rook-ceph namespace:

# kubectl create namespace rook-ceph
namespace/rook-ceph created

# kubectl get ns
NAME              STATUS   AGE
default           Active   3d1h
kube-node-lease   Active   3d1h
kube-public       Active   3d1h
kube-system       Active   3d1h
rook-ceph         Active   5s

Now we can apply the configmap we created to the newly created namespace and validate its there:

# kubectl create -f config-override.yaml 
configmap/rook-config-override created

# kubectl get configmap -n rook-ceph
NAME                   DATA   AGE
rook-config-override   1      66s

# kubectl describe configmap -n rook-ceph
Name:         rook-config-override
Namespace:    rook-ceph
Labels:       <none>
Annotations:  <none>

Data
====
config:
----
[global]
public network =  192.168.200.0/24
cluster network = 192.168.100.0/24
public addr = ""
cluster addr = ""

Events:  <none>

Before we actually start to do the deploy we need to update one more thing in our Rook cluster.yaml.  Inside the cluster.yaml file we need to change hostNetwork from the default of false to true:

 sed -i 's/hostNetwork: false/hostNetwork: true/g' cluster.yaml

Now we can begin the process of deploying the Rook/Ceph cluster that includes launching the operator, cluster and toolbox.   I will place sleep statements in between each command to ensure the pods are up before I run the next command.  Also note there will be an error when creating the cluster about the rook-ceph namespace already existing and this is normal:

# kubectl create -f operator.yaml
namespace/rook-ceph-system created
customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/volumes.rook.io created
clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created
role.rbac.authorization.k8s.io/rook-ceph-system created
clusterrole.rbac.authorization.k8s.io/rook-ceph-global created
clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created
serviceaccount/rook-ceph-system created
rolebinding.rbac.authorization.k8s.io/rook-ceph-system created
clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created
deployment.apps/rook-ceph-operator created

# sleep 60

# kubectl create -f cluster.yaml 
serviceaccount/rook-ceph-osd created
serviceaccount/rook-ceph-mgr created
role.rbac.authorization.k8s.io/rook-ceph-osd created
role.rbac.authorization.k8s.io/rook-ceph-mgr-system created
role.rbac.authorization.k8s.io/rook-ceph-mgr created
rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created
rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created
rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created
rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created
rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created
cephcluster.ceph.rook.io/rook-ceph created
Error from server (AlreadyExists): error when creating "cluster.yaml": namespaces "rook-ceph" already exists

# sleep 60

# kubectl create -f toolbox.yaml 
pod/rook-ceph-tools created

Lets validate the Rook/Ceph operator, cluster and toolbox is up and running:

# kubectl get pods --all-namespaces -o wide
NAMESPACE          NAME                                      READY   STATUS      RESTARTS   AGE     IP           NODE          NOMINATED NODE   READINESS GATES
kube-system        coredns-fb8b8dccf-h6wfn                   1/1     Running     0          3d1h    10.244.1.2   kube-node2    <none>           <none>
kube-system        coredns-fb8b8dccf-mv7p5                   1/1     Running     0          3d1h    10.244.0.7   kube-master   <none>           <none>
kube-system        etcd-kube-master                          1/1     Running     0          3d1h    10.0.0.81    kube-master   <none>           <none>
kube-system        kube-apiserver-kube-master                1/1     Running     0          3d1h    10.0.0.81    kube-master   <none>           <none>
kube-system        kube-controller-manager-kube-master       1/1     Running     1          3d1h    10.0.0.81    kube-master   <none>           <none>
kube-system        kube-flannel-ds-amd64-szhg9               1/1     Running     0          3d1h    10.0.0.83    kube-node2    <none>           <none>
kube-system        kube-flannel-ds-amd64-t4fxs               1/1     Running     0          3d1h    10.0.0.82    kube-node1    <none>           <none>
kube-system        kube-flannel-ds-amd64-wbsdp               1/1     Running     0          3d1h    10.0.0.81    kube-master   <none>           <none>
kube-system        kube-proxy-sn7j7                          1/1     Running     0          3d1h    10.0.0.83    kube-node2    <none>           <none>
kube-system        kube-proxy-wtzm5                          1/1     Running     0          3d1h    10.0.0.81    kube-master   <none>           <none>
kube-system        kube-proxy-xlwd9                          1/1     Running     0          3d1h    10.0.0.82    kube-node1    <none>           <none>
kube-system        kube-scheduler-kube-master                1/1     Running     1          3d1h    10.0.0.81    kube-master   <none>           <none>
rook-ceph-system   rook-ceph-agent-55fqp                     1/1     Running     0          17m     10.0.0.83    kube-node2    <none>           <none>
rook-ceph-system   rook-ceph-agent-5v9v5                     1/1     Running     0          17m     10.0.0.81    kube-master   <none>           <none>
rook-ceph-system   rook-ceph-agent-spx29                     1/1     Running     0          17m     10.0.0.82    kube-node1    <none>           <none>
rook-ceph-system   rook-ceph-operator-57547fc866-ltp8z       1/1     Running     0          18m     10.244.2.4   kube-node1    <none>           <none>
rook-ceph-system   rook-discover-brxmt                       1/1     Running     0          17m     10.244.2.5   kube-node1    <none>           <none>
rook-ceph-system   rook-discover-hl748                       1/1     Running     0          17m     10.244.1.8   kube-node2    <none>           <none>
rook-ceph-system   rook-discover-qj5kd                       1/1     Running     0          17m     10.244.0.9   kube-master   <none>           <none>
rook-ceph          rook-ceph-mgr-a-5dbb44d7f8-vzs46          1/1     Running     0          16m     10.0.0.82    kube-node1    <none>           <none>
rook-ceph          rook-ceph-mon-a-5fb9568cb4-gvqln          1/1     Running     0          16m     10.0.0.81    kube-master   <none>           <none>
rook-ceph          rook-ceph-mon-b-b65c555bf-vz7ps           1/1     Running     0          16m     10.0.0.82    kube-node1    <none>           <none>
rook-ceph          rook-ceph-mon-c-69cf744c4d-8g4l6          1/1     Running     0          16m     10.0.0.83    kube-node2    <none>           <none>
rook-ceph          rook-ceph-osd-0-77499f547-d2vjx           1/1     Running     0          15m     10.0.0.81    kube-master   <none>           <none>
rook-ceph          rook-ceph-osd-1-698f76d786-lqn4w          1/1     Running     0          15m     10.0.0.82    kube-node1    <none>           <none>
rook-ceph          rook-ceph-osd-2-558c59d577-wfdlr          1/1     Running     0          15m     10.0.0.83    kube-node2    <none>           <none>
rook-ceph          rook-ceph-osd-prepare-kube-master-p55sw   0/2     Completed   0          15m     10.0.0.81    kube-master   <none>           <none>
rook-ceph          rook-ceph-osd-prepare-kube-node1-q7scn    0/2     Completed   0          15m     10.0.0.82    kube-node1    <none>           <none>
rook-ceph          rook-ceph-osd-prepare-kube-node2-8rm4d    0/2     Completed   0          15m     10.0.0.83    kube-node2    <none>           <none>
rook-ceph          rook-ceph-tools                           1/1     Running     0          3m24s   10.244.1.9   kube-node2    <none>           <none>

# kubectl -n rook-ceph exec -it rook-ceph-tools -- /bin/bash
bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory
[root@rook-ceph-tools /]# ceph status
  cluster:
    id:     b58f2a5c-2fc7-43e7-b410-2d541e78a90e
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum a,b,c
    mgr: a(active)
    osd: 3 osds: 3 up, 3 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0  objects, 0 B
    usage:   57 GiB used, 49 GiB / 105 GiB avail
    pgs:     
 
[root@rook-ceph-tools /]# exit
exit

At this point we have a fully operational cluster but is it really using the networks for OSD public and private traffic?   Lets explore that a bit further by first running the netstat command on any node in the cluster that has an OSD pod running.  Since my cluster is small I will show all 3 nodes below:

[root@kube-master]# netstat -tulpn | grep LISTEN | grep osd
tcp        0      0 192.168.100.81:6800     0.0.0.0:*               LISTEN      29719/ceph-osd      
tcp        0      0 192.168.200.81:6800     0.0.0.0:*               LISTEN      29719/ceph-osd      
tcp        0      0 192.168.200.81:6801     0.0.0.0:*               LISTEN      29719/ceph-osd      
tcp        0      0 192.168.100.81:6801     0.0.0.0:*               LISTEN      29719/ceph-osd
[root@kube-node1]# netstat -tulpn | grep LISTEN | grep osd
tcp        0      0 192.168.100.82:6800     0.0.0.0:*               LISTEN      18770/ceph-osd      
tcp        0      0 192.168.100.82:6801     0.0.0.0:*               LISTEN      18770/ceph-osd      
tcp        0      0 192.168.200.82:6801     0.0.0.0:*               LISTEN      18770/ceph-osd      
tcp        0      0 192.168.200.82:6802     0.0.0.0:*               LISTEN      18770/ceph-osd

[root@kube-node2]# netstat -tulpn | grep LISTEN | grep osd
tcp        0      0 192.168.100.83:6800     0.0.0.0:*               LISTEN      22659/ceph-osd      
tcp        0      0 192.168.200.83:6800     0.0.0.0:*               LISTEN      22659/ceph-osd      
tcp        0      0 192.168.200.83:6801     0.0.0.0:*               LISTEN      22659/ceph-osd      
tcp        0      0 192.168.100.83:6801     0.0.0.0:*               LISTEN      22659/ceph-osd

From the above we should see the OSD processes listening on the corresponding public and private networks we configured in the configmap.   However lets further confirm by going back into the toolbox and doing a ceph osd dump:

# kubectl -n rook-ceph exec -it rook-ceph-tools -- /bin/bash
bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory

[root@rook-ceph-tools]# ceph osd dump
epoch 14
fsid 05a8b767-e3e8-42aa-b792-69f479c807f7
created 2019-04-02 13:24:24.549423
modified 2019-04-02 13:25:28.441850
flags sortbitwise,recovery_deletes,purged_snapdirs
crush_version 7
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85
require_min_compat_client jewel
min_compat_client firefly
require_osd_release mimic
max_osd 3
osd.0 up   in  weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) 192.168.200.81:6800/29719 192.168.100.81:6800/29719 192.168.100.81:6801/29719 192.168.200.81:6801/29719 exists,up 2feb0edf-6652-4148-8264-6ba52d04ff80
osd.1 up   in  weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) 192.168.200.82:6801/18770 192.168.100.82:6800/18770 192.168.100.82:6801/18770 192.168.200.82:6802/18770 exists,up f8df61b4-4ac8-4705-9f97-eb09a1cc0d6c
osd.2 up   in  weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) 192.168.200.83:6800/22659 192.168.100.83:6800/22659 192.168.100.83:6801/22659 192.168.200.83:6801/22659 exists,up db555c80-9d81-4662-aed9-4bce1c0d5d78

As you can see it can be fairly straight forward to configure Rook to deploy a Ceph cluster using segmented networks to ensure the replication traffic runs on dedicated network and does not interfere with public client performance.  Hopefully this quick demonstrate showed that.

Stacking OpenShift with Rook and CNV

In previous blogs I was working with Rook/Ceph on Kubernetes and demonstrating how to setup a Ceph cluster and even replace failed OSDs. With that in mind I wanted to shift gears a bit and bring it more into alignment with OpenShift and Container Native Virtualization(CNV).

The following blog will guide us through a simple OpenShift deployment with Rook/Ceph and CNV configured. I will also demonstrate the use of a Rook PVC that provides the back end storage for a CNV deployed virtual instance.

The configuration for this lab is four virtual machines where one node is the master and compute and the other 3 nodes compute.  Each of these nodes has a base install of Red Hat Enterprise Linux 7 on it and the physical host they are on allows for nested virtualization.

Before we start with the installation of various software lets make sure we do a bit of user setup to ensure our install runs smoothly.  The next few steps will need to be done on all nodes to ensure a user origin (this could be any non root user) is created and has sudo rights without use of a password:

# useradd origin
# passwd origin
# echo -e 'Defaults:origin !requiretty\norigin ALL = (root) NOPASSWD:ALL' | tee /etc/sudoers.d/openshift 
# chmod 440 /etc/sudoers.d/openshift

Then we need to perform the the following steps to setup keyless authentication for the origin user from the master node to the rest of the nodes that will make up the cluster:

# ssh-keygen -q -N ""
# vi /home/origin/.ssh/config
Host ocp-master
    Hostname ocp-master.schmaustech.com
    User origin
Host ocp-node1
    Hostname ocp-node1.schmaustech.com
    User origin
Host ocp-node2
    Hostname ocp-node2.schmaustech.com
    User origin
Host ocp-node3
    Hostname ocp-node3.schmaustech.com
    User origin

# chmod 600 /home/origin/.ssh/config
# ssh-copy-id ocp-master
# ssh-copy-id ocp-node1
# ssh-copy-id ocp-node2
# ssh-copy-id ocp-node3

Now we can move on to enabling the necessary repositries on all nodes to ensure we can get access to the right packages we will need for installation:

[origin@ocp-master ~]$ sudo subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-extras-rpms --enable=rhel-7-server-rh-common-rpms --enable=rhel-7-server-ose-3.11-rpms --enable=rhel-7-server-ansible-2.6-rpms --enable=rhel-7-server-cnv-1.4-tech-preview-rpms

Next lets install the initial required packages on all the nodes:

[origin@ocp-master ~]$ sudo yum -y install openshift-ansible docker-1.13.1 kubevirt-ansible kubevirt-virtctl

On the master node lets configure the Ansible hosts file for our OpenShift installation.   The following is the example I used and I simply replaced /etc/ansible/hosts with it.

[OSEv3:children]
masters
nodes
etcd
[OSEv3:vars]
# admin user created in previous section
ansible_ssh_user=origin
ansible_become=true
oreg_url=registry.access.redhat.com/openshift3/ose-${component}:${version}
openshift_deployment_type=openshift-enterprise
#  use HTPasswd for authentication
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}]
# define default sub-domain for Master node
openshift_master_default_subdomain=apps.schmaustech.com
# allow unencrypted connection within cluster
openshift_docker_insecure_registries=172.30.0.0/16
[masters]
ocp-master.schmaustech.com openshift_schedulable=true containerized=false
[etcd]
ocp-master.schmaustech.com
[nodes]
# defined values for [openshift_node_group_name] in the file below
# [/usr/share/ansible/openshift-ansible/roles/openshift_facts/defaults/main.yml]
ocp-master.schmaustech.com openshift_node_group_name='node-config-all-in-one'
ocp-node1.schmaustech.com openshift_node_group_name='node-config-compute'
ocp-node2.schmaustech.com openshift_node_group_name='node-config-compute'
ocp-node3.schmaustech.com openshift_node_group_name='node-config-compute'

With the Ansible host file in place we are ready to run the OpenShift prerequisite playbook:

[origin@ocp-master ~]$ ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml

Once the prerequisite playbook executes sucessfully we can then run the OpenShift deploy cluster playbook:

[origin@ocp-master ~]$ ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml

Lets validate OpenShift is up and running:

[origin@ocp-master ~]$ oc get nodes
NAME         STATUS    ROLES                  AGE       VERSION
ocp-master   Ready     compute,infra,master   15h       v1.11.0+d4cacc0
ocp-node1    Ready     compute                14h       v1.11.0+d4cacc0
ocp-node2    Ready     compute                14h       v1.11.0+d4cacc0
ocp-node3    Ready     compute                14h       v1.11.0+d4cacc0

[origin@ocp-master ~]$ oc get pods --all-namespaces -o wide
NAMESPACE                           NAME                                           READY     STATUS      RESTARTS   AGE       IP              NODE         NOMINATED NODE
default                             docker-registry-1-g4hgd                        1/1       Running     0          14h       10.128.0.4      ocp-master   <none>
default                             registry-console-1-zwhrd                       1/1       Running     0          14h       10.128.0.6      ocp-master   <none>
default                             router-1-v8pkp                                 1/1       Running     0          14h       192.168.3.100   ocp-master   <none>
kube-service-catalog                apiserver-gxjst                                1/1       Running     0          14h       10.128.0.17     ocp-master   <none>
kube-service-catalog                controller-manager-2v6qs                       1/1       Running     3          14h       10.128.0.18     ocp-master   <none>
openshift-ansible-service-broker    asb-1-d8clq                                    1/1       Running     0          14h       10.128.0.21     ocp-master   <none>
openshift-console                   console-566f847459-pk52j                       1/1       Running     0          14h       10.128.0.12     ocp-master   <none>
openshift-monitoring                alertmanager-main-0                            3/3       Running     0          14h       10.128.0.14     ocp-master   <none>
openshift-monitoring                alertmanager-main-1                            3/3       Running     0          14h       10.128.0.15     ocp-master   <none>
openshift-monitoring                alertmanager-main-2                            3/3       Running     0          14h       10.128.0.16     ocp-master   <none>
openshift-monitoring                cluster-monitoring-operator-79d6c544f5-c8rfs   1/1       Running     0          14h       10.128.0.7      ocp-master   <none>
openshift-monitoring                grafana-8497b48bd5-bqzxb                       2/2       Running     0          14h       10.128.0.10     ocp-master   <none>
openshift-monitoring                kube-state-metrics-7d8b57fc8f-ktdq4            3/3       Running     0          14h       10.128.0.19     ocp-master   <none>
openshift-monitoring                node-exporter-5gmbc                            2/2       Running     0          14h       192.168.3.103   ocp-node3    <none>
openshift-monitoring                node-exporter-fxthd                            2/2       Running     0          14h       192.168.3.102   ocp-node2    <none>
openshift-monitoring                node-exporter-gj27b                            2/2       Running     0          14h       192.168.3.101   ocp-node1    <none>
openshift-monitoring                node-exporter-r6vjs                            2/2       Running     0          14h       192.168.3.100   ocp-master   <none>
openshift-monitoring                prometheus-k8s-0                               4/4       Running     1          14h       10.128.0.11     ocp-master   <none>
openshift-monitoring                prometheus-k8s-1                               4/4       Running     1          14h       10.128.0.13     ocp-master   <none>
openshift-monitoring                prometheus-operator-5677fb6f87-4czth           1/1       Running     0          14h       10.128.0.8      ocp-master   <none>
openshift-node                      sync-7rqcb                                     1/1       Running     0          14h       192.168.3.103   ocp-node3    <none>
openshift-node                      sync-829ql                                     1/1       Running     0          14h       192.168.3.101   ocp-node1    <none>
openshift-node                      sync-mwq6v                                     1/1       Running     0          14h       192.168.3.102   ocp-node2    <none>
openshift-node                      sync-vc4hw                                     1/1       Running     0          15h       192.168.3.100   ocp-master   <none>
openshift-sdn                       ovs-n55b8                                      1/1       Running     0          14h       192.168.3.101   ocp-node1    <none>
openshift-sdn                       ovs-nvtgq                                      1/1       Running     0          14h       192.168.3.103   ocp-node3    <none>
openshift-sdn                       ovs-t8dgh                                      1/1       Running     0          14h       192.168.3.102   ocp-node2    <none>
openshift-sdn                       ovs-wgw2v                                      1/1       Running     0          15h       192.168.3.100   ocp-master   <none>
openshift-sdn                       sdn-7r9kn                                      1/1       Running     0          14h       192.168.3.101   ocp-node1    <none>
openshift-sdn                       sdn-89284                                      1/1       Running     0          15h       192.168.3.100   ocp-master   <none>
openshift-sdn                       sdn-hmgjg                                      1/1       Running     0          14h       192.168.3.103   ocp-node3    <none>
openshift-sdn                       sdn-n7lzh                                      1/1       Running     0          14h       192.168.3.102   ocp-node2    <none>
openshift-template-service-broker   apiserver-md5sr                                1/1       Running     0          14h       10.128.0.22     ocp-master   <none>
openshift-web-console               webconsole-674f79b6fc-cjrhw                    1/1       Running     0          14h       10.128.0.9      ocp-master   <none>

With OpenShift up and running we can move onto install Rook/Ceph cluster.  The first step is to clone the Rook Git repo down to the master node and make an adjustment for the kubelet-plugins.  Please note here I am cloning down a colleagues Rook clone and not direct from the Rook project:

[origin@ocp-master ~]$ git clone https://github.com/ksingh7/ocp4-rook.git
[origin@ocp-master ~]$ sed -i.bak s+/etc/kubernetes/kubelet-plugins/volume/exec+/usr/libexec/kubernetes/kubelet-plugins/volume/exec+g /home/origin/ocp4-rook/ceph/operator.yaml

With the repository cloned we can now apply the the security context constraints needed by the Rook pods using the scc.yaml and then launch the Rook operator with operator.yaml:

[origin@ocp-master ~]$ oc create -f /home/origin/ocp4-rook/ceph/scc.yaml
[origin@ocp-master ~]$ oc create -f /home/origin/ocp4-rook/ceph/operator.yaml

Lets validate the Rook operator came up:

[origin@ocp-master ~]$ oc get pods -n rook-ceph-system 
NAME                                 READY     STATUS    RESTARTS   AGE
rook-ceph-agent-77x5n                1/1       Running   0          1h
rook-ceph-agent-cdvqr                1/1       Running   0          1h
rook-ceph-agent-gz7tl                1/1       Running   0          1h
rook-ceph-agent-rsbwh                1/1       Running   0          1h
rook-ceph-operator-b76466dcd-zmscb   1/1       Running   0          1h
rook-discover-6p5ht                  1/1       Running   0          1h
rook-discover-fnrf4                  1/1       Running   0          1h
rook-discover-grr5w                  1/1       Running   0          1h
rook-discover-mllt7                  1/1       Running   0          1h

Once the operator is up we can proceed on deploying the Ceph cluster and once that is up deploy the Ceph toolbox pod:

[origin@ocp-master ~]$ oc create -f /home/origin/ocp4-rook/ceph/cluster.yaml  
[origin@ocp-master ~]$ oc create -f /home/origin/ocp4-rook/ceph/toolbox.yaml

Lets validate the Ceph cluster is up:

[origin@ocp-master ~]$ oc get pods -n rook-ceph
NAME                                     READY     STATUS      RESTARTS   AGE
rook-ceph-mgr-a-785ddd6d6c-d4w56         1/1       Running     0          1h
rook-ceph-mon-a-67855c796b-sdvqm         1/1       Running     0          1h
rook-ceph-mon-b-6d58cd7656-xkrdz         1/1       Running     0          1h
rook-ceph-mon-c-869b8d9d9-m7544          1/1       Running     0          1h
rook-ceph-osd-0-d6cbd5776-987p9          1/1       Running     0          1h
rook-ceph-osd-1-cfddf997-pzq69           1/1       Running     0          1h
rook-ceph-osd-2-79fc94c6d5-krtnj         1/1       Running     0          1h
rook-ceph-osd-3-f9b55c4d6-7jp7c          1/1       Running     0          1h
rook-ceph-osd-prepare-ocp-master-ztmhs   0/2       Completed   0          1h
rook-ceph-osd-prepare-ocp-node1-mgbcd    0/2       Completed   0          1h
rook-ceph-osd-prepare-ocp-node2-98rtw    0/2       Completed   0          1h
rook-ceph-osd-prepare-ocp-node3-ngscg    0/2       Completed   0          1h
rook-ceph-tools                          1/1       Running     0          1h

Lets also validate from the Ceph toolbox that the cluster health is ok:

[origin@ocp-master ~]$ oc -n rook-ceph rsh rook-ceph-tools
sh-4.2# ceph status
  cluster:
    id:     6ddab3e4-1730-412f-89b8-0738708adac8
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum b,a,c
    mgr: a(active)
    osd: 4 osds: 4 up, 4 in
 
  data:
    pools:   1 pools, 100 pgs
    objects: 281  objects, 1.1 GiB
    usage:   51 GiB used, 169 GiB / 220 GiB avail
    pgs:     100 active+clean

Now that we have confirmed the Ceph cluster is deployed lets configure a Ceph storage class and also make it the default storage class for the environment:

[origin@ocp-master ~]$ oc create -f /home/origin/ocp4-rook/ceph/storageclass.yaml
[origin@ocp-master ~]$ oc patch storageclass rook-ceph-block -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

And now if we display the storage class we can see Rook/Ceph is our default:

[origin@ocp-master ~]$ oc get storageclass
NAME                        PROVISIONER          AGE
rook-ceph-block (default)   ceph.rook.io/block   6h

Proceeding with our stack installation lets get CNV installed.  Again with the use of the Ansible playbook we used earlier for OpenShift this is a relatively easy task:

[origin@ocp-master ~]$ oc login -u system:admin
[origin@ocp-master ~]$ cd /usr/share/ansible/kubevirt-ansible
[origin@ocp-master ~]$ ansible-playbook -i /etc/ansible/hosts -e @vars/cnv.yml playbooks/kubevirt.yml -e apb_action=provision

Once the installation completes lets run the following command to ensure the pods for CNV are up:

[origin@ocp-master ~]$ oc get pods --all-namespaces -o wide|egrep "kubevirt|cdi"
cdi                                 cdi-apiserver-7bfd97d585-tqjgt                 1/1       Running     0          6h        10.129.0.11     ocp-node3    
cdi                                 cdi-deployment-6689fcb476-4klcj                1/1       Running     0          6h        10.131.0.12     ocp-node1    
cdi                                 cdi-operator-5889d7588c-wvgl4                  1/1       Running     0          6h        10.130.0.12     ocp-node2    
cdi                                 cdi-uploadproxy-79c9fb9f59-pkskw               1/1       Running     0          6h        10.129.0.13     ocp-node3    
cdi                                 virt-launcher-f29vm-h6mc9                      1/1       Running     0          6h        10.129.0.15     ocp-node3    
kubevirt-web-ui                     console-854d4585c8-hgdhv                       1/1       Running     0          6h        10.129.0.10     ocp-node3    
kubevirt-web-ui                     kubevirt-web-ui-operator-6b4574bb95-bmsw7      1/1       Running     0          6h        10.130.0.11     ocp-node2    
kubevirt                            kubevirt-cpu-node-labeller-fvx9n               1/1       Running     0          6h        10.128.0.29     ocp-master   
kubevirt                            kubevirt-cpu-node-labeller-jr858               1/1       Running     0          6h        10.131.0.13     ocp-node1    
kubevirt                            kubevirt-cpu-node-labeller-tgq5g               1/1       Running     0          6h        10.129.0.14     ocp-node3    
kubevirt                            kubevirt-cpu-node-labeller-xqpbl               1/1       Running     0          6h        10.130.0.13     ocp-node2    
kubevirt                            virt-api-865b95d544-hg58l                      1/1       Running     0          6h        10.129.0.8      ocp-node3    
kubevirt                            virt-api-865b95d544-jrkxh                      1/1       Running     0          6h        10.131.0.10     ocp-node1    
kubevirt                            virt-controller-5c89d4978d-q79lh               1/1       Running     0          6h        10.130.0.8      ocp-node2    
kubevirt                            virt-controller-5c89d4978d-t58l7               1/1       Running     0          6h        10.130.0.10     ocp-node2    
kubevirt                            virt-handler-gblbk                             1/1       Running     0          6h        10.128.0.28     ocp-master   
kubevirt                            virt-handler-jnwx6                             1/1       Running     0          6h        10.130.0.9      ocp-node2    
kubevirt                            virt-handler-r94fb                             1/1       Running     0          6h        10.129.0.9      ocp-node3    
kubevirt                            virt-handler-z7775                             1/1       Running     0          6h        10.131.0.11     ocp-node1    
kubevirt                            virt-operator-68984b585c-265bq                 1/1       Running     0          6h        10.129.0.7      ocp-node3    

Now that CNV is up running lets pull down a Fedora 29 image and upload it into a PVC of the default storageclass which of course is Rook/Ceph:

[origin@ocp-master ~]$ curl -L -o /home/origin/f29.qcow2 http://ftp.usf.edu/pub/fedora/linux/releases/29/Cloud/x86_64/images/Fedora-Cloud-Base-29-1.2.x86_64.qcow2
[origin@ocp-master ~]$ virtctl image-upload --pvc-name=f29vm --pvc-size=5Gi --image-path=/home/origin/f29.qcow2 --uploadproxy-url=https://`oc describe route cdi-uploadproxy-route|grep Endpoints|cut -f2` --insecure

We can execute the following to see that the PVC has been created:

[origin@ocp-master ~]$ oc get pvc
NAME      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
f29vm     Bound     pvc-4815df9e-4987-11e9-a732-525400767d62   5Gi        RWO            rook-ceph-block   6h

Besides the PVC we will also need a virtual machine configuration yaml file.  The one below is an example that will be used in this demonstration:

apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
  creationTimestamp: null
  labels:
    kubevirt-vm: f29vm
  name: f29vm
spec:
  running: true
  template:
    metadata:
      creationTimestamp: null
      labels:
        kubevirt.io/domain: f29vm
    spec:
      domain:
        cpu:
          cores: 2
        devices:
          disks:
          - disk:
              bus: virtio
            name: osdisk
            volumeName: osdisk
          - disk:
              bus: virtio
            name: cloudinitdisk
            volumeName: cloudinitvolume
          interfaces:
          - name: default
            bridge: {}
        resources:
          requests:
            memory: 1024M
      terminationGracePeriodSeconds: 0
      volumes:
      - name: osdisk
        persistentVolumeClaim:
          claimName: f29vm
      - name: cloudinitdisk
        cloudInitNoCloud:
          userData: |-
            #cloud-config
            password: ${PASSWORD}
            disable_root: false
            chpasswd: { expire: False }
            ssh_authorized_keys:
            - "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUs1KbLraX74mBM/ksoGwbsEejfpCVeMzbW7JLJjGXF8G1jyVAE3T0Uf5mO8nbNOfkjAjw24lxSsEScF2wslBzA5MIm+GB6Z+ZzR55FcRlZeouGVrfLmb67mYc2c/F/mq35TruHdRk2G5Y0+6cf8cfDs414+yiVA0heHQvWNfO7kb1z9kIOhyD6OOwdNT5jK/1O0+p6SdP+pEal51BsEf6GRGYLWc9SLIEcqtjoprnundr5UPvmC1l/pkqFQigMehwhthrdXC4GseWiyj9CnBkccxQCKvHjzko/wqsWGQLwDG3pBsHhthvbY0G5+VPB9a8YV58WJhC6nHpUTDA8jpB origin@ocp-master"
      networks:
      - name: default
        pod: {}

At this point we have all the necessary components to launch our containerized virtual machine instance.   The following command does the creation using the yaml file we created in the previous step:

[origin@ocp-master ~]$ oc create -f /home/origin/f29vm.yaml

There are multiple ways to validate the virtual machine has been instantiated.   I like to do the following to confirm the instance is running and has an IP address:

[origin@ocp-master ~]$ oc get vms
NAME      AGE       RUNNING   VOLUME
f29vm     6h        true      
[origin@ocp-master ~]$ oc get vmi
NAME      AGE       PHASE     IP            NODENAME
f29vm     6h        Running   10.129.0.15   ocp-node3

One final step you can do is actually log into the instance assuming a key was set in the yaml file:

[origin@ocp-master ~]$ ssh -i /home/origin/.ssh/id_rsa -o "StrictHostKeyChecking=no" fedora@10.129.0.15
[fedora@f29vm ~]$ cat /etc/fedora-release
Fedora release 29 (Twenty Nine)

Hopefully this demonstrated how easy it is to get OpenShift, Rook and CNV up and running and how one can then leverage the storage of Rook to provide a backend for the virtual instance that gets spun up in CNV.   What is awesome is that I have taken the steps above and put them into a DCI job where I can automatically rerun the deployment using newer version of the code base for testing.   If you are not familiar with DCI I will leave with this tease link to DCI: https://doc.distributed-ci.io/

Replace Failed OSD in Rook Deployed Ceph

If you have been reading some of my recent articles on Rook you have seen how to install a Ceph cluster with Rook on Kubernetes. This article extends on that Kubernetes installation and discusses how to replace a failed OSD in the Ceph cluster.

First lets review our current running Ceph cluster observing the rook-ceph-system, rook-ceph and inside the toolbox the Ceph status:

# kubectl get pods --all-namespaces -o wide
NAMESPACE          NAME                                      READY   STATUS      RESTARTS   AGE    IP            NODE          NOMINATED NODE   READINESS GATES
kube-system        coredns-86c58d9df4-22fps                  1/1     Running     4          3d2h   10.244.3.55   kube-node3               
kube-system        coredns-86c58d9df4-jp2zb                  1/1     Running     6          3d2h   10.244.2.66   kube-node2               
kube-system        etcd-kube-master                          1/1     Running     3          3d5h   10.0.0.81     kube-master              
kube-system        kube-apiserver-kube-master                1/1     Running     3          3d5h   10.0.0.81     kube-master              
kube-system        kube-controller-manager-kube-master       1/1     Running     5          3d5h   10.0.0.81     kube-master              
kube-system        kube-flannel-ds-amd64-5m9x5               1/1     Running     6          3d5h   10.0.0.83     kube-node2               
kube-system        kube-flannel-ds-amd64-7xgf4               1/1     Running     3          3d5h   10.0.0.81     kube-master              
kube-system        kube-flannel-ds-amd64-dhdzm               1/1     Running     5          3d2h   10.0.0.84     kube-node3               
kube-system        kube-flannel-ds-amd64-m6fx5               1/1     Running     3          3d5h   10.0.0.82     kube-node1               
kube-system        kube-proxy-bnbzn                          1/1     Running     3          3d5h   10.0.0.82     kube-node1               
kube-system        kube-proxy-gjxlg                          1/1     Running     4          3d2h   10.0.0.84     kube-node3               
kube-system        kube-proxy-kkxdb                          1/1     Running     3          3d5h   10.0.0.81     kube-master              
kube-system        kube-proxy-knzsl                          1/1     Running     6          3d5h   10.0.0.83     kube-node2               
kube-system        kube-scheduler-kube-master                1/1     Running     4          3d5h   10.0.0.81     kube-master              
rook-ceph-system   rook-ceph-agent-748v8                     1/1     Running     0          103m   10.0.0.83     kube-node2               
rook-ceph-system   rook-ceph-agent-9vznf                     1/1     Running     0          103m   10.0.0.82     kube-node1               
rook-ceph-system   rook-ceph-agent-hfdv6                     1/1     Running     0          103m   10.0.0.81     kube-master              
rook-ceph-system   rook-ceph-agent-lfh7m                     1/1     Running     0          103m   10.0.0.84     kube-node3               
rook-ceph-system   rook-ceph-operator-76cf7f88f-qmvn5        1/1     Running     0          103m   10.244.1.65   kube-node1               
rook-ceph-system   rook-discover-25h5z                       1/1     Running     0          103m   10.244.1.66   kube-node1               
rook-ceph-system   rook-discover-dcm7k                       1/1     Running     0          103m   10.244.0.41   kube-master              
rook-ceph-system   rook-discover-t4qs7                       1/1     Running     0          103m   10.244.3.61   kube-node3               
rook-ceph-system   rook-discover-w2nv5                       1/1     Running     0          103m   10.244.2.72   kube-node2               
rook-ceph          rook-ceph-mgr-a-8649f78d9b-k6gwl          1/1     Running     0          100m   10.244.3.62   kube-node3               
rook-ceph          rook-ceph-mon-a-576d9d49cc-q9pm6          1/1     Running     0          101m   10.244.0.42   kube-master              
rook-ceph          rook-ceph-mon-b-85f7b6cb6b-pnrhs          1/1     Running     0          101m   10.244.1.67   kube-node1               
rook-ceph          rook-ceph-mon-c-668f7f658d-hjf2v          1/1     Running     0          101m   10.244.2.74   kube-node2               
rook-ceph          rook-ceph-osd-0-6f76d5cc4c-t75gg          1/1     Running     0          100m   10.244.2.76   kube-node2               
rook-ceph          rook-ceph-osd-1-5759cd47c4-szvfg          1/1     Running     0          100m   10.244.3.64   kube-node3               
rook-ceph          rook-ceph-osd-2-6d69b78fbf-7s4bm          1/1     Running     0          100m   10.244.0.44   kube-master              
rook-ceph          rook-ceph-osd-3-7b457fc56d-22gw6          1/1     Running     0          100m   10.244.1.69   kube-node1               
rook-ceph          rook-ceph-osd-prepare-kube-master-72kfz   0/2     Completed   0          100m   10.244.0.43   kube-master              
rook-ceph          rook-ceph-osd-prepare-kube-node1-jp68h    0/2     Completed   0          100m   10.244.1.68   kube-node1               
rook-ceph          rook-ceph-osd-prepare-kube-node2-j89pc    0/2     Completed   0          100m   10.244.2.75   kube-node2               
rook-ceph          rook-ceph-osd-prepare-kube-node3-drh4t    0/2     Completed   0          100m   10.244.3.63   kube-node3               
rook-ceph          rook-ceph-tools-76c7d559b6-qvh2r          1/1     Running     0          6s     10.0.0.82     kube-node1               

# kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash

# ceph status
  cluster:
    id:     edc7cac7-21a3-45ae-80a9-5d470afb7576
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum c,a,b
    mgr: a(active)
    osd: 4 osds: 4 up, 4 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0  objects, 0 B
    usage:   17 GiB used, 123 GiB / 140 GiB avail
    pgs:     
 
# ceph osd tree  
ID CLASS WEIGHT  TYPE NAME            STATUS REWEIGHT PRI-AFF 
-1       0.13715 root default                                 
-5       0.03429     host kube-master                         
 2   hdd 0.03429         osd.2            up  1.00000 1.00000 
-4       0.03429     host kube-node1                          
 3   hdd 0.03429         osd.3            up  1.00000 1.00000 
-2       0.03429     host kube-node2                          
 0   hdd 0.03429         osd.0            up  1.00000 1.00000 
-3       0.03429     host kube-node3                          
 1   hdd 0.03429         osd.1            up  1.00000 1.00000 

At this point the Ceph cluster is clean and in a healthy state.  However I am going to introduce some chaos and which will cause osd1 to go down.  In my case since this is a virtual lab I am going to just kill the OSD process and clear out osd1 data to mimic a failed drive.

Now when we look at the cluster state in the toolbox we can see OSD1 is down:

# kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash

# ceph status
  cluster:
    id:     edc7cac7-21a3-45ae-80a9-5d470afb7576
    health: HEALTH_WARN
            1 osds down
            1 host (1 osds) down
 
  services:
    mon: 3 daemons, quorum c,a,b
    mgr: a(active)
    osd: 4 osds: 3 up, 4 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0  objects, 0 B
    usage:   17 GiB used, 123 GiB / 140 GiB avail
    pgs:     
 
[root@kube-node1 /]# ceph osd tree
ID CLASS WEIGHT  TYPE NAME            STATUS REWEIGHT PRI-AFF 
-1       0.13715 root default                                 
-5       0.03429     host kube-master                         
 2   hdd 0.03429         osd.2            up  1.00000 1.00000 
-4       0.03429     host kube-node1                          
 3   hdd 0.03429         osd.3            up  1.00000 1.00000 
-2       0.03429     host kube-node2                          
 0   hdd 0.03429         osd.0            up  1.00000 1.00000 
-3       0.03429     host kube-node3                          
 1   hdd 0.03429         osd.1          down  1.00000 1.00000 

Given I removed the contents of the OSD lets go ahead and replace the failed drive. The first steps are to go into the toolbox and run the usual commands to remove a Ceph OSD from the cluster:

# kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash

# ceph osd out osd.1
marked out osd.1. 

# ceph osd crush remove osd.1
removed item id 1 name 'osd.1' from crush map

# ceph auth del osd.1
updated

# ceph osd rm osd.1
removed osd.1

Lets exit out of the toolbox and go back to the master node command line and delete the Ceph OSD 3 deployment:

# kubectl delete deployment -n rook-ceph rook-ceph-osd-1
deployment.extensions "rook-ceph-osd-1" deleted

Now would be the time to replace the physically failed disk. In my case the disk is still good I just simulated the failure by downing the OSD process and removing the data.

To get the new disk back into the cluster we only need to restart the rook-ceph-operator pod and we can do so in Kubernetes with the following scale deployment commands:

# kubectl scale deployment rook-ceph-operator --replicas=0 -n rook-ceph-system
deployment.extensions/rook-ceph-operator scaled

# kubectl get pods --all-namespaces -o wide|grep operator

# kubectl scale deployment rook-ceph-operator --replicas=1 -n rook-ceph-system
deployment.extensions/rook-ceph-operator scaled

# kubectl get pods --all-namespaces -o wide|grep operator
rook-ceph-system   rook-ceph-operator-76cf7f88f-g9pxr        0/1     ContainerCreating   0          2s              kube-node2               

When the rook-ceph-operator is restarted it will go through and re-run each rook-ceph-osd-prepare container which will scan the system it is on and look for any disks that should be incorporated into the cluster based on the original cluster.yaml settings when the Ceph cluster was deployed with Rook.  In this case it will see the new disk on kube-node-3 and incorporate that into OSD1.

We can confirm our assessment by seeing a new container for OSD1 was spawned and also by logging into the toolbox and running the familiar Ceph commands:

# kubectl get pods -n rook-ceph -o wide
NAME                                      READY   STATUS      RESTARTS   AGE     IP            NODE          NOMINATED NODE   READINESS GATES
rook-ceph-mgr-a-8649f78d9b-k6gwl          1/1     Running     0          110m    10.244.3.62   kube-node3    <none>           <none>
rook-ceph-mon-a-576d9d49cc-q9pm6          1/1     Running     0          110m    10.244.0.42   kube-master   <none>           <none>
rook-ceph-mon-b-85f7b6cb6b-pnrhs          1/1     Running     0          110m    10.244.1.67   kube-node1    <none>           <none>
rook-ceph-mon-c-668f7f658d-hjf2v          1/1     Running     0          110m    10.244.2.74   kube-node2    <none>           <none>
rook-ceph-osd-0-6f76d5cc4c-t75gg          1/1     Running     0          109m    10.244.2.76   kube-node2    <none>           <none>
rook-ceph-osd-1-69f5d5ffd-kndd7           1/1     Running     0          67s     10.244.3.68   kube-node3    <none>           <none>
rook-ceph-osd-2-6d69b78fbf-7s4bm          1/1     Running     0          109m    10.244.0.44   kube-master   <none>           <none>
rook-ceph-osd-3-7b457fc56d-22gw6          1/1     Running     0          109m    10.244.1.69   kube-node1    <none>           <none>
rook-ceph-osd-prepare-kube-master-n2t7g   0/2     Completed   0          79s     10.244.0.47   kube-master   <none>           <none>
rook-ceph-osd-prepare-kube-node1-ttznt    0/2     Completed   0          77s     10.244.1.72   kube-node1    <none>           <none>
rook-ceph-osd-prepare-kube-node2-9kxcl    0/2     Completed   0          75s     10.244.2.79   kube-node2    <none>           <none>
rook-ceph-osd-prepare-kube-node3-cpf4s    0/2     Completed   0          73s     10.244.3.66   kube-node3    <none>           <none>
rook-ceph-tools-76c7d559b6-qvh2r          1/1     Running     0          9m28s   10.0.0.82     kube-node1    <none>           <none>

# ceph status
  cluster:
    id:     edc7cac7-21a3-45ae-80a9-5d470afb7576
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum c,a,b
    mgr: a(active)
    osd: 4 osds: 4 up, 4 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0  objects, 0 B
    usage:   17 GiB used, 123 GiB / 140 GiB avail
    pgs:     

# ceph osd tree 
ID CLASS WEIGHT  TYPE NAME            STATUS REWEIGHT PRI-AFF 
-1       0.13715 root default                                 
-5       0.03429     host kube-master                         
 2   hdd 0.03429         osd.2            up  1.00000 1.00000 
-4       0.03429     host kube-node1                          
 3   hdd 0.03429         osd.3            up  1.00000 1.00000 
-2       0.03429     host kube-node2                          
 0   hdd 0.03429         osd.0            up  1.00000 1.00000 
-3       0.03429     host kube-node3                          
 1       0.03429         osd.1            up  1.00000 1.00000 

As you can see replacing a failed OSD with Rook is about as uneventful as replacing a failed OSD in a standard deployed Ceph cluster.   Hopefully this demonstration provided the proof of that.

Further Reading:

Rook: https://github.com/rook/rook