Monitoring Sensors and Taking Action

I recently wrote a blog around using Microshift to run my Zigbee2MQTT workload. This blog described all the details on how to deploy Microshift and then deploy the components inside of Microshift to enable some home automation. Of course with Zigbee2MQTT there is an intuitive web interface to interact with the smart devices. However I wanted to take another approach that felt more realistic when it comes to edge use cases. I felt that in a industrial scenario there would be some code that would most likely subscribed and monitoring the MQTT queue. An action would be performed when a certain event was observed and the action itself might publish something into the MQTT queue. The rest of this blog will cover a simple scenario like I just described.

First we continue to use the same lab environment I used in my previous blog. The only difference here in the diagram below is we have now added a smart power outlet and a temperature/humidity sensor that can both be controlled remotely via the Zigbee protocol like all my other devices.

The Script

With my lab in place I decided I wanted ot write something in Perl. Some might think why use such an antiquated language like Perl and part of that is because I am old school. For my scenario I envisioned using the humidity sensor to detect when the humidity levels got too high. The threshold would then trigger an action on the event to turn on/off a dehumidifier plugged into the smart outlet. The basic process flow looks like the following diagram:

The script itself can take four different parameters:

• --hostname: hostname or IP address of MQTT host (required)
• --port: port for MQTT (optional but will default to 1883 if not provided)
• --threshold: humidity value that determines when action should be taken
• --help: prints the usage of script

The script itself is located here

When one runs the script without any flags the usage and an example will be displayed.

./mqtt-humidity.pl 
Usage:
      --hostname,-h   Hostname or IP address of MQTT host
      --port,-p       Port for MQTT (defaults to default 1883)
      --threshold,-t  Threshold for humidity (defaults to 60)
      --help,-h       Print this help

    Example:

    mqtt-humidity.pl -ho 10.43.26.170 -p 1883 -t 65

The Demonstration of Script

To demonstrate this script I went ahead and plugged in a light into my smart outlet which was in the off setting. I launched the script in a terminal window. Then I took the temperature/humidity sensor, cupped it in my hands and blew into my hands. The moisture in my breath is enough to temporarily raise the value. The script provides output so we can see the values changing and sure enough when I breathed into my hands with the sensor the value jumped to 81.37% which triggered the action event and turned on the light. I then set the sensor back on my desk and over the course of 5 minutes the value slowly receded. Once it dropped below the threshold value the light then turned back off. The output of my script run is below:

$ perl mqtt-humidity.pl -ho 10.43.26.170 -p 1883 -t 60
Temp C = 23.43 : Temp F = 74.174 : Humidity = 51.22
Temp C = 23.43 : Temp F = 74.174 : Humidity = 81.37 <-- Smart outlet turned on
Temp C = 23.43 : Temp F = 74.174 : Humidity = 84.37
Temp C = 23.43 : Temp F = 74.174 : Humidity = 82.37
Temp C = 23.43 : Temp F = 74.174 : Humidity = 80.28
Temp C = 23.43 : Temp F = 74.174 : Humidity = 78.79
Temp C = 23.63 : Temp F = 74.534 : Humidity = 78.79
Temp C = 23.63 : Temp F = 74.534 : Humidity = 73.21
Temp C = 23.63 : Temp F = 74.534 : Humidity = 74.34
Temp C = 23.63 : Temp F = 74.534 : Humidity = 72.91
Temp C = 23.63 : Temp F = 74.534 : Humidity = 71.65
Temp C = 23.63 : Temp F = 74.534 : Humidity = 70.55
Temp C = 23.63 : Temp F = 74.534 : Humidity = 69.15
Temp C = 23.63 : Temp F = 74.534 : Humidity = 67.93
Temp C = 23.63 : Temp F = 74.534 : Humidity = 66.57
Temp C = 23.63 : Temp F = 74.534 : Humidity = 64.87
Temp C = 23.63 : Temp F = 74.534 : Humidity = 63.28
Temp C = 23.63 : Temp F = 74.534 : Humidity = 62.08
Temp C = 23.63 : Temp F = 74.534 : Humidity = 60.71 
Temp C = 23.63 : Temp F = 74.534 : Humidity = 59.12 <-- Smart outlet turned off
Temp C = 23.63 : Temp F = 74.534 : Humidity = 57.72
Temp C = 23.63 : Temp F = 74.534 : Humidity = 56.7
Temp C = 23.63 : Temp F = 74.534 : Humidity = 55.6
Temp C = 23.63 : Temp F = 74.534 : Humidity = 54.23
Temp C = 23.63 : Temp F = 74.534 : Humidity = 53.21
Temp C = 23.63 : Temp F = 74.534 : Humidity = 52.14
Temp C = 23.63 : Temp F = 74.534 : Humidity = 51.05
Temp C = 23.63 : Temp F = 74.534 : Humidity = 50.04
Temp C = 23.22 : Temp F = 73.796 : Humidity = 50.04
Temp C = 23.22 : Temp F = 73.796 : Humidity = 50.04
^C

Now this was a very simple example but imagine the possibilities. For example what if this was a greenhouse that needed to keep the humidity and/or even the temperature at a certain range. If the device that reduces the humidity/temperature (dehumidifier -or- exhaust fan) in the greenhouse could take Zigbee commands directly and control the speed of operation we might be able to not only turn it on/off but also increase/decrease speed of operation. All of this ensures that whatever was growing in the greenhouse is not damaged and also ensures we are powering devices only when we need to have them powered. The bottom line is it saves businesses like the greenhouse operational costs when they are operating efficiently.

Deploy Rook/Ceph Cluster on Dedicated Networks

Recently a colleague of mine was trying to get Rook to deploy a Ceph cluster that used dedicated public and private networks to segment the Ceph replication traffic and the client access traffic to the OSDs of the cluster.   In a regular Ceph deployment this is rather trivial but when in the context of Kubernetes it becomes a little more complex given that Rook is deploying the cluster containers.  The following is procedure I applied to ensure my OSDs were listening on the appropriate networks.

Before we get into the steps on how to achieve this configuration lets quick take a look at the setup I used.  First I have a three node Kubernetes configuration (1 master with allowed scheduling and two workers):

# kubectl get nodes
NAME          STATUS   ROLES    AGE     VERSION
kube-master   Ready    master   2d22h   v1.14.0
kube-node1    Ready    worker   2d22h   v1.14.0
kube-node2    Ready    worker   2d22h   v1.14.0

On each of the nodes I have 3 network interfaces: eth0 on 10.0.0.0/24 (Kubernetes public), eth1 on 192.168.100.0/24 (Ceph private/cluster) & eth2 on 192.168.200.0/24 (Ceph public):

# ip a|grep eth[0-2]
2: eth0:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    inet 10.0.0.81/24 brd 10.0.0.255 scope global noprefixroute eth0
3: eth1:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    inet 192.168.100.81/24 brd 192.168.100.255 scope global noprefixroute eth1
4: eth2:  mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    inet 192.168.200.81/24 brd 192.168.200.255 scope global noprefixroute eth2

Before we begin lets see the current vanilla pods and namespaces on the Kubernetes cluster:

# kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                  READY   STATUS    RESTARTS   AGE   IP           NODE          NOMINATED NODE   READINESS GATES
kube-system   coredns-fb8b8dccf-h6wfn               1/1     Running   0          3d    10.244.1.2   kube-node2               
kube-system   coredns-fb8b8dccf-mv7p5               1/1     Running   0          3d    10.244.0.7   kube-master              
kube-system   etcd-kube-master                      1/1     Running   0          3d    10.0.0.81    kube-master              
kube-system   kube-apiserver-kube-master            1/1     Running   0          3d    10.0.0.81    kube-master              
kube-system   kube-controller-manager-kube-master   1/1     Running   1          3d    10.0.0.81    kube-master              
kube-system   kube-flannel-ds-amd64-szhg9           1/1     Running   0          3d    10.0.0.83    kube-node2               
kube-system   kube-flannel-ds-amd64-t4fxs           1/1     Running   0          3d    10.0.0.82    kube-node1               
kube-system   kube-flannel-ds-amd64-wbsdp           1/1     Running   0          3d    10.0.0.81    kube-master              
kube-system   kube-proxy-sn7j7                      1/1     Running   0          3d    10.0.0.83    kube-node2               
kube-system   kube-proxy-wtzm5                      1/1     Running   0          3d    10.0.0.81    kube-master              
kube-system   kube-proxy-xlwd9                      1/1     Running   0          3d    10.0.0.82    kube-node1               
kube-system   kube-scheduler-kube-master            1/1     Running   1          3d    10.0.0.81    kube-master              

# kubectl get ns
NAME              STATUS   AGE
default           Active   3d
kube-node-lease   Active   3d
kube-public       Active   3d
kube-system       Active   3d

Before can deploy the cluster we need to create a configmap for the rook-ceph namespace.  This namespace is normally created when the cluster is deployed however we want specific configuration items to be incorporated into the cluster upon deployment and so to do this we will create the rook-ceph namespace and apply a configmap that we create to that namespace.

First create a configmap file that looks like the following and notice I am referencing my Ceph cluster networks.  I will save this file with an arbitrary name like config-override.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: rook-config-override
  namespace: rook-ceph
data:
  config: |
    [global]
    public network =  192.168.200.0/24
    cluster network = 192.168.100.0/24
    public addr = ""
    cluster addr = ""

Next I will create the rook-ceph namespace:

# kubectl create namespace rook-ceph
namespace/rook-ceph created

# kubectl get ns
NAME              STATUS   AGE
default           Active   3d1h
kube-node-lease   Active   3d1h
kube-public       Active   3d1h
kube-system       Active   3d1h
rook-ceph         Active   5s

Now we can apply the configmap we created to the newly created namespace and validate its there:

# kubectl create -f config-override.yaml 
configmap/rook-config-override created

# kubectl get configmap -n rook-ceph
NAME                   DATA   AGE
rook-config-override   1      66s

# kubectl describe configmap -n rook-ceph
Name:         rook-config-override
Namespace:    rook-ceph
Labels:       <none>
Annotations:  <none>

Data
====
config:
----
[global]
public network =  192.168.200.0/24
cluster network = 192.168.100.0/24
public addr = ""
cluster addr = ""

Events:  <none>

Before we actually start to do the deploy we need to update one more thing in our Rook cluster.yaml.  Inside the cluster.yaml file we need to change hostNetwork from the default of false to true:

 sed -i 's/hostNetwork: false/hostNetwork: true/g' cluster.yaml

Now we can begin the process of deploying the Rook/Ceph cluster that includes launching the operator, cluster and toolbox.   I will place sleep statements in between each command to ensure the pods are up before I run the next command.  Also note there will be an error when creating the cluster about the rook-ceph namespace already existing and this is normal:

# kubectl create -f operator.yaml
namespace/rook-ceph-system created
customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/volumes.rook.io created
clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created
role.rbac.authorization.k8s.io/rook-ceph-system created
clusterrole.rbac.authorization.k8s.io/rook-ceph-global created
clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created
serviceaccount/rook-ceph-system created
rolebinding.rbac.authorization.k8s.io/rook-ceph-system created
clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created
deployment.apps/rook-ceph-operator created

# sleep 60

# kubectl create -f cluster.yaml 
serviceaccount/rook-ceph-osd created
serviceaccount/rook-ceph-mgr created
role.rbac.authorization.k8s.io/rook-ceph-osd created
role.rbac.authorization.k8s.io/rook-ceph-mgr-system created
role.rbac.authorization.k8s.io/rook-ceph-mgr created
rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created
rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created
rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created
rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created
rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created
cephcluster.ceph.rook.io/rook-ceph created
Error from server (AlreadyExists): error when creating "cluster.yaml": namespaces "rook-ceph" already exists

# sleep 60

# kubectl create -f toolbox.yaml 
pod/rook-ceph-tools created

Lets validate the Rook/Ceph operator, cluster and toolbox is up and running:

# kubectl get pods --all-namespaces -o wide
NAMESPACE          NAME                                      READY   STATUS      RESTARTS   AGE     IP           NODE          NOMINATED NODE   READINESS GATES
kube-system        coredns-fb8b8dccf-h6wfn                   1/1     Running     0          3d1h    10.244.1.2   kube-node2    <none>           <none>
kube-system        coredns-fb8b8dccf-mv7p5                   1/1     Running     0          3d1h    10.244.0.7   kube-master   <none>           <none>
kube-system        etcd-kube-master                          1/1     Running     0          3d1h    10.0.0.81    kube-master   <none>           <none>
kube-system        kube-apiserver-kube-master                1/1     Running     0          3d1h    10.0.0.81    kube-master   <none>           <none>
kube-system        kube-controller-manager-kube-master       1/1     Running     1          3d1h    10.0.0.81    kube-master   <none>           <none>
kube-system        kube-flannel-ds-amd64-szhg9               1/1     Running     0          3d1h    10.0.0.83    kube-node2    <none>           <none>
kube-system        kube-flannel-ds-amd64-t4fxs               1/1     Running     0          3d1h    10.0.0.82    kube-node1    <none>           <none>
kube-system        kube-flannel-ds-amd64-wbsdp               1/1     Running     0          3d1h    10.0.0.81    kube-master   <none>           <none>
kube-system        kube-proxy-sn7j7                          1/1     Running     0          3d1h    10.0.0.83    kube-node2    <none>           <none>
kube-system        kube-proxy-wtzm5                          1/1     Running     0          3d1h    10.0.0.81    kube-master   <none>           <none>
kube-system        kube-proxy-xlwd9                          1/1     Running     0          3d1h    10.0.0.82    kube-node1    <none>           <none>
kube-system        kube-scheduler-kube-master                1/1     Running     1          3d1h    10.0.0.81    kube-master   <none>           <none>
rook-ceph-system   rook-ceph-agent-55fqp                     1/1     Running     0          17m     10.0.0.83    kube-node2    <none>           <none>
rook-ceph-system   rook-ceph-agent-5v9v5                     1/1     Running     0          17m     10.0.0.81    kube-master   <none>           <none>
rook-ceph-system   rook-ceph-agent-spx29                     1/1     Running     0          17m     10.0.0.82    kube-node1    <none>           <none>
rook-ceph-system   rook-ceph-operator-57547fc866-ltp8z       1/1     Running     0          18m     10.244.2.4   kube-node1    <none>           <none>
rook-ceph-system   rook-discover-brxmt                       1/1     Running     0          17m     10.244.2.5   kube-node1    <none>           <none>
rook-ceph-system   rook-discover-hl748                       1/1     Running     0          17m     10.244.1.8   kube-node2    <none>           <none>
rook-ceph-system   rook-discover-qj5kd                       1/1     Running     0          17m     10.244.0.9   kube-master   <none>           <none>
rook-ceph          rook-ceph-mgr-a-5dbb44d7f8-vzs46          1/1     Running     0          16m     10.0.0.82    kube-node1    <none>           <none>
rook-ceph          rook-ceph-mon-a-5fb9568cb4-gvqln          1/1     Running     0          16m     10.0.0.81    kube-master   <none>           <none>
rook-ceph          rook-ceph-mon-b-b65c555bf-vz7ps           1/1     Running     0          16m     10.0.0.82    kube-node1    <none>           <none>
rook-ceph          rook-ceph-mon-c-69cf744c4d-8g4l6          1/1     Running     0          16m     10.0.0.83    kube-node2    <none>           <none>
rook-ceph          rook-ceph-osd-0-77499f547-d2vjx           1/1     Running     0          15m     10.0.0.81    kube-master   <none>           <none>
rook-ceph          rook-ceph-osd-1-698f76d786-lqn4w          1/1     Running     0          15m     10.0.0.82    kube-node1    <none>           <none>
rook-ceph          rook-ceph-osd-2-558c59d577-wfdlr          1/1     Running     0          15m     10.0.0.83    kube-node2    <none>           <none>
rook-ceph          rook-ceph-osd-prepare-kube-master-p55sw   0/2     Completed   0          15m     10.0.0.81    kube-master   <none>           <none>
rook-ceph          rook-ceph-osd-prepare-kube-node1-q7scn    0/2     Completed   0          15m     10.0.0.82    kube-node1    <none>           <none>
rook-ceph          rook-ceph-osd-prepare-kube-node2-8rm4d    0/2     Completed   0          15m     10.0.0.83    kube-node2    <none>           <none>
rook-ceph          rook-ceph-tools                           1/1     Running     0          3m24s   10.244.1.9   kube-node2    <none>           <none>

# kubectl -n rook-ceph exec -it rook-ceph-tools -- /bin/bash
bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory
[root@rook-ceph-tools /]# ceph status
  cluster:
    id:     b58f2a5c-2fc7-43e7-b410-2d541e78a90e
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum a,b,c
    mgr: a(active)
    osd: 3 osds: 3 up, 3 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0  objects, 0 B
    usage:   57 GiB used, 49 GiB / 105 GiB avail
    pgs:     
 
[root@rook-ceph-tools /]# exit
exit

At this point we have a fully operational cluster but is it really using the networks for OSD public and private traffic?   Lets explore that a bit further by first running the netstat command on any node in the cluster that has an OSD pod running.  Since my cluster is small I will show all 3 nodes below:

[root@kube-master]# netstat -tulpn | grep LISTEN | grep osd
tcp        0      0 192.168.100.81:6800     0.0.0.0:*               LISTEN      29719/ceph-osd      
tcp        0      0 192.168.200.81:6800     0.0.0.0:*               LISTEN      29719/ceph-osd      
tcp        0      0 192.168.200.81:6801     0.0.0.0:*               LISTEN      29719/ceph-osd      
tcp        0      0 192.168.100.81:6801     0.0.0.0:*               LISTEN      29719/ceph-osd
[root@kube-node1]# netstat -tulpn | grep LISTEN | grep osd
tcp        0      0 192.168.100.82:6800     0.0.0.0:*               LISTEN      18770/ceph-osd      
tcp        0      0 192.168.100.82:6801     0.0.0.0:*               LISTEN      18770/ceph-osd      
tcp        0      0 192.168.200.82:6801     0.0.0.0:*               LISTEN      18770/ceph-osd      
tcp        0      0 192.168.200.82:6802     0.0.0.0:*               LISTEN      18770/ceph-osd

[root@kube-node2]# netstat -tulpn | grep LISTEN | grep osd
tcp        0      0 192.168.100.83:6800     0.0.0.0:*               LISTEN      22659/ceph-osd      
tcp        0      0 192.168.200.83:6800     0.0.0.0:*               LISTEN      22659/ceph-osd      
tcp        0      0 192.168.200.83:6801     0.0.0.0:*               LISTEN      22659/ceph-osd      
tcp        0      0 192.168.100.83:6801     0.0.0.0:*               LISTEN      22659/ceph-osd

From the above we should see the OSD processes listening on the corresponding public and private networks we configured in the configmap.   However lets further confirm by going back into the toolbox and doing a ceph osd dump:

# kubectl -n rook-ceph exec -it rook-ceph-tools -- /bin/bash
bash: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory
bash: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory

[root@rook-ceph-tools]# ceph osd dump
epoch 14
fsid 05a8b767-e3e8-42aa-b792-69f479c807f7
created 2019-04-02 13:24:24.549423
modified 2019-04-02 13:25:28.441850
flags sortbitwise,recovery_deletes,purged_snapdirs
crush_version 7
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85
require_min_compat_client jewel
min_compat_client firefly
require_osd_release mimic
max_osd 3
osd.0 up   in  weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) 192.168.200.81:6800/29719 192.168.100.81:6800/29719 192.168.100.81:6801/29719 192.168.200.81:6801/29719 exists,up 2feb0edf-6652-4148-8264-6ba52d04ff80
osd.1 up   in  weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) 192.168.200.82:6801/18770 192.168.100.82:6800/18770 192.168.100.82:6801/18770 192.168.200.82:6802/18770 exists,up f8df61b4-4ac8-4705-9f97-eb09a1cc0d6c
osd.2 up   in  weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) 192.168.200.83:6800/22659 192.168.100.83:6800/22659 192.168.100.83:6801/22659 192.168.200.83:6801/22659 exists,up db555c80-9d81-4662-aed9-4bce1c0d5d78

As you can see it can be fairly straight forward to configure Rook to deploy a Ceph cluster using segmented networks to ensure the replication traffic runs on dedicated network and does not interfere with public client performance.  Hopefully this quick demonstrate showed that.

Stacking OpenShift with Rook and CNV

In previous blogs I was working with Rook/Ceph on Kubernetes and demonstrating how to setup a Ceph cluster and even replace failed OSDs. With that in mind I wanted to shift gears a bit and bring it more into alignment with OpenShift and Container Native Virtualization(CNV).

The following blog will guide us through a simple OpenShift deployment with Rook/Ceph and CNV configured. I will also demonstrate the use of a Rook PVC that provides the back end storage for a CNV deployed virtual instance.

The configuration for this lab is four virtual machines where one node is the master and compute and the other 3 nodes compute.  Each of these nodes has a base install of Red Hat Enterprise Linux 7 on it and the physical host they are on allows for nested virtualization.

Before we start with the installation of various software lets make sure we do a bit of user setup to ensure our install runs smoothly.  The next few steps will need to be done on all nodes to ensure a user origin (this could be any non root user) is created and has sudo rights without use of a password:

# useradd origin
# passwd origin
# echo -e 'Defaults:origin !requiretty\norigin ALL = (root) NOPASSWD:ALL' | tee /etc/sudoers.d/openshift 
# chmod 440 /etc/sudoers.d/openshift

Then we need to perform the the following steps to setup keyless authentication for the origin user from the master node to the rest of the nodes that will make up the cluster:

# ssh-keygen -q -N ""
# vi /home/origin/.ssh/config
Host ocp-master
    Hostname ocp-master.schmaustech.com
    User origin
Host ocp-node1
    Hostname ocp-node1.schmaustech.com
    User origin
Host ocp-node2
    Hostname ocp-node2.schmaustech.com
    User origin
Host ocp-node3
    Hostname ocp-node3.schmaustech.com
    User origin

# chmod 600 /home/origin/.ssh/config
# ssh-copy-id ocp-master
# ssh-copy-id ocp-node1
# ssh-copy-id ocp-node2
# ssh-copy-id ocp-node3

Now we can move on to enabling the necessary repositries on all nodes to ensure we can get access to the right packages we will need for installation:

[origin@ocp-master ~]$ sudo subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-extras-rpms --enable=rhel-7-server-rh-common-rpms --enable=rhel-7-server-ose-3.11-rpms --enable=rhel-7-server-ansible-2.6-rpms --enable=rhel-7-server-cnv-1.4-tech-preview-rpms

Next lets install the initial required packages on all the nodes:

[origin@ocp-master ~]$ sudo yum -y install openshift-ansible docker-1.13.1 kubevirt-ansible kubevirt-virtctl

On the master node lets configure the Ansible hosts file for our OpenShift installation.   The following is the example I used and I simply replaced /etc/ansible/hosts with it.

[OSEv3:children]
masters
nodes
etcd
[OSEv3:vars]
# admin user created in previous section
ansible_ssh_user=origin
ansible_become=true
oreg_url=registry.access.redhat.com/openshift3/ose-${component}:${version}
openshift_deployment_type=openshift-enterprise
#  use HTPasswd for authentication
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider'}]
# define default sub-domain for Master node
openshift_master_default_subdomain=apps.schmaustech.com
# allow unencrypted connection within cluster
openshift_docker_insecure_registries=172.30.0.0/16
[masters]
ocp-master.schmaustech.com openshift_schedulable=true containerized=false
[etcd]
ocp-master.schmaustech.com
[nodes]
# defined values for [openshift_node_group_name] in the file below
# [/usr/share/ansible/openshift-ansible/roles/openshift_facts/defaults/main.yml]
ocp-master.schmaustech.com openshift_node_group_name='node-config-all-in-one'
ocp-node1.schmaustech.com openshift_node_group_name='node-config-compute'
ocp-node2.schmaustech.com openshift_node_group_name='node-config-compute'
ocp-node3.schmaustech.com openshift_node_group_name='node-config-compute'

With the Ansible host file in place we are ready to run the OpenShift prerequisite playbook:

[origin@ocp-master ~]$ ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/prerequisites.yml

Once the prerequisite playbook executes sucessfully we can then run the OpenShift deploy cluster playbook:

[origin@ocp-master ~]$ ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/deploy_cluster.yml

Lets validate OpenShift is up and running:

[origin@ocp-master ~]$ oc get nodes
NAME         STATUS    ROLES                  AGE       VERSION
ocp-master   Ready     compute,infra,master   15h       v1.11.0+d4cacc0
ocp-node1    Ready     compute                14h       v1.11.0+d4cacc0
ocp-node2    Ready     compute                14h       v1.11.0+d4cacc0
ocp-node3    Ready     compute                14h       v1.11.0+d4cacc0

[origin@ocp-master ~]$ oc get pods --all-namespaces -o wide
NAMESPACE                           NAME                                           READY     STATUS      RESTARTS   AGE       IP              NODE         NOMINATED NODE
default                             docker-registry-1-g4hgd                        1/1       Running     0          14h       10.128.0.4      ocp-master   <none>
default                             registry-console-1-zwhrd                       1/1       Running     0          14h       10.128.0.6      ocp-master   <none>
default                             router-1-v8pkp                                 1/1       Running     0          14h       192.168.3.100   ocp-master   <none>
kube-service-catalog                apiserver-gxjst                                1/1       Running     0          14h       10.128.0.17     ocp-master   <none>
kube-service-catalog                controller-manager-2v6qs                       1/1       Running     3          14h       10.128.0.18     ocp-master   <none>
openshift-ansible-service-broker    asb-1-d8clq                                    1/1       Running     0          14h       10.128.0.21     ocp-master   <none>
openshift-console                   console-566f847459-pk52j                       1/1       Running     0          14h       10.128.0.12     ocp-master   <none>
openshift-monitoring                alertmanager-main-0                            3/3       Running     0          14h       10.128.0.14     ocp-master   <none>
openshift-monitoring                alertmanager-main-1                            3/3       Running     0          14h       10.128.0.15     ocp-master   <none>
openshift-monitoring                alertmanager-main-2                            3/3       Running     0          14h       10.128.0.16     ocp-master   <none>
openshift-monitoring                cluster-monitoring-operator-79d6c544f5-c8rfs   1/1       Running     0          14h       10.128.0.7      ocp-master   <none>
openshift-monitoring                grafana-8497b48bd5-bqzxb                       2/2       Running     0          14h       10.128.0.10     ocp-master   <none>
openshift-monitoring                kube-state-metrics-7d8b57fc8f-ktdq4            3/3       Running     0          14h       10.128.0.19     ocp-master   <none>
openshift-monitoring                node-exporter-5gmbc                            2/2       Running     0          14h       192.168.3.103   ocp-node3    <none>
openshift-monitoring                node-exporter-fxthd                            2/2       Running     0          14h       192.168.3.102   ocp-node2    <none>
openshift-monitoring                node-exporter-gj27b                            2/2       Running     0          14h       192.168.3.101   ocp-node1    <none>
openshift-monitoring                node-exporter-r6vjs                            2/2       Running     0          14h       192.168.3.100   ocp-master   <none>
openshift-monitoring                prometheus-k8s-0                               4/4       Running     1          14h       10.128.0.11     ocp-master   <none>
openshift-monitoring                prometheus-k8s-1                               4/4       Running     1          14h       10.128.0.13     ocp-master   <none>
openshift-monitoring                prometheus-operator-5677fb6f87-4czth           1/1       Running     0          14h       10.128.0.8      ocp-master   <none>
openshift-node                      sync-7rqcb                                     1/1       Running     0          14h       192.168.3.103   ocp-node3    <none>
openshift-node                      sync-829ql                                     1/1       Running     0          14h       192.168.3.101   ocp-node1    <none>
openshift-node                      sync-mwq6v                                     1/1       Running     0          14h       192.168.3.102   ocp-node2    <none>
openshift-node                      sync-vc4hw                                     1/1       Running     0          15h       192.168.3.100   ocp-master   <none>
openshift-sdn                       ovs-n55b8                                      1/1       Running     0          14h       192.168.3.101   ocp-node1    <none>
openshift-sdn                       ovs-nvtgq                                      1/1       Running     0          14h       192.168.3.103   ocp-node3    <none>
openshift-sdn                       ovs-t8dgh                                      1/1       Running     0          14h       192.168.3.102   ocp-node2    <none>
openshift-sdn                       ovs-wgw2v                                      1/1       Running     0          15h       192.168.3.100   ocp-master   <none>
openshift-sdn                       sdn-7r9kn                                      1/1       Running     0          14h       192.168.3.101   ocp-node1    <none>
openshift-sdn                       sdn-89284                                      1/1       Running     0          15h       192.168.3.100   ocp-master   <none>
openshift-sdn                       sdn-hmgjg                                      1/1       Running     0          14h       192.168.3.103   ocp-node3    <none>
openshift-sdn                       sdn-n7lzh                                      1/1       Running     0          14h       192.168.3.102   ocp-node2    <none>
openshift-template-service-broker   apiserver-md5sr                                1/1       Running     0          14h       10.128.0.22     ocp-master   <none>
openshift-web-console               webconsole-674f79b6fc-cjrhw                    1/1       Running     0          14h       10.128.0.9      ocp-master   <none>

With OpenShift up and running we can move onto install Rook/Ceph cluster.  The first step is to clone the Rook Git repo down to the master node and make an adjustment for the kubelet-plugins.  Please note here I am cloning down a colleagues Rook clone and not direct from the Rook project:

[origin@ocp-master ~]$ git clone https://github.com/ksingh7/ocp4-rook.git
[origin@ocp-master ~]$ sed -i.bak s+/etc/kubernetes/kubelet-plugins/volume/exec+/usr/libexec/kubernetes/kubelet-plugins/volume/exec+g /home/origin/ocp4-rook/ceph/operator.yaml

With the repository cloned we can now apply the the security context constraints needed by the Rook pods using the scc.yaml and then launch the Rook operator with operator.yaml:

[origin@ocp-master ~]$ oc create -f /home/origin/ocp4-rook/ceph/scc.yaml
[origin@ocp-master ~]$ oc create -f /home/origin/ocp4-rook/ceph/operator.yaml

Lets validate the Rook operator came up:

[origin@ocp-master ~]$ oc get pods -n rook-ceph-system 
NAME                                 READY     STATUS    RESTARTS   AGE
rook-ceph-agent-77x5n                1/1       Running   0          1h
rook-ceph-agent-cdvqr                1/1       Running   0          1h
rook-ceph-agent-gz7tl                1/1       Running   0          1h
rook-ceph-agent-rsbwh                1/1       Running   0          1h
rook-ceph-operator-b76466dcd-zmscb   1/1       Running   0          1h
rook-discover-6p5ht                  1/1       Running   0          1h
rook-discover-fnrf4                  1/1       Running   0          1h
rook-discover-grr5w                  1/1       Running   0          1h
rook-discover-mllt7                  1/1       Running   0          1h

Once the operator is up we can proceed on deploying the Ceph cluster and once that is up deploy the Ceph toolbox pod:

[origin@ocp-master ~]$ oc create -f /home/origin/ocp4-rook/ceph/cluster.yaml  
[origin@ocp-master ~]$ oc create -f /home/origin/ocp4-rook/ceph/toolbox.yaml

Lets validate the Ceph cluster is up:

[origin@ocp-master ~]$ oc get pods -n rook-ceph
NAME                                     READY     STATUS      RESTARTS   AGE
rook-ceph-mgr-a-785ddd6d6c-d4w56         1/1       Running     0          1h
rook-ceph-mon-a-67855c796b-sdvqm         1/1       Running     0          1h
rook-ceph-mon-b-6d58cd7656-xkrdz         1/1       Running     0          1h
rook-ceph-mon-c-869b8d9d9-m7544          1/1       Running     0          1h
rook-ceph-osd-0-d6cbd5776-987p9          1/1       Running     0          1h
rook-ceph-osd-1-cfddf997-pzq69           1/1       Running     0          1h
rook-ceph-osd-2-79fc94c6d5-krtnj         1/1       Running     0          1h
rook-ceph-osd-3-f9b55c4d6-7jp7c          1/1       Running     0          1h
rook-ceph-osd-prepare-ocp-master-ztmhs   0/2       Completed   0          1h
rook-ceph-osd-prepare-ocp-node1-mgbcd    0/2       Completed   0          1h
rook-ceph-osd-prepare-ocp-node2-98rtw    0/2       Completed   0          1h
rook-ceph-osd-prepare-ocp-node3-ngscg    0/2       Completed   0          1h
rook-ceph-tools                          1/1       Running     0          1h

Lets also validate from the Ceph toolbox that the cluster health is ok:

[origin@ocp-master ~]$ oc -n rook-ceph rsh rook-ceph-tools
sh-4.2# ceph status
  cluster:
    id:     6ddab3e4-1730-412f-89b8-0738708adac8
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum b,a,c
    mgr: a(active)
    osd: 4 osds: 4 up, 4 in
 
  data:
    pools:   1 pools, 100 pgs
    objects: 281  objects, 1.1 GiB
    usage:   51 GiB used, 169 GiB / 220 GiB avail
    pgs:     100 active+clean

Now that we have confirmed the Ceph cluster is deployed lets configure a Ceph storage class and also make it the default storage class for the environment:

[origin@ocp-master ~]$ oc create -f /home/origin/ocp4-rook/ceph/storageclass.yaml
[origin@ocp-master ~]$ oc patch storageclass rook-ceph-block -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

And now if we display the storage class we can see Rook/Ceph is our default:

[origin@ocp-master ~]$ oc get storageclass
NAME                        PROVISIONER          AGE
rook-ceph-block (default)   ceph.rook.io/block   6h

Proceeding with our stack installation lets get CNV installed.  Again with the use of the Ansible playbook we used earlier for OpenShift this is a relatively easy task:

[origin@ocp-master ~]$ oc login -u system:admin
[origin@ocp-master ~]$ cd /usr/share/ansible/kubevirt-ansible
[origin@ocp-master ~]$ ansible-playbook -i /etc/ansible/hosts -e @vars/cnv.yml playbooks/kubevirt.yml -e apb_action=provision

Once the installation completes lets run the following command to ensure the pods for CNV are up:

[origin@ocp-master ~]$ oc get pods --all-namespaces -o wide|egrep "kubevirt|cdi"
cdi                                 cdi-apiserver-7bfd97d585-tqjgt                 1/1       Running     0          6h        10.129.0.11     ocp-node3    
cdi                                 cdi-deployment-6689fcb476-4klcj                1/1       Running     0          6h        10.131.0.12     ocp-node1    
cdi                                 cdi-operator-5889d7588c-wvgl4                  1/1       Running     0          6h        10.130.0.12     ocp-node2    
cdi                                 cdi-uploadproxy-79c9fb9f59-pkskw               1/1       Running     0          6h        10.129.0.13     ocp-node3    
cdi                                 virt-launcher-f29vm-h6mc9                      1/1       Running     0          6h        10.129.0.15     ocp-node3    
kubevirt-web-ui                     console-854d4585c8-hgdhv                       1/1       Running     0          6h        10.129.0.10     ocp-node3    
kubevirt-web-ui                     kubevirt-web-ui-operator-6b4574bb95-bmsw7      1/1       Running     0          6h        10.130.0.11     ocp-node2    
kubevirt                            kubevirt-cpu-node-labeller-fvx9n               1/1       Running     0          6h        10.128.0.29     ocp-master   
kubevirt                            kubevirt-cpu-node-labeller-jr858               1/1       Running     0          6h        10.131.0.13     ocp-node1    
kubevirt                            kubevirt-cpu-node-labeller-tgq5g               1/1       Running     0          6h        10.129.0.14     ocp-node3    
kubevirt                            kubevirt-cpu-node-labeller-xqpbl               1/1       Running     0          6h        10.130.0.13     ocp-node2    
kubevirt                            virt-api-865b95d544-hg58l                      1/1       Running     0          6h        10.129.0.8      ocp-node3    
kubevirt                            virt-api-865b95d544-jrkxh                      1/1       Running     0          6h        10.131.0.10     ocp-node1    
kubevirt                            virt-controller-5c89d4978d-q79lh               1/1       Running     0          6h        10.130.0.8      ocp-node2    
kubevirt                            virt-controller-5c89d4978d-t58l7               1/1       Running     0          6h        10.130.0.10     ocp-node2    
kubevirt                            virt-handler-gblbk                             1/1       Running     0          6h        10.128.0.28     ocp-master   
kubevirt                            virt-handler-jnwx6                             1/1       Running     0          6h        10.130.0.9      ocp-node2    
kubevirt                            virt-handler-r94fb                             1/1       Running     0          6h        10.129.0.9      ocp-node3    
kubevirt                            virt-handler-z7775                             1/1       Running     0          6h        10.131.0.11     ocp-node1    
kubevirt                            virt-operator-68984b585c-265bq                 1/1       Running     0          6h        10.129.0.7      ocp-node3    

Now that CNV is up running lets pull down a Fedora 29 image and upload it into a PVC of the default storageclass which of course is Rook/Ceph:

[origin@ocp-master ~]$ curl -L -o /home/origin/f29.qcow2 http://ftp.usf.edu/pub/fedora/linux/releases/29/Cloud/x86_64/images/Fedora-Cloud-Base-29-1.2.x86_64.qcow2
[origin@ocp-master ~]$ virtctl image-upload --pvc-name=f29vm --pvc-size=5Gi --image-path=/home/origin/f29.qcow2 --uploadproxy-url=https://`oc describe route cdi-uploadproxy-route|grep Endpoints|cut -f2` --insecure

We can execute the following to see that the PVC has been created:

[origin@ocp-master ~]$ oc get pvc
NAME      STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
f29vm     Bound     pvc-4815df9e-4987-11e9-a732-525400767d62   5Gi        RWO            rook-ceph-block   6h

Besides the PVC we will also need a virtual machine configuration yaml file.  The one below is an example that will be used in this demonstration:

apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
  creationTimestamp: null
  labels:
    kubevirt-vm: f29vm
  name: f29vm
spec:
  running: true
  template:
    metadata:
      creationTimestamp: null
      labels:
        kubevirt.io/domain: f29vm
    spec:
      domain:
        cpu:
          cores: 2
        devices:
          disks:
          - disk:
              bus: virtio
            name: osdisk
            volumeName: osdisk
          - disk:
              bus: virtio
            name: cloudinitdisk
            volumeName: cloudinitvolume
          interfaces:
          - name: default
            bridge: {}
        resources:
          requests:
            memory: 1024M
      terminationGracePeriodSeconds: 0
      volumes:
      - name: osdisk
        persistentVolumeClaim:
          claimName: f29vm
      - name: cloudinitdisk
        cloudInitNoCloud:
          userData: |-
            #cloud-config
            password: ${PASSWORD}
            disable_root: false
            chpasswd: { expire: False }
            ssh_authorized_keys:
            - "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUs1KbLraX74mBM/ksoGwbsEejfpCVeMzbW7JLJjGXF8G1jyVAE3T0Uf5mO8nbNOfkjAjw24lxSsEScF2wslBzA5MIm+GB6Z+ZzR55FcRlZeouGVrfLmb67mYc2c/F/mq35TruHdRk2G5Y0+6cf8cfDs414+yiVA0heHQvWNfO7kb1z9kIOhyD6OOwdNT5jK/1O0+p6SdP+pEal51BsEf6GRGYLWc9SLIEcqtjoprnundr5UPvmC1l/pkqFQigMehwhthrdXC4GseWiyj9CnBkccxQCKvHjzko/wqsWGQLwDG3pBsHhthvbY0G5+VPB9a8YV58WJhC6nHpUTDA8jpB origin@ocp-master"
      networks:
      - name: default
        pod: {}

At this point we have all the necessary components to launch our containerized virtual machine instance.   The following command does the creation using the yaml file we created in the previous step:

[origin@ocp-master ~]$ oc create -f /home/origin/f29vm.yaml

There are multiple ways to validate the virtual machine has been instantiated.   I like to do the following to confirm the instance is running and has an IP address:

[origin@ocp-master ~]$ oc get vms
NAME      AGE       RUNNING   VOLUME
f29vm     6h        true      
[origin@ocp-master ~]$ oc get vmi
NAME      AGE       PHASE     IP            NODENAME
f29vm     6h        Running   10.129.0.15   ocp-node3

One final step you can do is actually log into the instance assuming a key was set in the yaml file:

[origin@ocp-master ~]$ ssh -i /home/origin/.ssh/id_rsa -o "StrictHostKeyChecking=no" fedora@10.129.0.15
[fedora@f29vm ~]$ cat /etc/fedora-release
Fedora release 29 (Twenty Nine)

Hopefully this demonstrated how easy it is to get OpenShift, Rook and CNV up and running and how one can then leverage the storage of Rook to provide a backend for the virtual instance that gets spun up in CNV.   What is awesome is that I have taken the steps above and put them into a DCI job where I can automatically rerun the deployment using newer version of the code base for testing.   If you are not familiar with DCI I will leave with this tease link to DCI: https://doc.distributed-ci.io/

Rook & Ceph on Kubernetes

In a previous article I wrote about using Rook to deploy a Ceph storage cluster within Minikube (link below). The original post described what Rook can provide and demonstrated the ease of quickly setting up an all in one Ceph cluster. However I wanted explore Rook further in a multi-node configuration and how it integrates with applications in Kubernetes.

First I needed to set up a base Kubernetes environment which consisted of 1 master and 3 worker nodes. I used the following steps on all nodes to prepare them for Kubernetes: add hostname to host files, disable Selinux and swap, enable br_netfilter, install supporting utilities, enable Kubernetes repo, install docker, install Kubernetes binaries and enable/disable relevant services.

# echo "10.0.0.81   kube-master" >> /etc/hosts
# echo "10.0.0.82   kube-node1" >> /etc/hosts
# echo "10.0.0.83   kube-node2" >> /etc/hosts
# echo "10.0.0.84   kube-node3" >> /etc/hosts
# setenforce 0
# sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux
# swapoff -a
# sed -i.bak -r 's/(.+ swap .+)/#\1/' /etc/fstab
# modprobe br_netfilter
# echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables
# echo 'br_netfilter' > /etc/modules-load.d/netfilter.conf
# echo net.bridge.bridge-nf-call-iptables=1 >> /etc/sysctl.d/10-bridge-nf-call-iptables.conf
# dnf install -y yum-utils device-mapper-persistent-data lvm2
# dnf install docker
# cat > /etc/yum.repos.d/kubernetes.repo < [kubernetes]
  > name=Kubernetes
  > baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
  > enabled=1
  > gpgcheck=1
  > repo_gpgcheck=1
  > gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
  >         https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
  > EOF
# dnf install -y kubelet kubeadm kubectl
# systemctl enable docker ; systemctl start docker ; systemctl enable kubelet ; systemctl start kubelet ; systemctl stop firewalld ; systemctl disable firewalld

Once the prerequisites are met on each node lets initialize the cluster on the master node:

# kubeadm init --apiserver-advertise-address=10.0.0.81 --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.13.2
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [kube-master localhost] and IPs [10.0.0.81 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [kube-master localhost] and IPs [10.0.0.81 127.0.0.1 ::1]
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kube-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.0.0.81]
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 19.511836 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "kube-master" as an annotation
[mark-control-plane] Marking the node kube-master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node kube-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: etmucm.238nrw6a48yu0njb
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 10.0.0.81:6443 --token etmucm.238nrw6a48yu0njb --discovery-token-ca-cert-hash sha256:963d6d9d31f2db9debfaa600ef802d05c448f7dc9e9cb92aec268cf2a8cfee7b

After the master is up and running you can join the remaining nodes using the following command which was presented in the output when you initialized the master:

# kubeadm join 10.0.0.81:6443 --token etmucm.238nrw6a48yu0njb --discovery-token-ca-cert-hash sha256:963d6d9d31f2db9debfaa600ef802d05c448f7dc9e9cb92aec268cf2a8cfee7b
[preflight] Running pre-flight checks
[discovery] Trying to connect to API Server "10.0.0.81:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.0.0.81:6443"
[discovery] Requesting info from "https://10.0.0.81:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "10.0.0.81:6443"
[discovery] Successfully established connection with API Server "10.0.0.81:6443"
[join] Reading configuration from the cluster...
[join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.13" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "kube-node1" as an annotation

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

I like to do some housekeeping once all my nodes are joined which includes enabling scheduling on master and labeling the worker nodes as such:

# kubectl taint node kube-master node-role.kubernetes.io/master:NoSchedule-
# kubectl label node kube-node1 node-role.kubernetes.io/worker=worker
# kubectl label node kube-node2 node-role.kubernetes.io/worker=worker
# kubectl label node kube-node3 node-role.kubernetes.io/worker=worker

Once you have joined the nodes you should have a cluster that looks like this:

# kubectl get nodes
NAME          STATUS   ROLES    AGE   VERSION
kube-master   Ready    master   19h   v1.13.2
kube-node1    Ready    worker   19h   v1.13.2
kube-node2    Ready    worker   19h   v1.13.2
kube-node3    Ready    worker   17h   v1.13.2

Next lets deploy Flannel for networking:

# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

And finally lets deploy Rook and Ceph cluster using the familiar steps from my previous article:

# git clone https://github.com/rook/rook.git
# cd ./rook/cluster/examples/kubernetes/ceph
# sed -i.bak s+/var/lib/rook+/data/rook+g cluster.yaml
# kubectl create -f operator.yaml
# kubectl create -f cluster.yaml
# kubectl create -f toolbox.yaml

Once all the containers have spun up you should have something that looks like the following:

# kubectl get pod -n rook-ceph -o wide
NAME                                      READY   STATUS      RESTARTS   AGE   IP            NODE          NOMINATED NODE   READINESS GATES
rook-ceph-mgr-a-8649f78d9b-txsfm          1/1     Running     1          19h   10.244.2.12   kube-node2               
rook-ceph-mon-a-598b7bd4cd-kpxnx          1/1     Running     0          19h   10.244.0.3    kube-master              
rook-ceph-mon-c-759b8984f5-ggzjb          1/1     Running     1          19h   10.244.2.15   kube-node2               
rook-ceph-mon-d-77d55dcddf-mwnf8          1/1     Running     0          16h   10.244.3.3    kube-node3               
rook-ceph-osd-0-77b448bbcc-mdhsw          1/1     Running     1          19h   10.244.2.14   kube-node2               
rook-ceph-osd-1-65db4b7c5d-hgfcj          1/1     Running     0          16h   10.244.1.8    kube-node1               
rook-ceph-osd-2-5b475cb56c-x5w6n          1/1     Running     0          19h   10.244.0.5    kube-master              
rook-ceph-osd-3-657789944d-swjxd          1/1     Running     0          16h   10.244.3.6    kube-node3               
rook-ceph-osd-prepare-kube-master-tlhxf   0/2     Completed   0          16h   10.244.0.6    kube-master              
rook-ceph-osd-prepare-kube-node1-lgtrf    0/2     Completed   0          16h   10.244.1.12   kube-node1               
rook-ceph-osd-prepare-kube-node2-5tbt6    0/2     Completed   0          16h   10.244.2.17   kube-node2               
rook-ceph-osd-prepare-kube-node3-rrp4z    0/2     Completed   0          16h   10.244.3.5    kube-node3               
rook-ceph-tools-76c7d559b6-7kprh          1/1     Running     0          16h   10.0.0.84     kube-node3               

And of course we can validate the Ceph cluster is up and healthy via the toolbox container as well:

# kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash

# ceph status
  cluster:
    id:     4be6e204-3d82-4cc4-9ea4-57f0e71f99c5
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum d,a,c
    mgr: a(active)
    osd: 4 osds: 4 up, 4 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0  objects, 0 B
    usage:   17 GiB used, 123 GiB / 140 GiB avail
    pgs:     
 
# ceph osd tree
ID CLASS WEIGHT  TYPE NAME            STATUS REWEIGHT PRI-AFF
-1       0.13715 root default                                 
-4       0.03429     host kube-master                         
 2   hdd 0.03429         osd.2            up  1.00000 1.00000
-3       0.03429     host kube-node1                          
 1   hdd 0.03429         osd.1            up  1.00000 1.00000
-2       0.03429     host kube-node2                          
 0   hdd 0.03429         osd.0            up  1.00000 1.00000
-9       0.03429     host kube-node3                          
 3   hdd 0.03429         osd.3            up  1.00000 1.00000

Everything we have done up to this point has been very similar to what I did in the previous article with Minikube except instead of a single node we have a multiple node configuration. Now lets take it a step further and get an application to use our Ceph storage cluster.

The first step in Kubernetes will be to created a storageclass.yaml that uses Ceph.  Populate the storageclass.yaml with the following:

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicapool
  namespace: rook-ceph
spec:
  failureDomain: host
  replicated:
    size: 3
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: rook-ceph-block
provisioner: ceph.rook.io/block
parameters:
  blockPool: replicapool
  # The value of "clusterNamespace" MUST be the same as the one in which your rook cluster exist
  clusterNamespace: rook-ceph
  # Specify the filesystem type of the volume. If not specified, it will use `ext4`.
  fstype: xfs
# Optional, default reclaimPolicy is "Delete". Other options are: "Retain", "Recycle" as documented in https://kubernetes.io/docs/concepts/storage/storage-classes/

Next lets create the storage class using the yaml we created and set it to default:

# kubectl create -f storageclass.yaml
cephblockpool.ceph.rook.io/replicapool created
storageclass.storage.k8s.io/rook-ceph-block created

# kubectl get storageclass
NAME              PROVISIONER          AGE
rook-ceph-block   ceph.rook.io/block   61s

# kubectl patch storageclass rook-ceph-block -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
storageclass.storage.k8s.io/rook-ceph-block patched

# kubectl get storageclass
NAME                        PROVISIONER          AGE
rook-ceph-block (default)   ceph.rook.io/block   3m30s

Now that we have a storageclass that uses Ceph as the backend we now need an application to consume the storageclass. Thankfully the Rook git repo includes a couple of examples: Wordpress and MySQL. Lets go ahead and create those apps doing the following:

# cd ./rook/cluster/examples/kubernetes

# kubectl create -f mysql.yaml
service/wordpress-mysql created
persistentvolumeclaim/mysql-pv-claim created
deployment.apps/wordpress-mysql created

# kubectl create -f wordpress.yaml
service/wordpress created
persistentvolumeclaim/wp-pv-claim created
deployment.extensions/wordpress created

We can confirm our two applications are running by the following:

# kubectl get pods -n default -o wide
NAME                               READY   STATUS    RESTARTS   AGE     IP            NODE         NOMINATED NODE   READINESS GATES
wordpress-7b6c4c79bb-7b4dq         1/1     Running   0          68s     10.244.1.14   kube-node1              
wordpress-mysql-6887bf844f-2m4h4   1/1     Running   0          2m47s   10.244.1.13   kube-node1              

Now lets confirm if they are actually using our Ceph storageclass:

# kubectl get pvc

NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
mysql-pv-claim   Bound    pvc-0c0be0ec-2317-11e9-a462-5254003ede95   20Gi       RWO            rook-ceph-block   3m36s
wp-pv-claim      Bound    pvc-46b4b266-2317-11e9-a462-5254003ede95   20Gi       RWO            rook-ceph-block   118s

And lets also confirm Wordpress is up and running from a user perspective. Note in this example we do not have an external IP and can only access the service via the cluster IP:

# kubectl get svc wordpress
NAME        TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
wordpress   LoadBalancer   10.104.120.47        80:32592/TCP 
  19m
# curl -v http://10.104.120.47
* About to connect() to 10.104.120.47 port 80 (#0)
*   Trying 10.104.120.47...
* Connected to 10.104.120.47 (10.104.120.47) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.104.120.47
> Accept: */*
>
< HTTP/1.1 302 Found
< Date: Mon, 28 Jan 2019 16:30:16 GMT
< Server: Apache/2.4.10 (Debian)
< X-Powered-By: PHP/5.6.28
< Expires: Wed, 11 Jan 1984 05:00:00 GMT
< Cache-Control: no-cache, must-revalidate, max-age=0
< Location: http://10.104.120.47/wp-admin/install.php
< Content-Length: 0
< Content-Type: text/html; charset=UTF-8
<
* Connection #0 to host 10.104.120.47 left intact

We can see from the above output we do connect but get a 302 code since Wordpress really needs to be configured first. But it does confirm our applications are up and using the Ceph storageclass.

To clean up the previous exercise lets do the following:

# kubectl delete -f wordpress.yaml
service "wordpress" deleted
persistentvolumeclaim "wp-pv-claim" deleted
deployment.extensions "wordpress" deleted

# kubectl delete -f mysql.yaml
service "wordpress-mysql" deleted
persistentvolumeclaim "mysql-pv-claim" deleted
deployment.apps "wordpress-mysql" deleted

# kubectl delete -n rook-ceph cephblockpools.ceph.rook.io replicapool
cephblockpool.ceph.rook.io "replicapool" deleted

# kubectl delete storageclass rook-ceph-block
storageclass.storage.k8s.io "rook-ceph-block" deleted

The above example was just a simple demonstration of the capabilities Rook/Ceph bring to Kubernetes from a block storage perspective. But leaves one wondering what other possibilities there might be.

Further Reading:

Rook: https://github.com/rook/rook
Kubernetes: https://kubernetes.io/
Previous Article:  https://www.linkedin.com/pulse/deploying-ceph-rook-benjamin-schmaus/