Recently a lot of customers have been asking about how to configure storage on a Single Node OpenShift (SNO) deployment. When dealing with a compact or multi-node OpenShift deployment I myself have always relied on using OpenShift Data Foundation (ODF) as the underpinning of my storage requirements after all it provides the ability to do block, object and file all from the same deployment. However in a SNO deployment ODF does not seem to be an option due to the way the operator has been designed. However there is a way to at least get some resemblance to ODF without a lot of hassle in a SNO environment. The following blog demonstrates a non-supported way on how I go about getting the dynamic block storage I need in my SNO cluster using Rook.
Before we begin lets quickly go over the environment of this SNO cluster. As with any SNO cluster it is a single node acting as both the control plane and worker node. This particular deployment was based on OpenShift 4.10.11 and deployed using the Assisted Installer at cloud.redhat.com.
$ oc get nodes NAME STATUS ROLES AGE VERSION master-0.sno3.schmaustech.com Ready master,worker 3h v1.23.5+9ce5071
Inside the node via the debug pod we can see that we have an extra 160GB disk available to use toward our Rook deployment:
$ oc debug node/master-0.sno3.schmaustech.com Starting pod/master-0sno3schmaustechcom-debug ... To use host binaries, run `chroot /host` Pod IP: 192.168.0.206 If you don't see a command prompt, try pressing enter. sh-4.4# chroot /host sh-4.4# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 120G 0 disk |-sda1 8:1 0 1M 0 part |-sda2 8:2 0 127M 0 part |-sda3 8:3 0 384M 0 part /boot `-sda4 8:4 0 119.5G 0 part /sysroot sdb 8:16 0 160G 0 disk sr0 11:0 1 999M 0 rom more
Now that we have provided the environment background lets go ahead and start to configure Rook on the SNO node. The first step will be to configure the custom resource definitions that Rook requires before the operator and the Ceph cluster can be deployed. Since I have not changed the crds.yaml file from its defaults we can consume it directly from the Rook Github repository and apply it to our SNO node:
$ oc create -f https://raw.githubusercontent.com/rook/rook/master/deploy/examples/crds.yaml customresourcedefinition.apiextensions.k8s.io/cephblockpoolradosnamespaces.ceph.rook.io created customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created customresourcedefinition.apiextensions.k8s.io/cephbuckettopics.ceph.rook.io created customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created customresourcedefinition.apiextensions.k8s.io/cephfilesystemsubvolumegroups.ceph.rook.io created customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created
With the custom resource definitions applied we can move onto adding in the common resources that are necessary to start the operator and the Ceph cluster. Again since I am not changing anything in the defaults from the Rook Github repository we can apply directly from the source to the SNO node:
$ oc create -f https://raw.githubusercontent.com/rook/rook/master/deploy/examples/common.yaml namespace/rook-ceph created clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created clusterrole.rbac.authorization.k8s.io/psp:rook created clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created clusterrole.rbac.authorization.k8s.io/rook-ceph-global created clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created clusterrole.rbac.authorization.k8s.io/rook-ceph-system created clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ podsecuritypolicy.policy/00-rook-privileged created role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created role.rbac.authorization.k8s.io/rbd-csi-nodeplugin created role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created role.rbac.authorization.k8s.io/rook-ceph-mgr created role.rbac.authorization.k8s.io/rook-ceph-osd created role.rbac.authorization.k8s.io/rook-ceph-purge-osd created role.rbac.authorization.k8s.io/rook-ceph-rgw created role.rbac.authorization.k8s.io/rook-ceph-system created rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created rolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-role-cfg created rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd-psp created rolebinding.rbac.authorization.k8s.io/rook-ceph-rgw created rolebinding.rbac.authorization.k8s.io/rook-ceph-rgw-psp created rolebinding.rbac.authorization.k8s.io/rook-ceph-system created serviceaccount/rook-ceph-cmd-reporter created serviceaccount/rook-ceph-mgr created serviceaccount/rook-ceph-osd created serviceaccount/rook-ceph-purge-osd created serviceaccount/rook-ceph-rgw created serviceaccount/rook-ceph-system created serviceaccount/rook-csi-cephfs-plugin-sa created serviceaccount/rook-csi-cephfs-provisioner-sa created serviceaccount/rook-csi-rbd-plugin-sa created serviceaccount/rook-csi-rbd-provisioner-sa created
Next we need to create the Rook operator.yaml file which will be used to configure the Rook operator:
$ cat << EOF > ~/operator.yaml
kind: SecurityContextConstraints
apiVersion: security.openshift.io/v1
metadata:
name: rook-ceph
allowPrivilegedContainer: true
allowHostDirVolumePlugin: true
allowHostPID: false
allowHostNetwork: false
allowHostPorts: false
priority:
allowedCapabilities: ["MKNOD"]
allowHostIPC: true
readOnlyRootFilesystem: false
requiredDropCapabilities: []
defaultAddCapabilities: []
runAsUser:
type: RunAsAny
seLinuxContext:
type: MustRunAs
fsGroup:
type: MustRunAs
supplementalGroups:
type: RunAsAny
volumes:
- configMap
- downwardAPI
- emptyDir
- hostPath
- persistentVolumeClaim
- projected
- secret
users:
- system:serviceaccount:rook-ceph:rook-ceph-system
- system:serviceaccount:rook-ceph:default
- system:serviceaccount:rook-ceph:rook-ceph-mgr
- system:serviceaccount:rook-ceph:rook-ceph-osd
- system:serviceaccount:rook-ceph:rook-ceph-rgw
---
kind: SecurityContextConstraints
apiVersion: security.openshift.io/v1
metadata:
name: rook-ceph-csi
allowPrivilegedContainer: true
allowHostNetwork: true
allowHostDirVolumePlugin: true
priority:
allowedCapabilities: ["SYS_ADMIN"]
allowHostPorts: true
allowHostPID: true
allowHostIPC: true
readOnlyRootFilesystem: false
runAsUser:
type: RunAsAny
seLinuxContext:
type: RunAsAny
fsGroup:
type: RunAsAny
supplementalGroups:
type: RunAsAny
volumes:
- configMap
- projected
- emptyDir
- hostPath
users:
- system:serviceaccount:rook-ceph:rook-csi-rbd-plugin-sa
- system:serviceaccount:rook-ceph:rook-csi-rbd-provisioner-sa
- system:serviceaccount:rook-ceph:rook-csi-cephfs-plugin-sa
- system:serviceaccount:rook-ceph:rook-csi-cephfs-provisioner-sa
- system:serviceaccount:rook-ceph:rook-csi-nfs-plugin-sa
- system:serviceaccount:rook-ceph:rook-csi-nfs-provisioner-sa
---
kind: ConfigMap
apiVersion: v1
metadata:
name: rook-ceph-operator-config
namespace: rook-ceph
data:
ROOK_LOG_LEVEL: "INFO"
ROOK_CSI_ENABLE_CEPHFS: "true"
ROOK_CSI_ENABLE_RBD: "true"
ROOK_CSI_ENABLE_NFS: "false"
ROOK_CSI_ENABLE_GRPC_METRICS: "false"
CSI_ENABLE_ENCRYPTION: "false"
CSI_PROVISIONER_REPLICAS: "2"
CSI_ENABLE_CEPHFS_SNAPSHOTTER: "true"
CSI_ENABLE_RBD_SNAPSHOTTER: "true"
CSI_FORCE_CEPHFS_KERNEL_CLIENT: "true"
CSI_RBD_FSGROUPPOLICY: "ReadWriteOnceWithFSType"
CSI_CEPHFS_FSGROUPPOLICY: "ReadWriteOnceWithFSType"
CSI_NFS_FSGROUPPOLICY: "ReadWriteOnceWithFSType"
ROOK_CSI_ALLOW_UNSUPPORTED_VERSION: "false"
CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT: "false"
CSI_PLUGIN_PRIORITY_CLASSNAME: "system-node-critical"
CSI_PROVISIONER_PRIORITY_CLASSNAME: "system-cluster-critical"
ROOK_OBC_WATCH_OPERATOR_NAMESPACE: "true"
ROOK_ENABLE_DISCOVERY_DAEMON: "false"
CSI_ENABLE_VOLUME_REPLICATION: "false"
ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS: "15"
CSI_ENABLE_CSIADDONS: "false"
CSI_GRPC_TIMEOUT_SECONDS: "150"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: rook-ceph-operator
namespace: rook-ceph
labels:
operator: rook
storage-backend: ceph
app.kubernetes.io/name: rook-ceph
app.kubernetes.io/instance: rook-ceph
app.kubernetes.io/component: rook-ceph-operator
app.kubernetes.io/part-of: rook-ceph-operator
spec:
selector:
matchLabels:
app: rook-ceph-operator
replicas: 1
template:
metadata:
labels:
app: rook-ceph-operator
spec:
serviceAccountName: rook-ceph-system
containers:
- name: rook-ceph-operator
image: rook/ceph:v1.9.3
args: ["ceph", "operator"]
securityContext:
runAsNonRoot: true
runAsUser: 2016
runAsGroup: 2016
volumeMounts:
- mountPath: /var/lib/rook
name: rook-config
- mountPath: /etc/ceph
name: default-config-dir
- mountPath: /etc/webhook
name: webhook-cert
ports:
- containerPort: 9443
name: https-webhook
protocol: TCP
env:
- name: ROOK_CURRENT_NAMESPACE_ONLY
value: "false"
- name: ROOK_DISCOVER_DEVICES_INTERVAL
value: "60m"
- name: ROOK_HOSTPATH_REQUIRES_PRIVILEGED
value: "true"
- name: ROOK_ENABLE_SELINUX_RELABELING
value: "true"
- name: ROOK_ENABLE_FSGROUP
value: "true"
- name: ROOK_DISABLE_DEVICE_HOTPLUG
value: "false"
- name: DISCOVER_DAEMON_UDEV_BLACKLIST
value: "(?i)dm-[0-9]+,(?i)rbd[0-9]+,(?i)nbd[0-9]+"
- name: ROOK_ENABLE_MACHINE_DISRUPTION_BUDGET
value: "false"
- name: ROOK_UNREACHABLE_NODE_TOLERATION_SECONDS
value: "5"
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumes:
- name: rook-config
emptyDir: {}
- name: default-config-dir
emptyDir: {}
- name: webhook-cert
emptyDir: {}
EOF
With the Rook operator.yaml saved we can now apply it to the SNO node and after a few minutes validate that the Rook operator is running:
$ oc create -f operator.yaml securitycontextconstraints.security.openshift.io/rook-ceph created securitycontextconstraints.security.openshift.io/rook-ceph-csi created configmap/rook-ceph-operator-config created deployment.apps/rook-ceph-operator created $ oc get pods -n rook-ceph NAME READY STATUS RESTARTS AGE rook-ceph-operator-84bf68d9bd-lv9l9 1/1 Running 0 1m
Finally we get the heart of this configuration which is the cluster.yaml file. In this file we need to make some modification since we only have a single node for the Ceph deployment via Rook. Here are things I modified from the default:
osd_pool_default_size needs to be 1
mon count needs to be 1 and allowMultiplePerNode needs to be true
mgr count needs to be 1 and allowMultiplePerNode needs to be true
storage device needs to be set to extra disk available (in my case sdb)
osdsPerDevice needs to be 1
managePodBudgets and manageMachineDisruptionBudgets both set to false
$ cat << EOF > ~/cluster.yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: rook-config-override
namespace: rook-ceph
data:
config: |
[global]
osd_pool_default_size = 1
---
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
name: rook-ceph
namespace: rook-ceph
spec:
cephVersion:
image: quay.io/ceph/ceph:v16.2.7
allowUnsupported: false
dataDirHostPath: /var/lib/rook
skipUpgradeChecks: false
continueUpgradeAfterChecksEvenIfNotHealthy: false
waitTimeoutForHealthyOSDInMinutes: 10
mon:
count: 1
allowMultiplePerNode: true
mgr:
count: 1
allowMultiplePerNode: true
modules:
- name: pg_autoscaler
enabled: true
dashboard:
enabled: true
ssl: true
monitoring:
enabled: false
network:
connections:
encryption:
enabled: false
compression:
enabled: false
crashCollector:
disable: false
cleanupPolicy:
confirmation: ""
sanitizeDisks:
method: quick
dataSource: zero
iteration: 1
allowUninstallWithVolumes: false
annotations:
labels:
resources:
removeOSDsIfOutAndSafeToRemove: false
priorityClassNames:
mon: system-node-critical
osd: system-node-critical
mgr: system-cluster-critical
storage:
useAllNodes: true
useAllDevices: false
devices:
- name: "sdb"
config:
osdsPerDevice: "1"
onlyApplyOSDPlacement: false
disruptionManagement:
managePodBudgets: false
osdMaintenanceTimeout: 30
pgHealthCheckTimeout: 0
manageMachineDisruptionBudgets: false
machineDisruptionBudgetNamespace: openshift-machine-api
healthCheck:
daemonHealth:
mon:
disabled: false
interval: 45s
osd:
disabled: false
interval: 60s
status:
disabled: false
interval: 60s
livenessProbe:
mon:
disabled: false
mgr:
disabled: false
osd:
disabled: false
startupProbe:
mon:
disabled: false
mgr:
disabled: false
osd:
disabled: false
EOF
Now with the cluster.yaml saved we can apply it to the cluster and let the Rook operator do the work of creating the Ceph cluster on our SNO node:
$ oc create -f cluster.yaml configmap/rook-config-override created cephcluster.ceph.rook.io/rook-ceph created
After a few minutes, depending on the speed of the SNO deployment, we can validate that the Ceph cluster is up and deployed on our SNO node:
$ oc get pods -n rook-ceph NAME READY STATUS RESTARTS AGE csi-cephfsplugin-provisioner-7577bb4d59-kxmq8 6/6 Running 0 118s csi-cephfsplugin-x2njd 3/3 Running 0 118s csi-rbdplugin-provisioner-847b498845-7z5qc 6/6 Running 0 119s csi-rbdplugin-tlw5d 3/3 Running 0 119s rook-ceph-crashcollector-master-0.sno3.schmaustech.com-858mmbx2 1/1 Running 0 48s rook-ceph-mgr-a-57fbb7fb47-9rjl5 1/1 Running 0 81s rook-ceph-mon-a-d94d79bb5-l6f8p 1/1 Running 0 110s rook-ceph-operator-84bf68d9bd-qkj6k 1/1 Running 0 17m rook-ceph-osd-0-6c98c84f66-96l5q 1/1 Running 0 48s rook-ceph-osd-prepare-master-0.sno3.schmaustech.com-l5s8t 0/1 Completed 0 60s
We can see from the above output that the pods for both the single mon and single osd are up and running along with the additional services for our single node Ceph cluster. We can further validate that the Ceph cluster is up and running by deploying a Ceph toolbox pod. For that we will just use the toolbox.yaml from the Rook Github repository. Once we create the pod we can validate it is running by filtering for the rook-ceph-tools pod in the rook-ceph namespace:
$ oc create -f https://raw.githubusercontent.com/rook/rook/master/deploy/examples/toolbox.yaml deployment.apps/rook-ceph-tools created $ oc get pods -n rook-ceph| grep rook-ceph-tools rook-ceph-tools-d6d7c985c-6zwc7 1/1 Running 0 54s
Now lets use the running toolbox to check on the Ceph cluster by issuing an exec command to it and passing in a ceph status:
$ oc -n rook-ceph exec -it rook-ceph-tools-d6d7c985c-6zwc7 -- ceph status
cluster:
id: c54ad01e-e9f8-48c9-806b-a7d4748eb977
health: HEALTH_OK
services:
mon: 1 daemons, quorum a (age 7m)
mgr: a(active, since 5m)
osd: 1 osds: 1 up (since 5m), 1 in (since 6m)
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 4.8 MiB used, 160 GiB / 160 GiB avail
pgs:
Sure enough our Ceph cluster is up and running and in a healthy state. Lets move on now to confirm we can consume storage from it. To do that we need to setup a storageclass configuration like the example below which will create a Ceph RBD block storageclass:
$ cat << EOF > ~/storageclass.yaml
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
name: replicapool
namespace: rook-ceph
spec:
failureDomain: host
replicated:
size: 1
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: rook-ceph-block
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
clusterID: rook-ceph
pool: replicapool
imageFormat: "2"
imageFeatures: layering
csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Delete
allowVolumeExpansion: true
EOF
Once we have saved the storageclass.yaml file lets go ahead and apply it to the SNO node and then check that the storageclass was created:
$ oc create -f storageclass.yaml cephblockpool.ceph.rook.io/replicapool created storageclass.storage.k8s.io/rook-ceph-block created $ oc get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE rook-ceph-block rook-ceph.rbd.csi.ceph.com Delete Immediate true 3s oc get
Now that we have a storageclass created I like to do one more thing to ensure any outstanding persistent volume claims get fulfilled by the storageclass automatically. To do this I will patch the storageclass to be the default:
$ oc patch storageclass rook-ceph-block -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
storageclass.storage.k8s.io/rook-ceph-block patched
$ oc get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
rook-ceph-block (default) rook-ceph.rbd.csi.ceph.com Delete Immediate true 8m41s
At this point everything is configured to consume storage. What I did in the example below was to kick off an installation of Red Hat Advanced Cluster Management on my SNO node because I knew it would need a PV. Once it completed installation I confirmed by looking at the persistent volumes that indeed one had been created from our storageclass which gets its storage from the Ceph cluster:
$ oc get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-791551c6-cdc4-4c9e-9692-c9622dbef4e8 10Gi RWO Delete Bound open-cluster-management/search-redisgraph-pvc-0 rook-ceph-block 45s
Hopefully this was a helpful blog in providing a dynamic almost ODF like experience for a SNO node deployment!
