Tuesday, May 17, 2022

Rook on Singe Node OpenShift


Recently a lot of customers have been asking about how to configure storage on a Single Node OpenShift (SNO) deployment.   When dealing with a compact or multi-node OpenShift deployment I myself have always relied on using OpenShift Data Foundation (ODF) as the underpinning of my storage requirements after all it provides the ability to do block, object and file all from the same deployment.  However in a SNO deployment ODF does not seem to be an option due to the way the operator has been designed.  However there is a way to at least get some resemblance to ODF without a lot of hassle in a SNO environment.  The following blog demonstrates a non-supported way on how I go about getting the dynamic block storage I need in my SNO cluster using Rook.

Before we begin lets quickly go over the environment of this SNO cluster.   As with any SNO cluster it is a single node acting as both the control plane and worker node.  This particular deployment was based on OpenShift 4.10.11 and deployed using the Assisted Installer at cloud.redhat.com

$ oc get nodes
NAME                            STATUS   ROLES           AGE   VERSION
master-0.sno3.schmaustech.com   Ready    master,worker   3h    v1.23.5+9ce5071

Inside the node via the debug pod we can see that we have an extra 160GB disk available to use toward our Rook deployment:

$ oc debug node/master-0.sno3.schmaustech.com
Starting pod/master-0sno3schmaustechcom-debug ...
To use host binaries, run `chroot /host`
Pod IP: 192.168.0.206
If you don't see a command prompt, try pressing enter.
sh-4.4# chroot /host
sh-4.4# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0   120G  0 disk 
|-sda1   8:1    0     1M  0 part 
|-sda2   8:2    0   127M  0 part 
|-sda3   8:3    0   384M  0 part /boot
`-sda4   8:4    0 119.5G  0 part /sysroot
sdb      8:16   0   160G  0 disk 
sr0     11:0    1   999M  0 rom more

Now that we have provided the environment background lets go ahead and start to configure Rook on the SNO node.   The first step will be to configure the custom resource definitions that Rook requires before the operator and the Ceph cluster can be deployed.   Since I have not changed the crds.yaml file from its defaults we can consume it directly from the Rook Github repository and apply it to our SNO node:

$ oc create -f https://raw.githubusercontent.com/rook/rook/master/deploy/examples/crds.yaml 
customresourcedefinition.apiextensions.k8s.io/cephblockpoolradosnamespaces.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephbuckettopics.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephfilesystemsubvolumegroups.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created
customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created

With the custom resource definitions applied we can move onto adding in the common resources that are necessary to start the operator and the Ceph cluster.  Again since I am not changing anything in the defaults from the Rook Github repository we can apply directly from the source to the SNO node:
  
$ oc create -f https://raw.githubusercontent.com/rook/rook/master/deploy/examples/common.yaml 
namespace/rook-ceph created
clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created
clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created
clusterrole.rbac.authorization.k8s.io/psp:rook created
clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created
clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created
clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created
clusterrole.rbac.authorization.k8s.io/rook-ceph-global created
clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created
clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created
clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created
clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created
clusterrole.rbac.authorization.k8s.io/rook-ceph-system created
clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created
clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created
clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created
clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created
clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created
clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created
clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created
clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created
clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created
clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system-psp created
clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-plugin-sa-psp created
clusterrolebinding.rbac.authorization.k8s.io/rook-csi-cephfs-provisioner-sa-psp created
clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-plugin-sa-psp created
clusterrolebinding.rbac.authorization.k8s.io/rook-csi-rbd-provisioner-sa-psp created
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/00-rook-privileged created
role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created
role.rbac.authorization.k8s.io/rbd-csi-nodeplugin created
role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created
role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created
role.rbac.authorization.k8s.io/rook-ceph-mgr created
role.rbac.authorization.k8s.io/rook-ceph-osd created
role.rbac.authorization.k8s.io/rook-ceph-purge-osd created
role.rbac.authorization.k8s.io/rook-ceph-rgw created
role.rbac.authorization.k8s.io/rook-ceph-system created
rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created
rolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-role-cfg created
rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created
rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created
rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created
rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter-psp created
rolebinding.rbac.authorization.k8s.io/rook-ceph-default-psp created
rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created
rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-psp created
rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created
rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created
rolebinding.rbac.authorization.k8s.io/rook-ceph-osd-psp created
rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created
rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd-psp created
rolebinding.rbac.authorization.k8s.io/rook-ceph-rgw created
rolebinding.rbac.authorization.k8s.io/rook-ceph-rgw-psp created
rolebinding.rbac.authorization.k8s.io/rook-ceph-system created
serviceaccount/rook-ceph-cmd-reporter created
serviceaccount/rook-ceph-mgr created
serviceaccount/rook-ceph-osd created
serviceaccount/rook-ceph-purge-osd created
serviceaccount/rook-ceph-rgw created
serviceaccount/rook-ceph-system created
serviceaccount/rook-csi-cephfs-plugin-sa created
serviceaccount/rook-csi-cephfs-provisioner-sa created
serviceaccount/rook-csi-rbd-plugin-sa created
serviceaccount/rook-csi-rbd-provisioner-sa created

Next we need to create the Rook operator.yaml file which will be used to configure the Rook operator:

$ cat << EOF > ~/operator.yaml
kind: SecurityContextConstraints
apiVersion: security.openshift.io/v1
metadata:
  name: rook-ceph
allowPrivilegedContainer: true
allowHostDirVolumePlugin: true
allowHostPID: false
allowHostNetwork: false
allowHostPorts: false
priority:
allowedCapabilities: ["MKNOD"]
allowHostIPC: true
readOnlyRootFilesystem: false
requiredDropCapabilities: []
defaultAddCapabilities: []
runAsUser:
  type: RunAsAny
seLinuxContext:
  type: MustRunAs
fsGroup:
  type: MustRunAs
supplementalGroups:
  type: RunAsAny
volumes:
  - configMap
  - downwardAPI
  - emptyDir
  - hostPath
  - persistentVolumeClaim
  - projected
  - secret
users:
  - system:serviceaccount:rook-ceph:rook-ceph-system 
  - system:serviceaccount:rook-ceph:default 
  - system:serviceaccount:rook-ceph:rook-ceph-mgr 
  - system:serviceaccount:rook-ceph:rook-ceph-osd 
  - system:serviceaccount:rook-ceph:rook-ceph-rgw 
---
kind: SecurityContextConstraints
apiVersion: security.openshift.io/v1
metadata:
  name: rook-ceph-csi
allowPrivilegedContainer: true
allowHostNetwork: true
allowHostDirVolumePlugin: true
priority:
allowedCapabilities: ["SYS_ADMIN"]
allowHostPorts: true
allowHostPID: true
allowHostIPC: true
readOnlyRootFilesystem: false
runAsUser:
  type: RunAsAny
seLinuxContext:
  type: RunAsAny
fsGroup:
  type: RunAsAny
supplementalGroups:
  type: RunAsAny
volumes:
  - configMap
  - projected
  - emptyDir
  - hostPath
users:
  - system:serviceaccount:rook-ceph:rook-csi-rbd-plugin-sa 
  - system:serviceaccount:rook-ceph:rook-csi-rbd-provisioner-sa 
  - system:serviceaccount:rook-ceph:rook-csi-cephfs-plugin-sa 
  - system:serviceaccount:rook-ceph:rook-csi-cephfs-provisioner-sa 
  - system:serviceaccount:rook-ceph:rook-csi-nfs-plugin-sa 
  - system:serviceaccount:rook-ceph:rook-csi-nfs-provisioner-sa 
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: rook-ceph-operator-config
  namespace: rook-ceph 
data:
  ROOK_LOG_LEVEL: "INFO"
  ROOK_CSI_ENABLE_CEPHFS: "true"
  ROOK_CSI_ENABLE_RBD: "true"
  ROOK_CSI_ENABLE_NFS: "false"
  ROOK_CSI_ENABLE_GRPC_METRICS: "false"
  CSI_ENABLE_ENCRYPTION: "false"
  CSI_PROVISIONER_REPLICAS: "2"
  CSI_ENABLE_CEPHFS_SNAPSHOTTER: "true"
  CSI_ENABLE_RBD_SNAPSHOTTER: "true"
  CSI_FORCE_CEPHFS_KERNEL_CLIENT: "true"
  CSI_RBD_FSGROUPPOLICY: "ReadWriteOnceWithFSType"
  CSI_CEPHFS_FSGROUPPOLICY: "ReadWriteOnceWithFSType"
  CSI_NFS_FSGROUPPOLICY: "ReadWriteOnceWithFSType"
  ROOK_CSI_ALLOW_UNSUPPORTED_VERSION: "false"
  CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT: "false"
  CSI_PLUGIN_PRIORITY_CLASSNAME: "system-node-critical"
  CSI_PROVISIONER_PRIORITY_CLASSNAME: "system-cluster-critical"
  ROOK_OBC_WATCH_OPERATOR_NAMESPACE: "true"
  ROOK_ENABLE_DISCOVERY_DAEMON: "false"
  CSI_ENABLE_VOLUME_REPLICATION: "false"
  ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS: "15"
  CSI_ENABLE_CSIADDONS: "false"
  CSI_GRPC_TIMEOUT_SECONDS: "150"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rook-ceph-operator
  namespace: rook-ceph 
  labels:
    operator: rook
    storage-backend: ceph
    app.kubernetes.io/name: rook-ceph
    app.kubernetes.io/instance: rook-ceph
    app.kubernetes.io/component: rook-ceph-operator
    app.kubernetes.io/part-of: rook-ceph-operator
spec:
  selector:
    matchLabels:
      app: rook-ceph-operator
  replicas: 1
  template:
    metadata:
      labels:
        app: rook-ceph-operator
    spec:
      serviceAccountName: rook-ceph-system
      containers:
        - name: rook-ceph-operator
          image: rook/ceph:v1.9.3
          args: ["ceph", "operator"]
          securityContext:
            runAsNonRoot: true
            runAsUser: 2016
            runAsGroup: 2016
          volumeMounts:
            - mountPath: /var/lib/rook
              name: rook-config
            - mountPath: /etc/ceph
              name: default-config-dir
            - mountPath: /etc/webhook
              name: webhook-cert
          ports:
            - containerPort: 9443
              name: https-webhook
              protocol: TCP
          env:
            - name: ROOK_CURRENT_NAMESPACE_ONLY
              value: "false"
            - name: ROOK_DISCOVER_DEVICES_INTERVAL
              value: "60m"
            - name: ROOK_HOSTPATH_REQUIRES_PRIVILEGED
              value: "true"
            - name: ROOK_ENABLE_SELINUX_RELABELING
              value: "true"
            - name: ROOK_ENABLE_FSGROUP
              value: "true"
            - name: ROOK_DISABLE_DEVICE_HOTPLUG
              value: "false"
            - name: DISCOVER_DAEMON_UDEV_BLACKLIST
              value: "(?i)dm-[0-9]+,(?i)rbd[0-9]+,(?i)nbd[0-9]+"
            - name: ROOK_ENABLE_MACHINE_DISRUPTION_BUDGET
              value: "false"
            - name: ROOK_UNREACHABLE_NODE_TOLERATION_SECONDS
              value: "5"
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
      volumes:
        - name: rook-config
          emptyDir: {}
        - name: default-config-dir
          emptyDir: {}
        - name: webhook-cert
          emptyDir: {}

EOF

With the Rook operator.yaml saved we can now apply it to the SNO node and after a few minutes validate that the Rook operator is running:

$ oc create -f operator.yaml 
securitycontextconstraints.security.openshift.io/rook-ceph created
securitycontextconstraints.security.openshift.io/rook-ceph-csi created
configmap/rook-ceph-operator-config created
deployment.apps/rook-ceph-operator created

$ oc get pods -n rook-ceph
NAME                                                              READY   STATUS      RESTARTS   AGE
rook-ceph-operator-84bf68d9bd-lv9l9                               1/1     Running     0          1m

Finally we get the heart of this configuration which is the cluster.yaml file.   In this file we need to make some modification since we only have a single node for the Ceph deployment via Rook.   Here are things I modified from the default:
  • osd_pool_default_size needs to be 1
  • mon count needs to be 1 and allowMultiplePerNode needs to be true
  • mgr count needs to be 1 and allowMultiplePerNode needs to be true
  • storage device needs to be set to extra disk available (in my case sdb)
  • osdsPerDevice needs to be 1
  • managePodBudgets and manageMachineDisruptionBudgets both set to false
We also need to keep in mind that this configuration is not a redundant configuration with OSD replication across many nodes so its really just a configuration of convenience to provide dynamic storage for the potential of multiple applications requiring persistent volume claims.  We can proceed by saving out the cluster.yaml with the updates mentioned above:

$ cat << EOF > ~/cluster.yaml
kind: ConfigMap
apiVersion: v1
metadata:
  name: rook-config-override
  namespace: rook-ceph
data:
  config: |
    [global]
    osd_pool_default_size = 1
---
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph 
spec:
  cephVersion:
    image: quay.io/ceph/ceph:v16.2.7
    allowUnsupported: false
  dataDirHostPath: /var/lib/rook
  skipUpgradeChecks: false
  continueUpgradeAfterChecksEvenIfNotHealthy: false
  waitTimeoutForHealthyOSDInMinutes: 10
  mon:
    count: 1
    allowMultiplePerNode: true
  mgr:
    count: 1
    allowMultiplePerNode: true
    modules:
      - name: pg_autoscaler
        enabled: true
  dashboard:
    enabled: true
    ssl: true
  monitoring:
    enabled: false
  network:
    connections:
      encryption:
        enabled: false
      compression:
        enabled: false
  crashCollector:
    disable: false
  cleanupPolicy:
    confirmation: ""
    sanitizeDisks:
      method: quick
      dataSource: zero
      iteration: 1
    allowUninstallWithVolumes: false
  annotations:
  labels:
  resources:
  removeOSDsIfOutAndSafeToRemove: false
  priorityClassNames:
    mon: system-node-critical
    osd: system-node-critical
    mgr: system-cluster-critical
  storage: 
    useAllNodes: true
    useAllDevices: false
    devices:
    - name: "sdb"
    config:
      osdsPerDevice: "1"
    onlyApplyOSDPlacement: false
  disruptionManagement:
    managePodBudgets: false
    osdMaintenanceTimeout: 30
    pgHealthCheckTimeout: 0
    manageMachineDisruptionBudgets: false
    machineDisruptionBudgetNamespace: openshift-machine-api
  healthCheck:
    daemonHealth:
      mon:
        disabled: false
        interval: 45s
      osd:
        disabled: false
        interval: 60s
      status:
        disabled: false
        interval: 60s
    livenessProbe:
      mon:
        disabled: false
      mgr:
        disabled: false
      osd:
        disabled: false
    startupProbe:
      mon:
        disabled: false
      mgr:
        disabled: false
      osd:
        disabled: false

EOF

Now with the cluster.yaml saved we can apply it to the cluster and let the Rook operator do the work of creating the Ceph cluster on our SNO node:

$ oc create -f cluster.yaml 
configmap/rook-config-override created
cephcluster.ceph.rook.io/rook-ceph created

After a few minutes, depending on the speed of the SNO deployment, we can validate that the Ceph cluster is up and deployed on our SNO node:

$ oc get pods -n rook-ceph
NAME                                                              READY   STATUS      RESTARTS   AGE
csi-cephfsplugin-provisioner-7577bb4d59-kxmq8                     6/6     Running     0          118s
csi-cephfsplugin-x2njd                                            3/3     Running     0          118s
csi-rbdplugin-provisioner-847b498845-7z5qc                        6/6     Running     0          119s
csi-rbdplugin-tlw5d                                               3/3     Running     0          119s
rook-ceph-crashcollector-master-0.sno3.schmaustech.com-858mmbx2   1/1     Running     0          48s
rook-ceph-mgr-a-57fbb7fb47-9rjl5                                  1/1     Running     0          81s
rook-ceph-mon-a-d94d79bb5-l6f8p                                   1/1     Running     0          110s
rook-ceph-operator-84bf68d9bd-qkj6k                               1/1     Running     0          17m
rook-ceph-osd-0-6c98c84f66-96l5q                                  1/1     Running     0          48s
rook-ceph-osd-prepare-master-0.sno3.schmaustech.com-l5s8t         0/1     Completed   0          60s

We can see from the above output that the pods for both the single mon and single osd are up and running along with the additional services for our single node Ceph cluster.   We can further validate that the Ceph cluster is up and running by deploying a Ceph toolbox pod.   For that we will just use the toolbox.yaml from the Rook Github repository.  Once we create the pod we can validate it is running by filtering for the rook-ceph-tools pod in the rook-ceph namespace:

$ oc create -f https://raw.githubusercontent.com/rook/rook/master/deploy/examples/toolbox.yaml
deployment.apps/rook-ceph-tools created

$ oc get pods -n rook-ceph| grep rook-ceph-tools
rook-ceph-tools-d6d7c985c-6zwc7                                   1/1     Running     0          54s

Now lets use the running toolbox to check on the Ceph cluster by issuing an exec command to it and passing in a ceph status:

$ oc -n rook-ceph exec -it rook-ceph-tools-d6d7c985c-6zwc7 -- ceph status
  cluster:
    id:     c54ad01e-e9f8-48c9-806b-a7d4748eb977
    health: HEALTH_OK
 
  services:
    mon: 1 daemons, quorum a (age 7m)
    mgr: a(active, since 5m)
    osd: 1 osds: 1 up (since 5m), 1 in (since 6m)
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   4.8 MiB used, 160 GiB / 160 GiB avail
    pgs: 

Sure enough our Ceph cluster is up and running and in a healthy state.   Lets move on now to confirm we can consume storage from it.   To do that we need to setup a storageclass configuration like the example below which will create a Ceph RBD block storageclass:

$ cat << EOF > ~/storageclass.yaml
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicapool
  namespace: rook-ceph
spec:
  failureDomain: host
  replicated:
    size: 1
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: rook-ceph-block
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
    clusterID: rook-ceph
    pool: replicapool
    imageFormat: "2"
    imageFeatures: layering
    csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
    csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
    csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
    csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
    csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
    csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
    csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Delete
allowVolumeExpansion: true
EOF

Once we have saved the storageclass.yaml file lets go ahead and apply it to the SNO node and then check that the storageclass was created:

$ oc create -f storageclass.yaml
cephblockpool.ceph.rook.io/replicapool created
storageclass.storage.k8s.io/rook-ceph-block created

$ oc get sc
NAME              PROVISIONER                  RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
rook-ceph-block   rook-ceph.rbd.csi.ceph.com   Delete          Immediate           true                   3s
oc get 

Now that we have a storageclass created I like to do one more thing to ensure any outstanding persistent volume claims get fulfilled by the storageclass automatically.  To do this I will patch the storageclass to be the default:

$ oc patch storageclass  rook-ceph-block -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
storageclass.storage.k8s.io/rook-ceph-block patched

$ oc get sc
NAME                        PROVISIONER                  RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
rook-ceph-block (default)   rook-ceph.rbd.csi.ceph.com   Delete          Immediate           true                   8m41s

At this point everything is configured to consume storage.  What I did in the example below was to kick off an installation of Red Hat Advanced Cluster Management on my SNO node because I knew it would need a PV.  Once it completed installation I confirmed by looking at the persistent volumes that indeed one had been created from our storageclass which gets its storage from the Ceph cluster:

$ oc get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                             STORAGECLASS      REASON   AGE
pvc-791551c6-cdc4-4c9e-9692-c9622dbef4e8   10Gi       RWO            Delete           Bound    open-cluster-management/search-redisgraph-pvc-0   rook-ceph-block            45s


Hopefully this was a helpful blog in providing a dynamic almost ODF like experience for a SNO node deployment!