Practical Example of Red Hat Image Mode: Bootc

Red Hat Image Mode is a new approach to operating system (OS) deployment that lets users build, deploy, and manage Red Hat Enterprise Linux as a bootc container image. It reduces complexity across the enterprise by letting development, operations, and solution providers use the same container-native tools and techniques to manage everything from applications to the underlying OS.

The following blog is the step by step process I took to create a Bootc image mode iso image. The following steps were carried out on a Red Hat Enterprise Linux 9.4 host as root though sudo and a regular user could also be used.

The first requirement was getting the container tools and so I used the following to get those installed.

# dnf install container-tools
Updating Subscription Management repositories.
Last metadata expiration check: 2:20:17 ago on Fri 26 Jul 2024 07:17:27 AM CDT.
Dependencies resolved.
==============================================================================================================================================================================================================================================
 Package                                                Architecture                                  Version                                                   Repository                                                               Size
==============================================================================================================================================================================================================================================
Installing:
 container-tools                                        noarch                                        1-14.el9                                                  rhel-9-for-x86_64-appstream-rpms                                        8.3 k
Upgrading:
 aardvark-dns                                           x86_64                                        2:1.10.0-3.el9_4                                          rhel-9-for-x86_64-appstream-rpms                                        969 k
 buildah                                                x86_64                                        2:1.33.7-3.el9_4                                          rhel-9-for-x86_64-appstream-rpms                                        9.4 M
 netavark                                               x86_64                                        2:1.10.3-1.el9                                            rhel-9-for-x86_64-appstream-rpms                                        4.0 M
 podman                                                 x86_64                                        4:4.9.4-6.el9_4                                           rhel-9-for-x86_64-appstream-rpms                                         16 M
Installing dependencies:
 cockpit-podman                                         noarch                                        84.1-1.el9                                                rhel-9-for-x86_64-appstream-rpms                                        683 k
 podman-docker                                          noarch                                        4:4.9.4-6.el9_4                                           rhel-9-for-x86_64-appstream-rpms                                        106 k
 podman-remote                                          x86_64                                        4:4.9.4-6.el9_4                                           rhel-9-for-x86_64-appstream-rpms                                         10 M
 python3-podman                                         noarch                                        3:4.9.0-1.el9                                             rhel-9-for-x86_64-appstream-rpms                                        178 k
 python3-pyxdg                                          noarch                                        0.27-3.el9                                                rhel-9-for-x86_64-appstream-rpms                                        108 k
 python3-tomli                                          noarch                                        2.0.1-5.el9                                               rhel-9-for-x86_64-appstream-rpms                                         37 k
 skopeo                                                 x86_64                                        2:1.14.3-0.1.el9                                          rhel-9-for-x86_64-appstream-rpms                                        8.5 M
 toolbox                                                x86_64                                        0.0.99.5-2.el9                                            rhel-9-for-x86_64-appstream-rpms                                        2.5 M
 udica                                                  noarch                                        0.2.8-1.el9                                               rhel-9-for-x86_64-appstream-rpms                                         54 k

Transaction Summary
==============================================================================================================================================================================================================================================
Install  10 Packages
Upgrade   4 Packages

Total download size: 52 M
Is this ok [y/N]: y
Downloading Packages:
(1/14): container-tools-1-14.el9.noarch.rpm                                                                                                                                                                    29 kB/s | 8.3 kB     00:00    
(2/14): python3-tomli-2.0.1-5.el9.noarch.rpm                                                                                                                                                                  126 kB/s |  37 kB     00:00    
(3/14): python3-pyxdg-0.27-3.el9.noarch.rpm                                                                                                                                                                   347 kB/s | 108 kB     00:00    
(4/14): cockpit-podman-84.1-1.el9.noarch.rpm                                                                                                                                                                  3.0 MB/s | 683 kB     00:00    
(5/14): udica-0.2.8-1.el9.noarch.rpm                                                                                                                                                                          616 kB/s |  54 kB     00:00    
(6/14): python3-podman-4.9.0-1.el9.noarch.rpm                                                                                                                                                                 1.3 MB/s | 178 kB     00:00    
(7/14): podman-docker-4.9.4-6.el9_4.noarch.rpm                                                                                                                                                                1.2 MB/s | 106 kB     00:00    
(8/14): toolbox-0.0.99.5-2.el9.x86_64.rpm                                                                                                                                                                     4.6 MB/s | 2.5 MB     00:00    
(9/14): netavark-1.10.3-1.el9.x86_64.rpm                                                                                                                                                                      4.4 MB/s | 4.0 MB     00:00    
(10/14): aardvark-dns-1.10.0-3.el9_4.x86_64.rpm                                                                                                                                                               3.3 MB/s | 969 kB     00:00    
(11/14): skopeo-1.14.3-0.1.el9.x86_64.rpm                                                                                                                                                                     3.9 MB/s | 8.5 MB     00:02    
(12/14): podman-remote-4.9.4-6.el9_4.x86_64.rpm                                                                                                                                                               3.2 MB/s |  10 MB     00:03    
(13/14): buildah-1.33.7-3.el9_4.x86_64.rpm                                                                                                                                                                    3.5 MB/s | 9.4 MB     00:02    
(14/14): podman-4.9.4-6.el9_4.x86_64.rpm                                                                                                                                                                      5.3 MB/s |  16 MB     00:02    
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                         9.6 MB/s |  52 MB     00:05     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                      1/1 
  Upgrading        : aardvark-dns-2:1.10.0-3.el9_4.x86_64                                                                                                                                                                                1/18 
  Upgrading        : netavark-2:1.10.3-1.el9.x86_64                                                                                                                                                                                      2/18 
  Upgrading        : podman-4:4.9.4-6.el9_4.x86_64                                                                                                                                                                                       3/18 
  Installing       : skopeo-2:1.14.3-0.1.el9.x86_64                                                                                                                                                                                      4/18 
  (...)
  Verifying        : podman-4:4.9.4-6.el9_4.x86_64                                                                                                                                                                                      17/18 
  Verifying        : podman-2:4.6.1-8.el9_3.x86_64                                                                                                                                                                                      18/18 
Installed products updated.

Upgraded:
  aardvark-dns-2:1.10.0-3.el9_4.x86_64                           buildah-2:1.33.7-3.el9_4.x86_64                           netavark-2:1.10.3-1.el9.x86_64                           podman-4:4.9.4-6.el9_4.x86_64                          
Installed:
  cockpit-podman-84.1-1.el9.noarch      container-tools-1-14.el9.noarch      podman-docker-4:4.9.4-6.el9_4.noarch      podman-remote-4:4.9.4-6.el9_4.x86_64      python3-podman-3:4.9.0-1.el9.noarch      python3-pyxdg-0.27-3.el9.noarch     
  python3-tomli-2.0.1-5.el9.noarch      skopeo-2:1.14.3-0.1.el9.x86_64       toolbox-0.0.99.5-2.el9.x86_64             udica-0.2.8-1.el9.noarch                 

Complete!

Once the tooling is in place we need to login to register.redhat.io with a Red Hat account.

# podman login registry.redhat.io
Username: myusername
Password: 
Login Succeeded!

Next we need to pull two images for this workflow locally:

• The latest Red Hat Bootc Image Builder image
• The latest Red Hat Bootc image

First let's pull the image builder image.

# sudo podman pull registry.redhat.io/rhel9/bootc-image-builder
Trying to pull registry.redhat.io/rhel9/bootc-image-builder:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob edab65b863ae done   | 
Copying blob ce39a10ee5db done   | 
Copying blob 6e743249fd30 done   | 
Copying config 7e467a06cb done   | 
Writing manifest to image destination
Storing signatures
7e467a06cbc49d0e601ab5acad54afcf16c7fd3187296c74f2780ec3e758977a

Then we can pull the Red Hat Enterprise Linux Bootc image. Note if there is a need to build a custom Bootc image that can also be done as well and is documented here

# podman pull registry.redhat.io/rhel9/rhel-bootc:latest
Trying to pull registry.redhat.io/rhel9/rhel-bootc:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob b696b6658912 done   | 
Copying blob b696b6658912 done   | 
Copying blob e0b929cd893f done   | 
(...)
Copying blob 2942d3f50802 done   | 
Copying blob 8ea0992b56d4 done   | 
Copying blob ad312c5c40cc done   | 
Copying blob bd9ddc54bea9 done   | 
Copying config 482f4c67cc done   | 
Writing manifest to image destination
Storing signatures
482f4c67cc158fa4b8db27c09832d3133bc45b1d989aa6d166ca2ef45f6c7178

Once we have pulled our images let's just review what we have with podman images.

# podman images
REPOSITORY                                    TAG         IMAGE ID      CREATED     SIZE
registry.redhat.io/rhel9/rhel-bootc           latest      482f4c67cc15  4 days ago  1.47 GB
registry.redhat.io/rhel9/bootc-image-builder  latest      7e467a06cbc4  4 days ago  521 MB

Next I created a directory structure under root home directory.

# mkdir ~/bootc
# cd bootc
# mkdir output

I also created a config.toml file which allows us to customize the image. In this example I embedding my user/password, public ssh-key and the groups I should belong to.

# cat <<EOL > config.toml 
[[blueprint.customizations.user]]
name = "myuser"
password = "password"
key = "ssh-rsa publick-key"
groups = ["wheel"]
EOL

Before we begin building let's review where we are and what is in the directory structure.

# pwd
/root/bootc
# ls
config.toml  output

Before we begin the build process we need to workaround an issue here by removing the signatures in our local copy of the RHEL Bootc image.

# skopeo copy --remove-signatures containers-storage:registry.redhat.io/rhel9/rhel-bootc:latest containers-storage:registry.redhat.io/rhel9/rhel-bootc:latest
INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled 
Copying blob f2d952b04649 skipped: already exists  
Copying blob bc1abff2c8d4 skipped: already exists  
Copying blob 491341ca1509 skipped: already exists  
(...)
Copying blob 6c118cde0f5c skipped: already exists  
Copying blob 5f70bf18a086 skipped: already exists  
Copying blob 12787d84fa13 skipped: already exists  
Copying config 482f4c67cc done   | 
Writing manifest to image destination

If everything looks good we can proceed to run the build process which consists of using podman to run the bootc-image-builder while passing in some directories and referencing the starting image we will use to build our ISO. The entire build process happens within a container and the ISO generated is dumped to the output directory we have mapped into the container. The process will take a bit to run and the log output is very long. I have provided the complete log run here with the condensed version below.

# podman run --rm -it --privileged --security-opt label=type:unconfined_t -v /var/lib/containers/storage:/var/lib/containers/storage -v /root/bootc/output:/output -v /root/bootc/config.toml:/config.toml registry.redhat.io/rhel9/bootc-image-builder --type iso --config /config.toml --local registry.redhat.io/rhel9/rhel-bootc:latest
Generating manifest manifest-iso.json
DONE
Building manifest-iso.json
starting -Pipeline source org.osbuild.containers-storage: 51102953cf5005007cf8b3bd76c39a0ae558aa34f311915042cdc0bc4c1fb246
Build
  root: <host>
Pipeline source org.osbuild.curl: 07337b98b3c859adfb37b011d83cf0511884147bf999e7869ffbf9074b529a4f
Build
  root: <host>
(...)
Writing to 'stdio:/run/osbuild/tree/install.iso' completed successfully.

⏱  Duration: 3s
org.osbuild.implantisomd5: bf37713d1bdb752cb80271b8243583e86665b003c69abff0e1730de24398fac1 {
  "filename": "install.iso"
}
['implantisomd5', '/run/osbuild/tree/install.iso']
Inserting md5sum into iso image...
md5 = 11cf490ec817e7904ba35961d5323195
Inserting fragment md5sums into iso image...
fragmd5 = 4f7338c869faf1a2881d46dc4ac1eebf46148d4bf2fa1d59149d74f43195
frags = 20
Setting supported flag to 0

⏱  Duration: 3s
manifest - finished successfully
build:        9133fb8610ab053dae7e281e6a6655dbb912c4530d32e4da75c06b8713a87c80
anaconda-tree:    fcd61d1236a42900977530f12f45fe452f2f0c8bf3c80a7ba60cb45ffe4bf36d
rootfs-image:    0a517e05ab42f947beec8dae4d2da338ca9cc7fe17b1daba013e24b1c60aeadf
efiboot-tree:    61a20c820b40436ce7bd6d1a74c6b97a05f7c8800b678083942e814cf9f7cc0e
bootiso-tree:    53d4713e1d036ca2d06ae3b0b64c8b9a4efb63cbc6fed4f13afe1e984b7af04d
bootiso:      bf37713d1bdb752cb80271b8243583e86665b003c69abff0e1730de24398fac1
Build complete!
Results saved in
.

Once the build process completes we will find an install.iso in the output/bootiso/ directory.

# cd ~/bootc/output/bootiso/
# ls -l
total 2211840
-rw-r--r--. 1 root root 2264924160 Jul 26 10:38 install.iso

The installation iso that was created can then be used to install RedHat Enterprise Linux on another host automatically and rather quickly compared to the old package based method.

Hopefully this provides a simple example to understand the concept of building and consuming Red Hat Image Mode.

Practical Example of Red Hat CoreOS Layering

In Red Hat OpenShift 4.14 a new concept called image layering was introduced which allows one to build a container layer they can then apply on top of the Red Hat CoreOS layer. More details about it can be found here. We need to leverage this technology to apply an image layer that contains irqbalance since this package is not part of the base Red Hat CoreOS aarch64 image nor is it available as an extension. Irqbalance will become part of Red Hat CoreOS for aarch64 in the future based on this merge request. The steps below will describe how create, build and apply the image layer containing the irqbalance package along with enabling it for aarch64.

The first step is to get the current rhel-coreos image from the cluster where we will be applying the image layer.  We can use the oc adm release info command to obtain this information from our OpenShift 4.15.23 cluster.

$ oc adm release info --image-for rhel-coreos
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a2f1b7530956b765f1b0337b824fde28d6987b519eec0aaadc9d261e9fd1e550

Next we take the release image output and place it into a Dockerfile that will have a run command to install and enable the package for irqbalance.

$ cat <<EOF > Dockerfile.irqbalance 
FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a2f1b7530956b765f1b0337b824fde28d6987b519eec0aaadc9d261e9fd1e550


RUN rpm-ostree install irqbalance && \  #install the irqbalance package
    systemctl enable irqbalance && \    #enable irqbalance service
    rm -r -f /etc/yum.repos.d/* \       #remove entitlements from the system the image is being built on otherwise creates issues 
    ostree container commit             #commit the ostree container
EOF

Once we have created the Dockerfile we can use podman to build the container. Note that I performed this process on an aarch64 host, specifically a Ampere Altra Developer Workstation with Red Hat Enterprise Linux 9 because I wanted my image to be made for an aarch64 Red Hat CoreOS host.  Note I am tagging my image with the version of OpenShift to remind me which OpenShift version.

$ podman build -t quay.io/redhat_emp1/ecosys-nvidia/ocp-4.15-irqbalance:4.15.23 -f Dockerfile.irqbalance .
STEP 1/2: FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a2f1b7530956b765f1b0337b824fde28d6987b519eec0aaadc9d261e9fd1e550
Trying to pull quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a2f1b7530956b765f1b0337b824fde28d6987b519eec0aaadc9d261e9fd1e550...
Getting image source signatures
Copying blob b839e1a7e4d1 done   | 
Copying blob d178eea3fd50 done   |
(...)
Copying blob 413ee3c4305f done   | 
Copying config 9f41514ccf done   | 
Writing manifest to image destination
STEP 2/2: RUN rpm-ostree install irqbalance &&     systemctl enable irqbalance &&     ostree container commit
Enabled rpm-md repositories: rhel-9-for-aarch64-appstream-rpms rhel-9-for-aarch64-baseos-rpms
Updating metadata for 'rhel-9-for-aarch64-appstream-rpms'...done
Updating metadata for 'rhel-9-for-aarch64-baseos-rpms'...done
Importing rpm-md...done
rpm-md repo 'rhel-9-for-aarch64-appstream-rpms'; generated: 2024-09-12T14:26:05Z solvables: 19074
rpm-md repo 'rhel-9-for-aarch64-baseos-rpms'; generated: 2024-09-11T06:47:28Z solvables: 7179
Resolving dependencies...done
Will download: 1 package (71.6?kB)
Downloading from 'rhel-9-for-aarch64-baseos-rpms'...done
Installing 1 packages:
  irqbalance-2:1.9.2-3.el9.aarch64 (rhel-9-for-aarch64-baseos-rpms)
Installing: irqbalance-2:1.9.2-3.el9.aarch64 (rhel-9-for-aarch64-baseos-rpms)
Created symlink /etc/systemd/system/multi-user.target.wants/irqbalance.service → /usr/lib/systemd/system/irqbalance.service.
COMMIT quay.io/redhat_emp1/ecosys-nvidia/ocp-4.15-irqbalance:4.15.23
--> 624c70f71a77
Successfully tagged quay.io/redhat_emp1/ecosys-nvidia/ocp-4.15-irqbalance:4.15.23
624c70f71a77ce3c30fd973b4afc09fc4558b43e64ec43851de2cf4d7ad7f6a0

Once the image is built we can push it to our favorite location in the registry. This image should be pushed to a location that we have access to from our cluster.

$ podman push quay.io/redhat_emp1/ecosys-nvidia/ocp-4.15-irqbalance:4.15.23
Getting image source signatures
Copying blob 79bb3562fc5d done   | 
Copying blob 6dbb46d6d565 done   | 
(...)
Copying blob 00ad2dbb774b done   | 
Copying config 624c70f71a done   | 
Writing manifest to image destination

Once the image is in a registry location we can generate a machine configuration file and specify the osImageURL with the location of our image. 

$ cat <<EOF >irqbalance-machine.yaml 
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master 
  name: irqbalance-layer-machineconfig 
spec:
  osImageURL: quay.io/redhat_emp1/ecosys-nvidia/ocp-4.15-irqbalance:4.15.23
EOF

The machine configuration file we created can then be applied to the cluster.  We can use oc create to do the work here and be aware the node(s) where this machine configuration gets applied will reboot.

$ oc create -f irqbalance-machine.yaml 
machineconfig.machineconfiguration.openshift.io/irqbalance-layer-machineconfig created

After the reboot we can validate that irqbalance is successfully installed and running by going into a debug container and checking for the package and the systemctl status.

$ oc debug node/$(oc get node -o json | jq -r '.items[0].metadata.name')
Starting pod/nvd-srv-37nvidiaengrdu2dcredhatcom-debug-7r745 ...
To use host binaries, run `chroot /host`
Pod IP: 10.6.135.16
If you don't see a command prompt, try pressing enter.
sh-4.4# chroot /host

sh-5.1# rpm -q irqbalance
irqbalance-1.9.2-3.el9.aarch64

sh-5.1# systemctl status irqbalance
● irqbalance.service - irqbalance daemon
     Loaded: loaded (/usr/lib/systemd/system/irqbalance.service; enabled; preset: enabled)
     Active: active (running) since Thu 2024-09-12 19:20:27 UTC; 1h 21min ago
       Docs: man:irqbalance(1)
             https://github.com/Irqbalance/irqbalance
   Main PID: 14874 (irqbalance)
      Tasks: 2 (limit: 783503)
     Memory: 5.2M
        CPU: 3.980s
     CGroup: /system.slice/irqbalance.service
             └─14874 /usr/sbin/irqbalance --foreground

sh-5.1#

Again this was a simple practical example of using image layering in OpenShift but hopefully this article gives a good enough example that one could expand on it to provide other packages, configurations and files in the event they need to be applied to the Red Hat CoreOS image.

Biometric Authentication in Fedora 39

 

I have been using a Apple Macintosh M1 Powerbook for a year now and I have grown accustom to using the fingerprint reader to login into the device rather then type my password every time to unlock it.   Which was why when I received a new Lenovo P1 ThinkPad that had Synaptics Prometheus MIS Touch fingerprint reader running Fedora 39, I was intrigued to see if I could get the reader to be used to authenticate my username.

Luckily this process was a little more trivial then I thought it would be since Freedesktop.org has the fprintd daemon.  However I found it a little more distracting finding a comprehensive set of steps.  Further not all fingerprint readers are supported so you have to determine if the driver is supported or not.   In my case my reader was supported.  The rest of this blog will briefly outline the steps on order to successfully authenticate with the fingerprint reader.

The first step is to install the required packages on the system:

$ sudo dnf install fprintd fprintd-pam

Then enable fprintd.service and start it with systemclt. 

$ sudo systemclt start fprintd.service

We can check that the service is running by using systemctl status.  I do want to point out thought that the service will stop after a few tries if no fingerprints are enrolled so do not be alarmed.

$ sudo systemclt status fprintd.service

Next let's erase any old fingerprint data in the event this was used before.  In most cases this step is not required but to be thorough it will not hurt.

$ fprintd-delete $USER

Now we are ready to enroll a fingerprint.  Once you execute the fprintd-enroll service we need to take our index finger and place it on the fingerprint reader multiple times until we get back to a command prompt.

$ fprintd-enroll

We can use the fprintd-verify command confirm our fingerprint works because we make our PAM authentication changes.  When we execute the command we need to put our index finger on the fingerprint reader and it should return to a prompt with an exit status of 0 if successful.

$ fprintd-verify

Now that we have verified our fingerprint works lets make the configuration changes that ensure PAM can use our fingerprint for authentication.

$ sudo authselect enable-feature with-fingerprint
$ sudo authselect apply-changes

We can verify our changes by running the following commands.

$ sudo authselect current

At this point we can go ahead and reboot the laptop.

When we arrive at the login screen click on the username.  Then we get either the classic password login or we can also use the fingerprint reader.  The latter of which should log us in when we place our index finger on the reader.

Again this might be trivial but I found it interesting enough to want to write a quick blog about it to at least capture the experience.  Hopefully this will help someone else who might be struggling trying to get their fingerprint reader to work.

Exploring Golang with MQTT: File Transfers

While I've dabbled in writing scripts using Perl, Bash, and Python, I wouldn't necessarily label myself as a developer. However, I do have a penchant for automation, organizing logic, and embracing challenges. It was this innate curiosity that led me to explore Golang recently and experiment with its integration with MQTT.  The harmonious combination between MQTT's lightweight messaging protocol and Go's concurrency model presents a compelling case for utilizing MQTT with Golang.

MQTT (Message Queuing Telemetry Transport) is a lightweight publish-subscribe messaging protocol designed for efficient communication between devices with limited bandwidth and processing capabilities. It excels in scenarios where real-time data exchange is crucial, such as IoT (Internet of Things) applications, telemetry systems, and messaging platforms.

Golang, or Go, is a modern and efficient programming language known for its simplicity, concurrency support, and performance. It provides built-in support for concurrent programming through goroutines and channels, making it ideal for building highly concurrent and scalable systems.

When you combine MQTT with Go, you leverage the strengths of both technologies to create robust, scalable, and real-time communication systems. Here's why this combination is so compelling:

Efficiency: Both MQTT and Go are designed for efficiency. MQTT's lightweight protocol minimizes bandwidth and processing overhead, making it suitable for resource-constrained environments. Go's efficient runtime and concurrency model allow you to handle a large number of concurrent connections and process messages concurrently with minimal overhead.

Concurrency: Go's built-in support for concurrency with goroutines and channels aligns perfectly with MQTT's asynchronous messaging paradigm. You can easily handle thousands of concurrent MQTT connections and process incoming messages concurrently, leveraging the power of parallelism to scale your applications.

Simplicity: MQTT and Go are both known for their simplicity and ease of use. With the paho.mqtt.golang library, integrating MQTT into your Go applications is straightforward and intuitive. You can quickly connect to MQTT brokers, publish and subscribe to topics, and handle messages with minimal boilerplate code.

Scalability: The combination of MQTT and Go enables you to build highly scalable systems that can handle massive workloads with ease. Whether you're building IoT platforms with millions of devices or real-time messaging systems with high throughput requirements, MQTT and Go provide the scalability you need to meet your application's demands.

Fast forward to my little Go project which came out of some of my research around transferring files via MQTT.  While maybe not the most practical I had read of others doing it with Python and even just using the Mosquito publisher and subscriber tools.   But again the goal here was to learn a little about Go and tie it into something that motivated me to figure it out.  Hence my own file transfer publisher and subscriber written in Go.

The publisher code, which can be found here, does the following:

• Establishes a connection to the broker server with the topic of transfer
• Watches the following directory on where it is run: /root/outbound
• Any files that are dropped into the directory are then published to the broker on the topic of transfer

Note: MQTT has a 256MB limit of data it can transfer.  Further it will chunk the data up into segment messages which creates another challenge.

The subscriber code, which can be found here, does the following:

• Establishes a connection to the broker server with the topic of transfer
• Listens for any published messages on the topic transfer
• When a message is published, in this case a file, the subscriber pulls it down and places it into the /root/inbound directory
• The subscriber will also try to determine the file type by using the mimetype library and looking at the first 512 bytes of the file.  If it cannot be determined it defaults to a .txt extension.

The go.mod file used for the project can be found here.

For the experiment I simply created the two inbound/outbound directories on my system.   In separate terminals I went ahead and ran each of the Go programs.   In a third terminal I set up a watch on the directory listing for inbound.  Then in a fourth terminal I went ahead and used the copy command to place some files into the outbound directory which the publisher code was watching.  The demo is below:

Now for a first pass at this experiment I was fairly pleased but there is definitely room for improvement in the following items:

• I want to be able to pass as arguments my MQTT server, the topic and what directory to watch.
• I need to figure out a better way to handle file names on the subscriber side.  Currently all the filenames end up having the name file with a Unix timestamp and then maybe if identified right the correct extension.   One thought I have for this is to actually bundle up the file on the publisher side into a JSON payload where we have the file name, the extension, the size and then the actual data blob.  Then on the subscriber side we would get that file and process it on receipt to obtain the real file name, extension and data of the file.
• We need to handle large files better since they are chunked up so on the subscriber side we need to be able to take the chunks and assemble them back together and then process them.

Using MQTT with Golang allows you to leverage the lightweight, efficient messaging protocol of MQTT and the concurrency and scalability of Go to build robust, scalable, and real-time communication systems. Whether you're building IoT applications, telemetry systems, or messaging platforms, this combination provides the performance, efficiency, and simplicity you need to succeed.

Near-Field Communication (NFC)

In an era where technology marketing revolves around AI, Edge, Cloud and Kubernetes, it's easy to overlook some of the subtle yet incredibly powerful innovations that have quietly revolutionized the way we interact with our devices and the world around us. Near-Field Communication (NFC) is one such technology, often overshadowed by its more glamorous counterparts. However, its impact is no less profound. Let's explore the wonders of NFC and discover what makes it such a game-changing technology.

My introduction to NFC began while running on a treadmill at the gym. Simply by tapping my iWatch against the NFC logo, I seamlessly paired the two devices. This allowed the treadmill to record my running workout effortlessly to the iWatch. Additionally, my iWatch transmitted my heart rate data to the treadmill in real-time, displaying it on the screen. All of this connectivity occurred through NFC, requiring minimal effort as the devices made contact. But what exactly was this technology called NFC?

Near-Field Communication, as the name suggests, is a short-range wireless communication technology that allows devices to exchange data when they are brought within close proximity, typically a few centimeters. NFC operates on the principle of electromagnetic induction, enabling communication between two NFC-enabled devices or between an NFC device and an NFC tag.

NFC relies on radio frequency identification (RFID) technology, which enables communication between devices by establishing a radio connection when they come into close proximity. Unlike Bluetooth or Wi-Fi, which require pairing and authentication processes, NFC communication is initiated simply by bringing two NFC-enabled devices close together.
NFC devices can operate in three modes: reader/writer mode, peer-to-peer mode, and card emulation mode.

• Reader/Writer Mode: In this mode, an NFC device acts as a reader or writer, interacting with NFC tags embedded in objects such as posters, labels, or smart cards. The device can read information from the tag or write data to it, enabling various applications such as contactless payments, access control, and information exchange.
• Peer-to-Peer Mode: In peer-to-peer mode, two NFC-enabled devices can communicate with each other to exchange data. This mode is commonly used for sharing files, photos, videos, or contact information between smartphones, tablets, or other NFC-equipped devices.
• Card Emulation Mode: In card emulation mode, an NFC-enabled device behaves like an NFC tag, allowing it to be used for contactless transactions or access control. This mode is frequently employed in mobile payment systems, where the device emulates a contactless smart card or payment card, enabling users to make secure transactions using their smartphones or wearable devices.

NFC technology has a wide range of applications across various industries and sectors:

• Contactless Payments: NFC-enabled smartphones, smartwatches, and payment cards allow users to make secure transactions by simply tapping their devices on NFC-enabled terminals at checkout counters.
• Access Control and Security: NFC tags or cards are used for access control in buildings, parking facilities, public transportation, and events, providing a convenient and secure way to authenticate users and grant them access.
• Smart Advertising and Marketing: NFC tags embedded in posters, flyers, product packaging, or retail displays enable interactive marketing campaigns, allowing consumers to access additional information, promotional offers, or multimedia content by tapping their smartphones on the tags.
• Transportation and Ticketing: NFC technology is widely used in public transportation systems for contactless ticketing and fare collection, streamlining the boarding process and enhancing passenger convenience.
• Healthcare and Wellness: NFC-enabled devices and wearable sensors are utilized in healthcare applications such as patient monitoring, medication adherence, and medical device connectivity, facilitating remote monitoring and personalized healthcare solutions.

Near-Field Communication (NFC) may not always seize the headlines like some of its glitzy counterparts, but its impact on our daily lives is undeniable. From enabling contactless payments and access control to facilitating seamless data exchange and interactive experiences, NFC technology has quietly permeated various aspects of our interconnected world. As we continue to embrace the era of connectivity and digital innovation, NFC stands as a testament to the power of simplicity and proximity in shaping the way we interact with technology and each other.

Simplicity of Linux Routing Brings OpenShift Portability

Anyone who has ever done a proof of concept at a customer site knows how daunting it can be. There is allocating the customer's environment from a physical space perspective, power and cooling, and then the elephant in the room: networking. Networking always tends to be the most challenging because the way a customer architects and secures their network varies from each and every customer. Hence, when delivering a proof of concept, wouldn't it be awesome if all we needed was a single ipaddress and uplink for connectivity? Linux has always given us the capability to provide such a simple, elegant solution. It's the very reason why router distros like OPNsense, OpenWRT, pfSense and IPFire are based on Linux. In the following blog, I will review configuring such a state with the idea of providing the simplicity of a single uplink as a proof of concept.

In this example, I wanted to deliver a working Red Hat OpenShift compact cluster that I could bring anywhere. A fourth node acting as the gateway box will also run some infrastructure components with a switch to tie it all together. In the diagram below, we can see the layout of the configuration and how the networking is set up. I should note that this could use four physical boxes, or in my testing, I had all 4 nodes virtualized on a single host. We can see I have an interface enp1s0 on the gateway node that is connected to the upstream network or maybe even the internet depending on circumstances and then another internal interface enp2s0 which is connected to the internal network switch. All the OpenShift nodes are connected to the internal network switch as well. The internal network will never change, but the external network could be anything and could change if we wanted it to. What this means when bringing this setup to another location is I just need to update the enp1s0 interface with the right ipaddress, gateway and external nameserver. Further, to ensure the OpenShift API and ingress wildcards resolve via the external DNS (whatever controls that),  we will just add two records and point them to the enp1s0 interface ipaddress. Nothing changes on the OpenShift cluster nodes or gateway node configurations for DHCP or bind.

The gateway node has Red Hat Enterprise Linux 9.3 installed on it along with DHCP and Bind services both of which are listening only on the internal enp2s0 interface. Below is the dhcpd.conf config I am using.

cat /etc/dhcp/dhcpd.conf
option domain-name "schmaustech.com";
option domain-name-servers 192.168.100.1;
default-lease-time 1200;
max-lease-time 1000;
authoritative;
log-facility local7;

subnet 192.168.100.0 netmask 255.255.255.0 {
        option routers                  192.168.100.1;
        option subnet-mask              255.255.255.0;
        option domain-search            "schmaustech.com";
        option domain-name-servers      192.168.100.1,192.168.100.1;
        option time-offset              -18000;     # Eastern Standard Time
    range   192.168.100.225   192.168.100.240;
        next-server 192.168.100.1;
        if exists user-class and option user-class = "iPXE" {
            filename "ipxe";
        } else {
            filename "pxelinux.0";
        }
        class "httpclients" {
          match if substring (option vendor-class-identifier, 0, 10) = "HTTPClient";
          option vendor-class-identifier "HTTPClient";
          filename "http://192.168.100.246/arm/EFI/BOOT/BOOTAA64.EFI";
    }
}

host adlink-vm1 {
   option host-name "adlink-vm1.schmaustech.com";
   hardware ethernet 52:54:00:89:8d:d8;
   fixed-address 192.168.100.128;
}

host adlink-vm2 {
   option host-name "adlink-vm2.schmaustech.com";
   hardware ethernet 52:54:00:b1:d4:9d;
   fixed-address 192.168.100.129;
}

host adlink-vm3 {
   option host-name "adlink-vm3.schmaustech.com";
   hardware ethernet 52:54:00:5a:69:d1;
   fixed-address 192.168.100.130;
}

host adlink-vm4 {
   option host-name "adlink-vm4.schmaustech.com";
   hardware ethernet 52:54:00:ef:25:04;
   fixed-address 192.168.100.131;
}

host adlink-vm5 {
   option host-name "adlink-vm5.schmaustech.com";
   hardware ethernet 52:54:00:b6:fb:7d;
   fixed-address 192.168.100.132;
}

host adlink-vm6 {
   option host-name "adlink-vm6.schmaustech.com";
   hardware ethernet 52:54:00:09:2e:34;
   fixed-address 192.168.100.133;
}

And the Bind named.conf and schmaustech.com zone files I have configured.

$ cat /etc/named.conf
options {
    listen-on port 53 { 127.0.0.1; 192.168.100.1; };
    listen-on-v6 port 53 { any; };
    forwarders { 192.168.0.10; };
    directory     "/var/named";
    dump-file     "/var/named/data/cache_dump.db";
    statistics-file "/var/named/data/named_stats.txt";
    memstatistics-file "/var/named/data/named_mem_stats.txt";
    recursing-file  "/var/named/data/named.recursing";
    secroots-file   "/var/named/data/named.secroots";
        allow-query    { any; };
    recursion yes;
    dnssec-enable yes;
    dnssec-validation yes;
    dnssec-lookaside auto;
    bindkeys-file "/etc/named.root.key";
    managed-keys-directory "/var/named/dynamic";
    pid-file "/run/named/named.pid";
    session-keyfile "/run/named/session.key";
};

logging {
        channel default_debug {
                file "data/named.run";
                severity dynamic;
        };
};

zone "." IN {
    type hint;
    file "named.ca";
};

include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";

zone "schmaustech.com" IN {
        type master;
        file "schmaustech.com.zone";
};

zone    "100.168.192.in-addr.arpa" IN {
       type master;
       file "100.168.192.in-addr.arpa";
};

$ cat /var/named/schmaustech.com.zone 
$TTL 1D
@   IN SOA  dns.schmaustech.com   root.dns.schmaustech.com. (
                                       2022121315     ; serial
                                       1D              ; refresh
                                       1H              ; retry
                                       1W              ; expire
                                       3H )            ; minimum

$ORIGIN         schmaustech.com.
schmaustech.com.            IN      NS      dns.schmaustech.com.
dns                     IN      A       192.168.100.1
adlink-vm1    IN    A    192.168.100.128
adlink-vm2    IN    A    192.168.100.129
adlink-vm3    IN    A    192.168.100.130
adlink-vm4    IN    A    192.168.100.131
adlink-vm5    IN    A    192.168.100.132
adlink-vm6    IN    A    192.168.100.133
api.adlink    IN    A    192.168.100.134
api-int.adlink    IN    A    192.168.100.134
*.apps.adlink    IN    A    192.168.100.135

In order to have the proper network address translation and service redirection we need to modify the default firewalld configuration on the gateway box.

First let's go ahead and see what the active zone is with firewalld. We will find that both interfaces are in the public zone which is the default.

$ sudo firewall-cmd --get-active-zone
public
  interfaces: enp2s0 enp1s0

We will first set our two interfaces to variables to make the rest of the commands easy to follow. Interface enp1s0 will be set to external and enp2s0 will be set to internal. Then we will go ahead and create an internal zone. Note we do not need to create an external zone because one exists by default with firewalld. We can then assign the interfaces to their respective zones.

$ sudo EXTERNAL=enp1s0
$ sudo INTERNAL=enp2s0

$ sudo firewall-cmd --set-default-zone=internal
success

$ sudo firewall-cmd --change-interface=$EXTERNAL --zone=external --permanent
The interface is under control of NetworkManager, setting zone to 'external'.
success

$ sudo firewall-cmd --change-interface=$INTERNAL --zone=internal --permanent
The interface is under control of NetworkManager, setting zone to 'internal'.
success

Next we can enable masquerading between the zones. We will find that by default masquerading was enabled for the external zone. However if one chose different zone names we need to point out that both need to be set.

$ sudo firewall-cmd --zone=external --add-masquerade --permanent
Warning: ALREADY_ENABLED: masquerade
success

$ sudo firewall-cmd --zone=internal --add-masquerade --permanent
success

Now we can add the rules to forward traffic between zones.

$ sudo firewall-cmd --direct --permanent --add-rule ipv4 nat POSTROUTING 0 -o $EXTERNAL -j MASQUERADE
success

$ sudo firewall-cmd --direct --permanent --add-rule ipv4 filter FORWARD 0 -i $INTERNAL -o $EXTERNAL -j ACCEPT
success

$ sudo firewall-cmd --direct --permanent --add-rule ipv4 filter FORWARD 0 -i $EXTERNAL -o $INTERNAL -m state --state RELATED,ESTABLISHED -j ACCEPT
success

At this point let's go ahead and reload our firewall and show the active zones again. Now we should see our interfaces are in their proper zones and active.

$ sudo firewall-cmd --reload
success

$ sudo firewall-cmd --get-active-zone
external
  interfaces: enp1s0
internal
  interfaces: enp2s0

If we look at each zone we can see the default configuration that currently exists for each zone.

$ sudo firewall-cmd --list-all --zone=external
external (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp1s0
  sources:
  services: ssh
  ports:
  protocols:
  forward: no
  masquerade: yes
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

$ sudo firewall-cmd --list-all --zone=internal
internal (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp2s0
  sources:
  services: cockpit dhcpv6-client mdns samba-client ssh
  ports:
  protocols:
  forward: no
  masquerade: yes
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

The zones need to be updated for OpenShift so we can ensure any external traffic bound for https and port 6443 is sent to the OpenShift ingress virtual ipaddress and OpenShift api virual ipaddress respectively. We also need to allow for DNS resolution traffic internally outbound on the internal zone so we can resolve anything outside of our OpenShift environment dns records (like registry.redhat.io).

$ sudo firewall-cmd --permanent --zone=external --add-service=https
success
$ sudo firewall-cmd --permanent --zone=internal --add-service=https
success
$ sudo firewall-cmd --permanent --zone=external --add-forward-port=port=443:proto=tcp:toport=443:toaddr=192.168.100.135
success
$ sudo firewall-cmd --permanent --zone=external --add-port=6443/tcp
success
$ sudo firewall-cmd --permanent --zone=internal --add-port=6443/tcp
success
$ sudo firewall-cmd --permanent --zone=external --add-forward-port=port=6443:proto=tcp:toport=6443:toaddr=192.168.100.134
success
$ sudo firewall-cmd --permanent --zone=internal --add-service=dns
success
$ sudo firewall-cmd --reload
success

After we reloaded our configuration let's take a look at the external and internal zones to validate our changes took place.

$ sudo firewall-cmd --list-all --zone=external
external (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp1s0
  sources: 
  services: https ssh
  ports: 6443/tcp
  protocols: 
  forward: yes
  masquerade: yes
  forward-ports: 
    port=443:proto=tcp:toport=443:toaddr=192.168.100.135
    port=6443:proto=tcp:toport=6443:toaddr=192.168.100.134
  source-ports: 
  icmp-blocks: 
  rich rules:

$ sudo firewall-cmd --list-all --zone=internal
internal (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp2s0
  sources: 
  services: cockpit dhcpv6-client dns https mdns samba-client ssh
  ports: 6443/tcp
  protocols: 
  forward: yes
  masquerade: yes
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

Up to this point we would have a working setup if we were on Red Hat Enterprise Linux 8.x. However there were changes made with Red Hat Enterprise Linux 9.x and hence we need to add a internal to external policy to ensure proper ingress/egress traffic flow.

$ sudo firewall-cmd --permanent --new-policy policy_int_to_ext
success
$ sudo firewall-cmd --permanent --policy policy_int_to_ext --add-ingress-zone internal
success
$ sudo firewall-cmd --permanent --policy policy_int_to_ext --add-egress-zone external
success
$ sudo firewall-cmd --permanent --policy policy_int_to_ext --set-priority 100
success
$ sudo firewall-cmd --permanent --policy policy_int_to_ext --set-target ACCEPT
success
$ sudo firewall-cmd --reload
success

Let's take a quick look at the policies we set to confirm it is there.

$ sudo firewall-cmd --info-policy=policy_int_to_ext
policy_int_to_ext (active)
  priority: 100
  target: ACCEPT
  ingress-zones: internal
  egress-zones: external
  services: 
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

Now that we have completed the firewalld configuration we should be ready to deploy OpenShift. Since I have written about deploying OpenShift quite a bit in my past I won't go into the detailed steps here. I will point out that I did use Red Hat Assisted Installer at https://cloud.redhat.com

Once the OpenShift installation has completed we can pull down the kubeconfig and run a few commands to show its operations and how its networking is configured on the nodes:

% oc get nodes -o wide
NAME                         STATUS   ROLES                         AGE     VERSION           INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                  CONTAINER-RUNTIME
adlink-vm4.schmaustech.com   Ready    control-plane,master,worker   2d23h   v1.27.6+f67aeb3   192.168.100.131   <none>        Red Hat Enterprise Linux CoreOS 414.92.202311061957-0 (Plow)   5.14.0-284.40.1.el9_2.aarch64   cri-o://1.27.1-13.1.rhaos4.14.git956c5f7.el9
adlink-vm5.schmaustech.com   Ready    control-plane,master,worker   2d23h   v1.27.6+f67aeb3   192.168.100.132   <none>        Red Hat Enterprise Linux CoreOS 414.92.202311061957-0 (Plow)   5.14.0-284.40.1.el9_2.aarch64   cri-o://1.27.1-13.1.rhaos4.14.git956c5f7.el9
adlink-vm6.schmaustech.com   Ready    control-plane,master,worker   2d22h   v1.27.6+f67aeb3   192.168.100.133   <none>        Red Hat Enterprise Linux CoreOS 414.92.202311061957-0 (Plow)   5.14.0-284.40.1.el9_2.aarch64   cri-o://1.27.1-13.1.rhaos4.14.git956c5f7.el9

We can see from the above output the nodes are running on the 192.168.100.0/24 network which is our internal network. However if we ping from my Mac to api.adlink.schmaustech.com we can see the response is coming from 192.168.0.75 which just happens to be the interface on enp1s0 of our gateway box. We can also see any ingress names like console-openshift-console.apps.adlink.schmaustech.com also resolve to the 192.168.0.75 address.

% ping api.adlink.schmaustech.com -t 1
PING api.adlink.schmaustech.com (192.168.0.75): 56 data bytes
64 bytes from 192.168.0.75: icmp_seq=0 ttl=63 time=4.242 ms

--- api.adlink.schmaustech.com ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 4.242/4.242/4.242/0.000 ms

% ping console-openshift-console.apps.adlink.schmaustech.com -t 1
PING console-openshift-console.apps.adlink.schmaustech.com (192.168.0.75): 56 data bytes
64 bytes from 192.168.0.75: icmp_seq=0 ttl=63 time=2.946 ms

--- console-openshift-console.apps.adlink.schmaustech.com ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 2.946/2.946/2.946/nan ms

Finally, if we curl the OpenShift console from my Mac, we can see we also get a 200 response, so the console is accessible from outside the private network OpenShift is installed on.

% curl -k -I https://console-openshift-console.apps.adlink.schmaustech.com
HTTP/1.1 200 OK
referrer-policy: strict-origin-when-cross-origin
set-cookie: csrf-token=+gglOP1AF2FjXsZ4E61xa53Dtagem8u5qFTG08ukPD6GnulryLllm7SQplizT51X5Huzqf4LTU47t7yzdCaL5g==; Path=/; Secure; SameSite=Lax
x-content-type-options: nosniff
x-dns-prefetch-control: off
x-frame-options: DENY
x-xss-protection: 1; mode=block
date: Tue, 28 Nov 2023 22:13:14 GMT
content-type: text/html; charset=utf-8
set-cookie: 1e2670d92730b515ce3a1bb65da45062=d15c9d1648c3a0f52dcf8c1991ce2d19; path=/; HttpOnly; Secure; SameSite=None

Hopefully this blog was helpful in explaining how one can reduce the headaches of networking when it comes to providing a proof of concept of OpenShift that needs to be portable and yet simple without reinstalling OpenShift. Using stock Red Hat Enterprise Linux and firewalld makes it pretty easy to build a NAT gateway and still forward specific traffic to expose what is required. Further, it makes it quite easy for me to carve up a single host and bring it to any one of my friends houses for OpenShift Night.