Recently I worked with a customer concerned about bandwidth, image replication and their edge locations. The customers concerns were warranted because they wanted to mirror a large set of software images to the edge sites but the connectivity to those edge sites while consistent was not necessarily the best for moving large data. Further to compound the problem the connectivity also was shared for other data transmitting services that the edge site relied on during daily business operations. The customer initially requested we add bandwidth capabilities to the software tooling that would be moving the images to the site. While at first glance this would seem to solve the issue it made me realize this might not be a scalable or efficient solution as tools change or as other software requirements for data movement evolve. Understanding the customers requirements and limitations at hand I approach this problem using some tools that are already built into Red Hat Device Edge and Red Hat OpenShift Container Platform. The rest of this blog will explore and demonstrate those options depending on the customers use case being: kubernetes container, non kubernetes container or a standard vanilla process on Linux.
OpenShift Pod Bandwidth Limiting
For OpenShift limiting ingress/egress bandwidth is fairly straight forward given Kubernetes traffic shaping capabilities. In the examples below we will run a basic Red Hat Universal Base Image container two different ways. One way will have no bandwidth restrictions and the other one will have bandwidth restrictions. Then inside each running container we can issue a curl command pulling the same file and see how the behavior differs. It is assumed this container would be the application container issuing the commands at the customer edge location.
Below lets create the normal pod running with no restrictions by first creating the custom resource file and then creating it on the OpenShift cluster.
$ cat << EOF > ~/edgepod-normal.yaml
kind: Pod
apiVersion: v1
metadata:
name: edgepod-normal
namespace: default
labels:
run: edgepod-normal
spec:
restartPolicy: Always
containers:
- resources: {}
stdin: true
terminationMessagePath: /dev/termination-log
stdinOnce: true
name: testpod-normal
imagePullPolicy: Always
terminationMessagePolicy: File
tty: true
image: registry.redhat.io/ubi9/ubi:latest
args:
- sh
EOF
$ oc create -f ~/edgepod-normal.yaml
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "testpod-normal" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "testpod-normal" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "testpod-normal" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "testpod-normal" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
pod/edgepod-normal created
$ oc get pods
NAME READY STATUS RESTARTS AGE
edgepod-normal 1/1 Running 0 5s
Now that we have our normal container running let's go ahead and create the same custom resource file and container with bandwidth restrictions. The custom resource file will be identical to the original one with the exception of the annotations we add around bandwidth for ingress and egress. For the bandwidth we will be restricting to 10M in this example.
$ cat << EOF > ~/edgepod-slow.yaml
kind: Pod
apiVersion: v1
metadata:
name: edgepod-slow
namespace: default
labels:
run: edgepod-normal
annotations: {
"kubernetes.io/ingress-bandwidth": "10M",
"kubernetes.io/egress-bandwidth": "10M"
}
spec:
restartPolicy: Always
containers:
- resources: {}
stdin: true
terminationMessagePath: /dev/termination-log
stdinOnce: true
name: testpod-normal
imagePullPolicy: Always
terminationMessagePolicy: File
tty: true
image: registry.redhat.io/ubi9/ubi:latest
args:
- sh
EOF
$ oc create -f ~/edgepod-slow.yaml
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "testpod-normal" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "testpod-normal" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "testpod-normal" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "testpod-normal" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
pod/edgepod-slow created
$ oc get pods
NAME READY STATUS RESTARTS AGE
edgepod-normal 1/1 Running 0 4m14s
edgepod-slow 1/1 Running 0 3s
Now that both containers are up running let's go into edgepod-normal and run our baseline test curl command.
$ oc exec -it edgepod-normal /bin/bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
[root@edgepod-normal /]# curl http://192.168.0.29/images/discovery_image_agx.iso -o test.iso
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 909M 100 909M 0 0 115M 0 0:00:07 0:00:07 --:--:-- 128M
We can see from the results above that we were able to transfer the 909M file in ~7 seconds with a speed of 128M. Let's run the same command inside our edgepod-slow pod.
$ oc exec -it edgepod-slow /bin/bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
[root@edgepod-slow /]# curl http://192.168.0.29/images/discovery_image_agx.iso -o test.iso
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 909M 100 909M 0 0 1107k 0 0:14:01 0:14:01 --:--:-- 1151k
We can see from the results above that our annotation for bandwidth restricted the container to roughly a 10M speed and it took ~14 minutes to transfer the same 909M file. So if we think back to my customers use case this could be an option for restricting a containers traffic if they are using OpenShift.
Red Hat Enterprise Linux Bandwidth Limiting
In the previous section we looked at how OpenShift can bandwidth limit certain containers running in the cluster. Since the edge has a variety of customers and use cases let's explore how to do the same bandwidth restrictions from a non-kubernetes perspective. We will be using Traffic Control (tc) which is a very useful Linux utility that gives us the ability to control and shape traffic in the kernel. This tool normally ships with a variety of Linux distributions. In our demonstration environment we will be using Red Hat Enterprise Linux 9 since that is the host I have up and running.
First let's go ahead and create a container called edgepod using the ubi9 image.
$ podman run -itd --name edgepod ubi9 bash
Resolved "ubi9" as an alias (/etc/containers/registries.conf.d/001-rhel-shortnames.conf)
Trying to pull registry.access.redhat.com/ubi9:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob d6427437202d done
Copying config 05936a40cf done
Writing manifest to image destination
Storing signatures
906716d99a39c5fc11373739a8aa20e192b348d0aaab2680775fe6ccc4dc00c3
Now let's go ahead and validate that the container is up and running.
$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
906716d99a39 registry.access.redhat.com/ubi9:latest bash 8 seconds ago Up 9 seconds edgepod
Once the container is up and running let's go ahead and run a baseline image pull inside the container to confirm how long it takes to pull the image. We will use the same image we pulled in the OpenShift example above for test.
$ podman exec -it edgepod curl http://192.168.0.29/images/discovery_image_agx.iso -o test.iso
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 909M 100 909M 0 0 52.8M 0 0:00:17 0:00:17 --:--:-- 52.9M
~
We can see from the results above that it took again about ~17 seconds to bring the 909M image over from the source. Now keep in mind this is our baseline.
Next we need to configure the Intermediate Functional Block (ifb) interface on the Red Hat Enterprise Linux host. The ifb pseudo network interface acts as a QoS concentrator for multiple different sources of traffic. We need to use this because tc only works on egress traffic on a real interface and the traffic we are trying to slow down is ingress traffic. To get started we need to load the module into the kernel. We will set the numifbs to one because the default is two and I just need one for my single interface. Once we load the module we can then set the link of the device to up and then confirm the interface is running.
$ sudo modprobe ifb numifbs=1
$ sudo ip link set dev ifb0 up
$ sudo ip address show ifb0
5: ifb0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 32
link/ether b6:5c:67:99:2c:82 brd ff:ff:ff:ff:ff:ff
inet6 fe80::b45c:67ff:fe99:2c82/64 scope link
valid_lft forever preferred_lft forever
Now that the ifb interface is up we need to go ahead and apply some tc rules. The rules are performing the following functions in order:
- Create an egress filter on the ifb0 device
- Add root class htb with rate limiting of 1mbps
- Create a matchall filter to classify all the traffic that runs on the port
- Create ingress on external interface enp1s0
- Forward all ingress traffic from enp1s0 to the ifb0 device
$ sudo tc qdisc add dev ifb0 root handle 1: htb r2q 1
$ sudo tc class add dev ifb0 parent 1: classid 1:1 htb rate 1mbps
$ sudo tc filter add dev ifb0 parent 1: matchall flowid 1:1
$ sudo tc qdisc add dev enp1s0 handle ffff: ingress
$ sudo tc filter add dev enp1s0 parent ffff: protocol ip u32 match u32 0 0 action mirred egress redirect dev ifb0
Now we have our bandwidth limiting capabilities configured let's run our test again and see our results.
$ podman exec -it edgepod curl http://192.168.0.29/images/discovery_image_agx.iso -o test.iso
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 909M 100 909M 0 0 933k 0 0:16:38 0:16:38 --:--:-- 872k
We can see that with our tc rules applied the image was transferred at a much slower rate as expected which would again ensure we are not using all the bandwidth if this were an edge site. Now this might leave some wondering isn't this being applied to the whole system. The answer is yes but if there is not a system wide requirement and only maybe a certain job or task that needs to be rate limited we could wrap the commands into a script, execute the process at hand (our curl command in this example) and then remove the rules with the commands below.
$ tc qdisc del dev enp1s0 ingress
$ tc qdisc del dev ifb0 root
And for sanity sake let's just run our command one more time to confirm we returned to baseline speeds.
$ podman exec -it edgepod curl http://192.168.0.29/images/discovery_image_agx.iso -o test.iso
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 909M 100 909M 0 0 52.0M 0 0:00:17 0:00:17 --:--:-- 46.9M
Hopefully this gives anyone working in edge environments with constrained bandwidth requirements ideas on how they can control certain processes and/or containers from using all the available bandwidth on the edge link. There are obviously a lot of other ways to use these concepts to further enable the most efficient use of the bandwidth availability at the edge but we will save that for some other time.
