The following probably seems pretty inconsequential but having used OpenShift for a vast amount of time I figured it might be wise to brush up on other ways to provide a Kubernetes environment. In this case I wanted to build a 3 node HA Kubernetes setup similar to OpenShift's hyper-converged 3 node scenario. I also wanted to move away from the Red Hat Linux which I have been using for 20 years and add some a new spark to my life.
First we started by installing the latest version of Ubuntu 26.04 onto 3 KVM virtual machines. These virtual machines were on a Ampere Arm based system so everything shown here today is Arm based. While we will not be covering the installation of Ubuntu here I can say it was pretty straight forward. It was almost Window-ish in that it was pretty much next->next->finish.
Once we have our 3 virtual machines installed with Ubuntu we want to make sure all 3 systems are updated.
sudo apt update && sudo apt upgrade -y
Next we are going to install some packages we will need in our Kubernetes deployment on all 3 systems. Installing Haproxy and Keepalived brought back a lot of memories from the days when I used these to load balance services back at the University of Minnesota.
$ sudo apt install -y docker.io haproxy keepalived
With the packages installed we decided to first configure Haproxy which will act as our loadbalancer. The following configuration file needs to be created on all 3 nodes and will provide the loadbalancing of the Kubernetes api.
$ cat <<EOF | sudo tee /etc/haproxy/haproxy.cfg
global
stats socket /var/lib/haproxy/run/haproxy.sock mode 600 level admin expose-fd listeners
defaults
maxconn 40000
mode tcp
log /var/run/haproxy/haproxy-log.sock local0 notice alert
log-format "%ci:%cp -> %fi:%fp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq"
option dontlognull
retries 3
timeout http-request 30s
timeout queue 1m
timeout connect 10s
timeout client 86400s
timeout server 86400s
timeout tunnel 86400s
timeout client-fin 1s
timeout server-fin 1s
frontend main
bind :::9445 v4v6
default_backend masters
listen health_check_http_url
bind :::9444 v4v6
mode http
monitor-uri /haproxy_ready
option dontlognull
listen monitor_check_http_url
bind :::9454 v4v6
mode http
monitor-uri /haproxy_monitor
monitor fail if { nbsrv(masters) lt 1 }
option dontlognull
listen stats
bind localhost:29445
mode http
stats enable
stats hide-version
stats uri /haproxy_stats
stats refresh 30s
stats auth Username:Password
backend masters
timeout check 10s
option httpchk GET /readyz HTTP/1.0
balance roundrobin
server adlink-vm1 192.168.0.128:6443 weight 1 verify none check check-ssl inter 5s fall 3 rise 1
server adlink-vm2 192.168.0.129:6443 weight 1 verify none check check-ssl inter 5s fall 3 rise 1
server adlink-vm3 192.168.0.130:6443 weight 1 verify none check check-ssl inter 5s fall 3 rise 1
EOF
Once the haproxy.cfg has been created on all the nodes we can go ahead and enable, start and check status of the Haproxy service on all 3 nodes.
$ sudo systemctl enable haproxy
Synchronizing state of haproxy.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable haproxy
$ sudo systemctl start haproxy
$ sudo systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; preset: enabled)
Active: active (running) since Fri 2026-05-29 15:49:39 UTC; 1h 53min ago
Invocation: 02fc45c6707c4c5b8b0c9b1106683e48
Docs: man:haproxy(1)
file:/usr/share/doc/haproxy/configuration.txt.gz
Main PID: 15789 (haproxy)
Status: "Ready."
Tasks: 9 (limit: 32645)
Memory: 46.6M (peak: 47.3M)
CPU: 119ms
CGroup: /system.slice/haproxy.service
├─15789 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
└─15791 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
May 29 15:49:39 adlink-vm1 systemd[1]: Starting haproxy.service - HAProxy Load Balancer...
May 29 15:49:39 adlink-vm1 haproxy[15789]: [NOTICE] (15789) : Initializing new worker (15791)
May 29 15:49:39 adlink-vm1 haproxy[15789]: [NOTICE] (15789) : Loading success.
May 29 15:49:39 adlink-vm1 systemd[1]: Started haproxy.service - HAProxy Load Balancer.
With Haproxy running we can turn our attention to Keepalived. Keepalived is used to manage the virtual ip addresses that would need to float across the underlying physical nodes. The example below is used for both the api vip and the ingress vip. It should be noted that on each node the unicast_src_ip needs to the be the primary ip address of the node and the peers are the ip address of the other two nodes in the cluster. The interface it should run on should also be updated based on the environment.
$ cat <<EOF | sudo tee /etc/keepalived/keepalived.conf
global_defs {
enable_script_security
script_user root
max_auto_priority -1
vrrp_garp_master_refresh 60
}
vrrp_instance api {
state BACKUP
interface enp1s0
virtual_router_id 55
priority 40
advert_int 1
unicast_src_ip 192.168.0.128
unicast_peer {
192.168.0.129
192.168.0.130
}
authentication {
auth_type PASS
auth_pass api_vip
}
virtual_ipaddress {
192.168.0.134/32 label vip
}
}
vrrp_instance ingress {
state BACKUP
interface enp1s0
virtual_router_id 166
priority 20
advert_int 1
unicast_src_ip 192.168.0.128
unicast_peer {
192.168.0.129
192.168.0.130
}
authentication {
auth_type PASS
auth_pass ingress_vip
}
virtual_ipaddress {
192.168.0.135/32 label vip
}
}
EOF
With the keepalived.conf created on all 3 nodes we can go ahead and enable, start and check status of the Keepalive deamons.
$ sudo systemctl enable keepalived
Synchronizing state of keepalived.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable keepalived
$ sudo systemctl start keepalived
$ sudo systemctl status keepalived
● keepalived.service - Keepalive Daemon (LVS and VRRP)
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; preset: enabled)
Active: active (running) since Fri 2026-05-29 17:46:34 UTC; 5s ago
Invocation: de54948695ec4d21ac1fc9a300c61791
Docs: man:keepalived(8)
man:keepalived.conf(5)
man:genhash(1)
https://keepalived.org
Main PID: 21347 (keepalived)
Tasks: 2 (limit: 32645)
Memory: 5.9M (peak: 6.4M)
CPU: 22ms
CGroup: /system.slice/keepalived.service
├─21347 /usr/sbin/keepalived --dont-fork
└─21348 /usr/sbin/keepalived --dont-fork
May 29 17:46:34 adlink-vm1 Keepalived[21347]: Command line: '/usr/sbin/keepalived' '--dont-fork'
May 29 17:46:34 adlink-vm1 Keepalived[21347]: Configuration file /etc/keepalived/keepalived.conf
May 29 17:46:34 adlink-vm1 Keepalived[21347]: Starting VRRP child process, pid=21348
May 29 17:46:34 adlink-vm1 Keepalived_vrrp[21348]: (/etc/keepalived/keepalived.conf: Line 45) Truncating auth_pass to 8 characters
May 29 17:46:34 adlink-vm1 Keepalived[21347]: Startup complete
May 29 17:46:34 adlink-vm1 systemd[1]: Started keepalived.service - Keepalive Daemon (LVS and VRRP).
May 29 17:46:34 adlink-vm1 Keepalived_vrrp[21348]: (api) Entering BACKUP STATE (init)
May 29 17:46:34 adlink-vm1 Keepalived_vrrp[21348]: (ingress) Entering BACKUP STATE (init)
May 29 17:46:38 adlink-vm1 Keepalived_vrrp[21348]: (api) Entering MASTER STATE
May 29 17:46:38 adlink-vm1 Keepalived_vrrp[21348]: (ingress) Entering MASTER STATE
With Keepalived setup we can move to applying a few additional changes to the 3 nodes themselves. The first one is to ensure swap is turned off in the environment both immediately and across reboots. The following syntax will achieve both.
$ sudo swapoff -a
$ sudo sed -i '/\/swap.img/ s/^/#/' /etc/fstab
~
Next we need to ensure the kernel modules overlay and br_netfilter are loaded at boot time on all 3 nodes.
$ cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF
We also need to manually load those modules now unless we would like to reboot then nodes instead.
$ sudo modprobe overlay; sudo modprobe br_netfilter
We can check that the modules are loaded with the following.
$ lsmod| egrep "overlay|br_netfilter"
br_netfilter 32768 0
bridge 417792 1 br_netfilter
overlay 217088 0
Next we need to set some sysctl variables to persist across reboots on all 3 nodes. Specifically we are setting bridge-nf-calls and ip fowarding.
$ cat <<EOF | sudo tee /etc/sysctl.d/kubernetes.conf
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
EOF
We can force a read of the systemctl values without rebooting by issuing the following command.
$ sudo sysctl --system
(...)
* Applying /etc/sysctl.d/kubernetes.conf ...
(...)
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
Next we need to install Containerd on all 3 nodes.
$ sudo apt install -y containerd
containerd is already the newest version (2.2.2-0ubuntu1).
containerd set to manually installed.
The following packages were automatically installed and are no longer required:
linux-headers-7.0.0-14 linux-image-unsigned-7.0.0-14-generic linux-modules-7.0.0-14-generic linux-tools-7.0.0-14-generic
linux-headers-7.0.0-14-generic linux-main-modules-zfs-7.0.0-14-generic linux-tools-7.0.0-14
Use 'sudo apt autoremove' to remove them.
Summary:
Upgrading: 0, Installing: 0, Removing: 0, Not Upgrading: 0
Once Containerd packages are installed on all the nodes we need to configure the default setup on all 3 nodes.
$ sudo mkdir -p /etc/containerd
$ containerd config default | sudo tee /etc/containerd/config.toml
version = 3
root = '/var/lib/containerd'
state = '/run/containerd'
temp = ''
disabled_plugins = []
required_plugins = []
(...)
[stream_processors.'io.containerd.ocicrypt.decoder.v1.tar.gzip']
accepts = ['application/vnd.oci.image.layer.v1.tar+gzip+encrypted']
returns = 'application/vnd.oci.image.layer.v1.tar+gzip'
path = 'ctd-decoder'
args = ['--decryption-keys-path', '/etc/containerd/ocicrypt/keys']
env = ['OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf']
Next we will update the config we created and set the SystemCgroup to true.
$ sudo sed -i "s/SystemdCgroup =.*/SystemdCgroup = true/" /etc/containerd/config.toml
$ grep SystemdCgroup /etc/containerd/config.toml
SystemdCgroup = true
Finally we can restart Containerd and enable it so its starts on reboots across all 3 nodes.
$ sudo systemctl restart containerd
$ sudo systemctl enable containerd
Now we can move onto setting up Kubernetes proper and the first thing we need are a few packages installed.
$ sudo apt-get install -y apt-transport-https ca-certificates curl gpg
Now we need to go ahead and create the following directory on all 3 nodes which is used to securely store cryptographic GPG/PGP keys for external, third-party software repositories. These keys allow your package manager (apt) to verify that the software you are downloading is authentic, untampered with, and published by a trusted source.
$ sudo mkdir -p -m 755 /etc/apt/keyrings
Next we can pull down the release key for Kubernetes 1.36 and store it in the keyrings directory we created on each of the 3 nodes.
$ curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.36/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
This command adds the official community-owned Kubernetes repository to the Debian/Ubuntu-based nodes. It allows the system's package manager (apt) to find, install, and update Kubernetes packages like kubeadm, kubelet, and kubectl.
$ echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.36/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.36/deb/ /
With the Kubernetes repository setup we can first confirm its listed with the following on all 3 nodes.
$ sudo apt-get update
Hit:1 http://security.ubuntu.com/ubuntu resolute-security InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu resolute InRelease
Get:4 http://us.archive.ubuntu.com/ubuntu resolute-updates InRelease [136 kB]
Get:3 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.36/deb InRelease [1,227 B]
Get:5 https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.36/deb Packages [4,389 B]
Hit:6 http://us.archive.ubuntu.com/ubuntu resolute-backports InRelease
Get:7 http://us.archive.ubuntu.com/ubuntu resolute-updates/main arm64 Packages [154 kB]
Get:8 http://us.archive.ubuntu.com/ubuntu resolute-updates/universe arm64 Packages [73.7 kB]
Fetched 369 kB in 1s (706 kB/s)
Reading package lists... Done
Next we can go ahead and install the following Kubernetes packages on all 3 nodes.
$ sudo apt-get install -y kubelet kubeadm kubectl
We will also want to set a hold on those packages so that they are not automatically updated in the event we do any other patching or upgrading on the 3 nodes.
$ sudo apt-mark hold kubelet kubeadm kubectl
kubelet set on hold.
kubeadm set on hold.
kubectl set on hold.
Next on all 3 nodes we can enable Kubelet.
$ sudo systemctl enable --now kubelet; sudo systemctl enable kubelet
On each node we can also preemptively pull the images we will use down.
$ sudo kubeadm config images pull
[config/images] Pulled registry.k8s.io/kube-apiserver:v1.36.1
[config/images] Pulled registry.k8s.io/kube-controller-manager:v1.36.1
[config/images] Pulled registry.k8s.io/kube-scheduler:v1.36.1
[config/images] Pulled registry.k8s.io/kube-proxy:v1.36.1
[config/images] Pulled registry.k8s.io/coredns/coredns:v1.14.2
[config/images] Pulled registry.k8s.io/pause:3.10.2
[config/images] Pulled registry.k8s.io/etcd:3.6.8-0
We are now finally ready to initialize the first control plane node of our Kubernetes cluster. We can do this with the following ensuring that we are pointing to our api VIP endpoint we configured in Keepalived.
$ sudo kubeadm init --control-plane-endpoint "192.168.0.134:6443" --upload-certs --pod-network-cidr=10.244.0.0/16
[init] Using Kubernetes version: v1.36.1
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [adlink-vm1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.128 192.168.0.134]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [adlink-vm1 localhost] and IPs [192.168.0.128 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [adlink-vm1 localhost] and IPs [192.168.0.128 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "super-admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml"
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests"
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 946.806µs
[control-plane-check] Waiting for healthy control plane components. This can take up to 4m0s
[control-plane-check] Checking kube-apiserver at https://192.168.0.128:6443/livez
[control-plane-check] Checking kube-controller-manager at https://127.0.0.1:10257/healthz
[control-plane-check] Checking kube-scheduler at https://127.0.0.1:10259/livez
[control-plane-check] kube-controller-manager is healthy after 10.64314ms
[control-plane-check] kube-scheduler is healthy after 11.778426ms
[control-plane-check] kube-apiserver is healthy after 2.501924671s
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
096583740846141bbda4c93d3ce39587fd7eddd1a99cf149921682353c6ee1b0
[mark-control-plane] Marking the node adlink-vm1 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node adlink-vm1 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: 9usthb.lyb116njx475fnsc
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Configured RBAC rules to allow the API server kubelet client certificate to access the kubelet API
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes running the following command on each as root:
kubeadm join 192.168.0.134:6443 --token 9usthb.lyb116njx475fnsc \
--discovery-token-ca-cert-hash sha256:ce9ffc6a67c203511652d91fcb7fcccceeb6f59ba399b1e27452d2dc7b864281 \
--control-plane --certificate-key 096583740846141bbda4c93d3ce39587fd7eddd1a99cf149921682353c6ee1b0
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.0.134:6443 --token 9usthb.lyb116njx475fnsc \
--discovery-token-ca-cert-hash sha256:ce9ffc6a67c203511652d91fcb7fcccceeb6f59ba399b1e27452d2dc7b864281
Once the Kubeadm init is complete on the first node it tells us the commands to run on the other two nodes based on whether they are control-plane nodes or worker nodes. In our case we will be making the additional two Ubuntu nodes control-plane node. We can run the following commando on both nodes but we are only showing one node as an example here.
$ sudo kubeadm join 192.168.0.134:6443 --token 9usthb.lyb116njx475fnsc \
--discovery-token-ca-cert-hash sha256:ce9ffc6a67c203511652d91fcb7fcccceeb6f59ba399b1e27452d2dc7b864281 \
--control-plane --certificate-key 096583740846141bbda4c93d3ce39587fd7eddd1a99cf149921682353c6ee1b0
[sudo: authenticate] Password:
[preflight] Running pre-flight checks
[preflight] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[preflight] Use 'kubeadm init phase upload-config kubeadm --config your-config-file' to re-upload it.
W0529 18:44:42.301522 32377 utils.go:69] The recommended value for "bindAddress" in "KubeProxyConfiguration" is: ::; the provided value is: 0.0.0.0
[preflight] Running pre-flight checks before initializing the new control plane instance
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action beforehand using 'kubeadm config images pull'
[download-certs] Downloading the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[download-certs] Saving the certificates to the folder: "/etc/kubernetes/pki"
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [adlink-vm2 localhost] and IPs [192.168.0.129 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [adlink-vm2 localhost] and IPs [192.168.0.129 127.0.0.1 ::1]
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [adlink-vm2 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.129 192.168.0.134]
[certs] Generating "front-proxy-client" certificate and key
[certs] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[certs] Using the existing "sa" key
[kubeconfig] Generating kubeconfig files
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[check-etcd] Checking that the etcd cluster is healthy
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/instance-config.yaml"
[patches] Applied patch of type "application/strategic-merge-patch+json" to target "kubeletconfiguration"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[etcd] Announced new etcd member joining to the existing etcd cluster
[etcd] Creating static Pod manifest for "etcd"
[etcd] Waiting for the new etcd member to join the cluster. This can take up to 40s
[kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 841.444µs
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap
[mark-control-plane] Marking the node adlink-vm2 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node adlink-vm2 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[control-plane-check] Waiting for healthy control plane components. This can take up to 4m0s
[control-plane-check] Checking kube-apiserver at https://192.168.0.129:6443/livez
[control-plane-check] Checking kube-controller-manager at https://127.0.0.1:10257/healthz
[control-plane-check] Checking kube-scheduler at https://127.0.0.1:10259/livez
[control-plane-check] kube-scheduler is healthy after 9.522453ms
[control-plane-check] kube-controller-manager is healthy after 10.606978ms
[control-plane-check] kube-apiserver is healthy after 19.855349ms
This node has joined the cluster and a new control plane instance was created:
* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
* A new etcd member was added to the local/stacked etcd cluster.
To start administering your cluster from this node, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Run 'kubectl get nodes' to see this node join the cluster.
Now that all 3 nodes have been joined to the cluster we need to see what their state is in. We can do this by ssh-ing to any one of the 3 nodes and setting up the kubeconfig access.
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
Once we have the kubeconfig setup we should be able to run our kubectl commands to get the node state. We can see our node state is NotReady. This is because we do not have Coredns pods yet or a CNI.
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
adlink-vm1 NotReady control-plane 21m v1.36.1
adlink-vm2 NotReady control-plane 55s v1.36.1
adlink-vm3 NotReady control-plane 32s v1.36.1
We need to deploy a CNI for our cluster to get to the ready state. To keep things simple we will deploy Flannel here.
~$ kubectl apply -f https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel.yml
namespace/kube-flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
Once Flannel is deployed we should see pods launch on our control-plane nodes in the kube-flannel namespace.
$ kubectl get pods -n kube-flannel
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-flannel kube-flannel-ds-6dglz 0/1 Init:1/2 0 7s
kube-flannel kube-flannel-ds-72pr6 0/1 Init:1/2 0 7s
kube-flannel kube-flannel-ds-s74wp 0/1 Init:1/2 0 7s
We can also take a look at our running pods across the cluster just to confirm everything looks good.
$ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-flannel kube-flannel-ds-6dglz 1/1 Running 0 2d20h
kube-flannel kube-flannel-ds-72pr6 1/1 Running 0 2d20h
kube-flannel kube-flannel-ds-s74wp 1/1 Running 0 2d20h
kube-system coredns-589f44dc88-n5mbw 1/1 Running 0 2d20h
kube-system coredns-589f44dc88-zn7dj 1/1 Running 0 2d20h
kube-system etcd-adlink-vm1 1/1 Running 0 2d20h
kube-system etcd-adlink-vm2 1/1 Running 0 2d20h
kube-system etcd-adlink-vm3 1/1 Running 0 2d20h
kube-system kube-apiserver-adlink-vm1 1/1 Running 0 2d20h
kube-system kube-apiserver-adlink-vm2 1/1 Running 0 2d20h
kube-system kube-apiserver-adlink-vm3 1/1 Running 0 2d20h
kube-system kube-controller-manager-adlink-vm1 1/1 Running 0 2d20h
kube-system kube-controller-manager-adlink-vm2 1/1 Running 0 2d20h
kube-system kube-controller-manager-adlink-vm3 1/1 Running 0 2d20h
kube-system kube-proxy-2hnvq 1/1 Running 0 2d20h
kube-system kube-proxy-mzb6h 1/1 Running 0 2d20h
kube-system kube-proxy-tmzfl 1/1 Running 0 2d20h
kube-system kube-scheduler-adlink-vm1 1/1 Running 0 2d20h
kube-system kube-scheduler-adlink-vm2 1/1 Running 0 2d20h
kube-system kube-scheduler-adlink-vm3 1/1 Running 0 2d20h
Let's see if our node state is now ready.
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
adlink-vm1 Ready control-plane 6m45s v1.36.1
adlink-vm2 Ready control-plane 4m17s v1.36.1
adlink-vm3 Ready control-plane 3m52s v1.36.1
Now that we have our Kubernetes cluster up and running let's launch a BusyBox pod. Below is the example pod yaml we will use.
$ cat busybox-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: default
spec:
containers:
- image: busybox
command:
- sleep
- "3600"
imagePullPolicy: IfNotPresent
name: busybox
restartPolicy: Always
Let's create it on the cluster.
$ kubectl create -f busybox-pod.yaml
pod/busybox created
Wait a minute the pod is sitting in pending.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
busybox 0/1 Pending 0 7s
Let's take a look at the description of the pod to see why its pending. Appears it cannot be scheduled on any nodes.
$ kubectl describe pod busybox
Name: busybox
Namespace: default
Priority: 0
Service Account: default
Node: <none>
Labels: <none>
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Containers:
busybox:
Image: busybox
Port: <none>
Host Port: <none>
Command:
sleep
3600
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-894dl (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
kube-api-access-894dl:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
Optional: false
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 4m17s default-scheduler 0/3 nodes are available: 3 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
The reason the BusyBox pod cannot be scheduled on any node is because all our nodes are control-plane nodes. We do not have any designated worker nodes at this point. However we can address this without the need for additional nodes by untainted our control plane nodes to allow workloads to run on them.
$ kubectl taint nodes adlink-vm1 node-role.kubernetes.io/control-plane:NoSchedule-
node/adlink-vm1 untainted
$ kubectl taint nodes adlink-vm2 node-role.kubernetes.io/control-plane:NoSchedule-
node/adlink-vm2 untainted
$ kubectl taint nodes adlink-vm3 node-role.kubernetes.io/control-plane:NoSchedule-
node/adlink-vm3 untainted
If we check our BusyBox pod again we can see now it went from Pending to Running.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 0 5m33s
Again while this blog might not seem to be bleeding edge it definitely gives a thorough overview of setting up Kubernetes in a 3 node HA scenario.
