In a previous blog I showed how to install and configure a three node high available Kubernetes cluster. That cluster was using Flannel as the CNI for networking. While Flannel is a perfectly acceptable choice there are other alternatives to a CNI on Kubernetes. One of those options is Calico which is an open-source networking and network security solution for Kubernetes. It acts as a Container Network Interface (CNI) plug-in, assigning routable IP addresses to pods and enforcing secure communication rules (Network Policies) to dictate how containers interact with each other and the outside world. It provides the following key features and capabilities:
- Network Policies: It provides granular "zero-trust" access controls, allowing administrators to limit traffic between pods, namespaces, and external endpoints.
- High-Performance Data Planes: Calico supports multiple data planes including the standard Linux kernel (iptables), Windows HNS, and high-performance eBPF.
- Advanced Networking: It can operate natively over Layer 3 using BGP (Border Gateway Protocol) routing for maximum performance without overlay network overhead. It also supports overlay networks (like VXLAN or IP-in-IP) if underlying infrastructure requires it.
- Traffic Security: It supports automatic WireGuard encryption for in-cluster pod traffic to secure data on the wire.
- Observability: Provides deep insights into workload traffic and metrics so developers can quickly identify and troubleshoot network issues.
Calico is popular because it scales well, operates on virtually any Kubernetes platform and is engineered for low CPU consumption. It replaces or enhances the default Kubernetes networking stack, offering a unified platform for both connection management and security.
Calico sounds like a great CNI option so let's get started by installing it on our Kubernetes cluster. First let's confirm flannel is no longer running on our cluster by looking at the currently running pods.
$ kubectl get pods -n kube-flannel
No resources found in kube-flannel namespace.
Calico can be installed via Helm or yaml manifest. I always prefer yaml manifest so the following steps will show that method. Now let's determining the version of Calico to use by visiting the official releases (here)[https://github.com/projectcalico/calico/releases]. As of this writing the latest version is v3.32.0 and that is what we will use.
We can curl the v3.32.0 release for the following manifests:
$ curl -O https://raw.githubusercontent.com/projectcalico/calico/v3.32.0/manifests/v1_crd_projectcalico_org.yaml
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2.90M 100 2.90M 0 0 6.65M 0 0
$ curl -O https://raw.githubusercontent.com/projectcalico/calico/v3.32.0/manifests/tigera-operator.yaml
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 19285 100 19285 0 0 247.7k 0 0
$ curl -O https://raw.githubusercontent.com/projectcalico/calico/v3.32.0/manifests/custom-resources.yaml
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1046 100 1046 0 0 11334 0 0
The manifest we pull down will work as is for the deployment however we need to make one change to the custom-resources.yaml to ensure the CIDR block identified in the ipPools reflects that of our pod network we defined when we created our Kubernetes cluster (10.244.0.0/16). We can use sed to make that inline change.
$ sed -i "s|cidr: 192.168.0.0/16|cidr: 10.244.0.0/16|" custom-resources.yaml
$ cat custom-resources.yaml |grep cidr
cidr: 10.244.0.0/16
Now that we have adjusted the manifests we are now ready to deploy Calico on the cluster. We will start by creating the v1_crd_projectcalico_org.yaml on the cluster.
$ kubectl create -f v1_crd_projectcalico_org.yaml
customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/gatewayapis.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/goldmanes.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/istios.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/managementclusterconnections.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/whiskers.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgpfilters.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
Warning: unrecognized format "cidr"
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/stagedglobalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/stagedkubernetesnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/stagednetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/tiers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusternetworkpolicies.policy.networking.k8s.io created
Next we can deploy the Tigera operator.
$ kubectl create -f tigera-operator.yaml
namespace/tigera-operator created
serviceaccount/tigera-operator created
clusterrole.rbac.authorization.k8s.io/tigera-operator-secrets created
clusterrole.rbac.authorization.k8s.io/tigera-operator created
clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created
rolebinding.rbac.authorization.k8s.io/tigera-operator-secrets created
deployment.apps/tigera-operator created
And finally we can deploy the custom-resources manifest.
$ kubectl create -f custom-resources.yaml
installation.operator.tigera.io/default created
apiserver.operator.tigera.io/default created
goldmane.operator.tigera.io/default created
whisker.operator.tigera.io/default created
We can observe the state of the deployment with the following command.
$ kubectl get tigerastatus -w
NAME AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
apiserver True False False 0s All objects available
calico False False True 20s Pod calico-system/calico-node-crn9x is running but not ready
goldmane False False True 20s Error creating / updating resource: StorageError: key not found, Code: 1, Key: projectcalico.org/networkpolicies/calico-system/calico-system.goldmane, ResourceVersion: 0, AdditionalErrorMsg: <nil>
ippools True False False 21s All objects available
tiers True Waiting for Tigera API server to be ready
whisker False False True 20s Error creating / updating resource: StorageError: key not found, Code: 1, Key: projectcalico.org/networkpolicies/calico-system/calico-system.whisker, ResourceVersion: 0, AdditionalErrorMsg: <nil>
Once the deployment completes we should see that all the Tigera components are now available.
$ kubectl get tigerastatus
NAME AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
apiserver True False False 85s All objects available
calico True False False 70s All objects available
goldmane True False False 50s All objects available
ippools True False False 106s All objects available
tiers True False False 80s All objects available
whisker True False False 70s All objects available
Further we can confirm all the pods are running as well in both the calico-system and tiger-operator namespace.
$ kubectl get pods -A| egrep "calico-system|tigera-operator"
calico-system calico-apiserver-6ccd896b97-nvgvs 1/1 Running 0 5m25s
calico-system calico-apiserver-6ccd896b97-prpn8 1/1 Running 0 5m25s
calico-system calico-kube-controllers-687c745c79-wdgqm 1/1 Running 0 5m24s
calico-system calico-node-bdxb9 1/1 Running 0 5m24s
calico-system calico-node-crn9x 1/1 Running 0 5m24s
calico-system calico-node-f6hrg 1/1 Running 0 5m24s
calico-system calico-typha-56dffbdfbf-hjtkx 1/1 Running 0 5m24s
calico-system calico-typha-56dffbdfbf-xjsbr 1/1 Running 0 5m16s
calico-system csi-node-driver-b4vlk 2/2 Running 0 5m24s
calico-system csi-node-driver-bzr2k 2/2 Running 0 5m24s
calico-system csi-node-driver-lcq5r 2/2 Running 0 5m24s
calico-system goldmane-6885dcb7d-b4jrv 1/1 Running 0 5m25s
calico-system whisker-7c95549979-cjhh8 2/2 Running 0 5m16s
tigera-operator tigera-operator-85dbff4478-dmmsr 1/1 Running 0 5m54s
At this point we have a complete install of Calico on our Kubernetes cluster but we also needs ingress access for our applications. In Calico the ingress is called a GatewayApi and we can create one by using the following manifest.
$ cat <<EOF >gatewayapi.yaml
apiVersion: operator.tigera.io/v1
kind: GatewayAPI
metadata:
name: default
EOF
Once we have the manifest we can create it on the cluster.
$ kubectl create -f gatewayapi.yaml
gatewayapi.operator.tigera.io/default created
We can see from the status that we now have the GatewayApi available.
$ kubectl get tigerastatus
NAME AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
apiserver True False False 20h All objects available
calico True False False 20h All objects available
gatewayapi True False False 7s All objects available
goldmane True False False 20h All objects available
ippools True False False 20h All objects available
tiers True False False 20h All objects available
whisker True False False 20h All objects available
We can also see additional GatewayApi resources are now at our disposal.
$ kubectl api-resources | grep gateway.networking.k8s.io
backendtlspolicies btlspolicy gateway.networking.k8s.io/v1 true BackendTLSPolicy
gatewayclasses gc gateway.networking.k8s.io/v1 false GatewayClass
gateways gtw gateway.networking.k8s.io/v1 true Gateway
grpcroutes gateway.networking.k8s.io/v1 true GRPCRoute
httproutes gateway.networking.k8s.io/v1 true HTTPRoute
referencegrants refgrant gateway.networking.k8s.io/v1beta1 true ReferenceGrant
tcproutes gateway.networking.k8s.io/v1alpha2 true TCPRoute
tlsroutes gateway.networking.k8s.io/v1alpha3 true TLSRoute
udproutes gateway.networking.k8s.io/v1alpha2 true UDPRoute
Now let's put those apis to work for us. The first thing we need to do is create a gateway. In Calico, Gateway API is the standard used to manage traffic rather than legacy Kubernetes Ingress. It utilizes the Gateway custom resource to configure load balancing and routing, backed by an Envoy proxy managed by the Tigera Operator. Below is an example of a Gateway that will allow web traffic on port 8080.
$ cat <<EOF >http-gateway.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: http-deployment-gateway
spec:
gatewayClassName: tigera-gateway-class
listeners:
- name: http
protocol: HTTP
port: 8080
EOF
Let's create our example on our cluster.
$ kubectl create -f http-gateway.yaml
gateway.gateway.networking.k8s.io/http-deployment-gateway created
Now let's create a new namespace where we will run a web server we will call it http-deployment-gateway to match that of the gateway we just created.
$ kubectl create namespace http-deployment-gateway
namespace/http-deployment-gateway created
Next let's define a manifest that does three things, creates the content the web server will deliver, the deployment for the web server which will mount up our Configmap content and finally a service. Notice the web server will run on port 8080, the service will reference 8080 and the content we are serving up is a basic "Hello Calico!" html file.
$ cat <<EOF >http-deployment.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: app-v1-html
namespace: http-deployment-gateway
data:
index.html: |
<html><body><h1>Hello Calico!</h1></body></html>
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: app-v1
namespace: http-deployment-gateway
spec:
replicas: 1
selector:
matchLabels:
app: http-deployment-gateway
version: v1
template:
metadata:
labels:
app: http-deployment-gateway
version: v1
spec:
containers:
- name: busybox
image: busybox:latest
command: ["sh", "-c", "httpd -f -p 8080 -h /var/www/html"]
ports:
- containerPort: 8080
volumeMounts:
- name: html
mountPath: /var/www/html
volumes:
- name: html
configMap:
name: app-v1-html
---
apiVersion: v1
kind: Service
metadata:
name: app-v1
namespace: http-deployment-gateway
spec:
selector:
app: http-deployment-gateway
version: v1
ports:
- port: 8080
targetPort: 8080
EOF
One we have the manifest let's create it on the cluster which created three objects.
$ kubectl create -f http-deployment.yaml
configmap/app-v1-html created
deployment.apps/app-v1 created
service/app-v1 created
We can confirm the pod is running by listing the pods in the http-depoyment-gateway namespace.
$ kubectl get pods -n http-deployment-gateway
NAME READY STATUS RESTARTS AGE
app-v1-6d98784d66-8486g 1/1 Running 0 15s
Now that we have our web server running we need to provide access to it. In Calico, the HTTPRoute kind is a standard Kubernetes Gateway API resource used to define Layer 7 routing rules for incoming HTTP and HTTPS traffic. It is used alongside the Calico Ingress Gateway (which is built on Envoy Gateway) to manage advanced traffic routing, like path-based matching, traffic splitting, and redirects. The below manifest is an example for web server running on port 8080. Here we are just simply redirecting all traffic to the backend application.
$ cat <<EOF >http-route-gateway.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-route-gateway
spec:
parentRefs:
- name: http-deployment-gateway
namespace: default
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: app-v1
namespace: http-deployment-gateway
port: 8080
weight: 100
EOF
Once we have the manifest let's create it on the cluster.
$ kubectl create -f http-route-gateway.yaml
httproute.gateway.networking.k8s.io/http-route-gateway created
Now we need to generate a ReferenceGrant. In Calico, a ReferenceGrant is a native Kubernetes Gateway API resource (gateway.networking.k8s.io). It acts as a security safeguard that explicitly permits cross-namespace references. It is primarily used with the Calico Ingress Gateway. Below is an example ReferenceGrant to allow traffic into our service.
$ cat <<EOF >http-reference-grant.yaml
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
name: http-deployment-gateway
namespace: http-deployment-gateway
spec:
from:
- group: gateway.networking.k8s.io
kind: HTTPRoute
namespace: default
to:
- group: ""
kind: Service
Once we have the manifest we can create it on the cluster.
$ kubectl create -f http-reference-grant.yaml
referencegrant.gateway.networking.k8s.io/http-deployment-gateway created
At this point we are ready to test out everything we have pieced together above. First we need to obtain the svc name and assign it to a variable.
$ HTTPGATEWAY=$(kubectl get svc -n tigera-gateway -o jsonpath='{.items[*].metadata.name}' | tr ' ' '\n'|grep http)
$ echo $HTTPGATEWAY
envoy-default-http-deployment-gateway-f4240cdd
Now we can open up in a terminal with cluster access and run the following port-forward command.
$ kubectl port-forward -n tigera-gateway svc/$HTTPGATEWAY 30135:8080
Forwarding from 127.0.0.1:30135 -> 8080
Forwarding from [::1]:30135 -> 8080
In a secondary terminal on the same host where we ran the port-forward command we can curl the localhost url on port 30135. The results should be the web servers html payload we configured in the configmap above.
$ curl http://localhost:30135/
<html><body><h1>Hello Calico!</h1></body></html>
At this point I feel pretty confident my Calico environment is working and again this is not anything bleeding edge but just good review for us in understanding how Calico works in Kubernetes.
