In a previous blog on NVKE, which is published here, we showed how to deploy an NVIDIA Kubernetes Engine (NVKE) cluster. A Kubernetes cluster is only as reachable as its API endpoint, and with NVKE that endpoint is a virtual IP (VIP) that has to follow the control plane no matter which node happens to be healthy. In that deployment kube-vip keeps the VIP alive by announcing it to the network with the Border Gateway Protocol (BGP), which means the upstream switch has to be configured to peer with the nodes and accept the route.
The more familiar way to handle a VIP is at the layer 2 level. Red Hat OpenShift is a good example, where HAProxy load balances the API traffic across the control plane nodes and keepalived uses the Virtual Router Redundancy Protocol (VRRP) to float a VIP between them. One node owns the address at a time and answers ARP for it, and when that node fails another one takes over and sends a gratuitous ARP so the neighbors update their caches. It is simple and it needs nothing from the network, but it comes with two constraints. Every node has to sit on the same layer 2 segment, and only one node carries the traffic for the VIP at any given moment, while the others wait their turn.
BGP removes both constraints. At its core it is just two routers agreeing to tell each other which prefixes they can reach. Each control plane node announces the VIP as a /32 route, and the upstream router learns that the address lives behind any of those nodes. The VIP becomes a routed address instead of an ARP trick, so the router can hold a route to every node, withdraw the one that goes away and, with equal cost multi-path, spread connections across all of them. It also means the cluster is no longer tied to the layer 2 layout of whatever rack it lands in. The price is that the network has to take part in the conversation, which is why the switch needs configuration. This write-up covers that switch side of the setup, using a Cisco 3750 in the lab.
Lab Overview
The lab is small with a Cisco Catalyst 3750 acting as the router and the BGP peer. Behind it sits a single hypervisor host running three virtual machines, asus3-vm1, asus3-vm2 and asus3-vm3, all attached to the 192.168.0.0/24 network. The switch owns 192.168.0.1 on that network, and the three VMs sit at 192.168.0.116, 192.168.0.117 and 192.168.0.118.
The cluster VIP is 10.250.0.10, and it lives outside of 192.168.0.0/24. That is the whole point. Nothing on the lab network has a route to 10.250.0.10/32 until the cluster nodes announce it, so the switch has to be configured to listen for that one prefix from each VM. Once the switch learns it, anything that can reach the 3750 can reach the Kubernetes API through 10.250.0.10.
NVKE ASN Modification
Before getting to the switch configuration there is one caveat worth calling out. BGP identifies each side of a peering with an Autonomous System Number (ASN), and for this lab the switch uses 65000 while the cluster nodes use 65100. Both are in the private range, which is what a lab like this should use. The NVKE code in its current state has the ASN hard coded to a value larger than what the Cisco 3750 can accept, so the peering would never have come up. For this walk-through the NVKE code was modified to allow 65000 and 65100 instead. That is a local change made to fit the hardware in my lab and not something to expect from a stock NVKE install. There are plans to make that ASN number configurable.
Update Cisco Switch
Let's get started with the switch. The first block enters configuration mode and builds two prefix lists before anything else. NVKE-DEMO-VIP-IN permits exactly one prefix, 10.250.0.10/32, which is the VIP and nothing else. NVKE-DEMO-DENY-OUT denies everything, so the switch will not leak any of its own routes into the cluster.
With the filters in place the block starts BGP with the local ASN of 65000 and sets the router ID to 192.168.0.1. It turns on neighbor change logging so session state changes land in the log, and it turns off synchronization and auto-summary, which keeps the router from summarizing the /32 into something larger. The line maximum-paths 3 is the one that matters most for NVKE. It allows the switch to install up to three equal cost routes to the VIP, one per control plane node, instead of picking a single winner.
The rest of the block defines the three neighbors, 192.168.0.116, 192.168.0.117 and 192.168.0.118. Each one gets the remote ASN of 65100, a description naming the VM it belongs to, and the two prefix lists applied inbound and outbound. Every neighbor gets identical treatment, so adding a fourth node later would just be another copy of the same four lines.
cisco3750#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
cisco3750(config)#ip prefix-list NVKE-DEMO-VIP-IN seq 10 permit 10.250.0.10/32
cisco3750(config)#ip prefix-list NVKE-DEMO-DENY-OUT seq 10 deny 0.0.0.0/0 le 32
cisco3750(config)#router bgp 65000
cisco3750(config-router)# bgp router-id 192.168.0.1
cisco3750(config-router)# bgp log-neighbor-changes
cisco3750(config-router)# no synchronization
cisco3750(config-router)# no auto-summary
cisco3750(config-router)# maximum-paths 3
cisco3750(config-router)# neighbor 192.168.0.116 remote-as 65100
cisco3750(config-router)# neighbor 192.168.0.116 description NVKE-asus3-vm1
cisco3750(config-router)# neighbor 192.168.0.116 prefix-list NVKE-DEMO-VIP-IN in
cisco3750(config-router)# neighbor 192.168.0.116 prefix-list NVKE-DEMO-DENY-OUT out
cisco3750(config-router)# neighbor 192.168.0.117 remote-as 65100
cisco3750(config-router)# neighbor 192.168.0.117 description NVKE-asus3-vm2
cisco3750(config-router)# neighbor 192.168.0.117 prefix-list NVKE-DEMO-VIP-IN in
cisco3750(config-router)# neighbor 192.168.0.117 prefix-list NVKE-DEMO-DENY-OUT out
cisco3750(config-router)# neighbor 192.168.0.118 remote-as 65100
cisco3750(config-router)# neighbor 192.168.0.118 description NVKE-asus3-vm3
cisco3750(config-router)# neighbor 192.168.0.118 prefix-list NVKE-DEMO-VIP-IN in
cisco3750(config-router)# neighbor 192.168.0.118 prefix-list NVKE-DEMO-DENY-OUT out
cisco3750(config-router)#end
At this point the configuration is in the running config. We can verify it by checking the BGP summary.
Checking the BGP Summary
The summary shows the switch as router identifier 192.168.0.1 in local AS 65000, with all three neighbors listed against AS 65100. Each one reports State as Active and a session that has never come up. In BGP, Active does not mean the session is healthy. It means the switch is trying to open a TCP connection to the neighbor and has not succeeded. That is exactly what is expected when the other side has not started speaking BGP yet, and it confirms the neighbors are defined correctly and the switch is waiting.
cisco3750#show ip bgp summary
BGP router identifier 192.168.0.1, local AS number 65000
BGP table version is 197, main routing table version 197
1 network entries using 117 bytes of memory
3 path entries using 156 bytes of memory
1 multipath network entries and 3 multipath paths
2/1 BGP path/bestpath attribute entries using 280 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 577 total bytes of memory
BGP activity 4/3 prefixes, 100/97 paths, scan interval 60 secs
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
192.168.0.116 4 65100 76549 74720 197 0 0 03:45:48 1
192.168.0.117 4 65100 68916 67270 197 0 0 02:10:41 1
192.168.0.118 4 65100 68932 67290 197 0 0 03:44:03 1
Checking a Single Neighbor
For a closer look, the neighbor detail for 192.168.0.116 shows the peer described as NVKE-asus3-vm1 in remote AS 65100, an external link, with the remote router ID still 0.0.0.0 because no OPEN message has been received. The message statistics are all zeros, the hold time is 180 seconds with a keepalive interval of 60, and the last line reports that there is no active TCP connection.
The useful lines are further down. Under the IPv4 Unicast address family the incoming filter shows up as NVKE-DEMO-VIP-IN and the outgoing filter as NVKE-DEMO-DENY-OUT, which confirms the prefix lists attached to the neighbor the way they were written. The address tracking line also notes that the routing table has a route to 192.168.0.116, so the switch knows how to reach the VM and the only thing missing is the VM answering.
cisco3750#show ip bgp neighbors 192.168.0.116
BGP neighbor is 192.168.0.116, remote AS 65100, external link
Description: NVKE-asus3-vm1
BGP version 4, remote router ID 192.168.0.116
BGP state = Established, up for 03:45:09
Last read 00:00:09, last write 00:00:07, hold time is 30, keepalive interval is 10 seconds
Neighbor capabilities:
Route refresh: advertised and received(new)
Address family IPv4 Unicast: advertised and received
Message statistics:
InQ depth is 0
OutQ depth is 0
Sent Rcvd
Opens: 35 35
Notifications: 2 0
Updates: 0 35
Keepalives: 74679 76475
Route Refresh: 0 0
Total: 74716 76545
Default minimum time between advertisement runs is 30 seconds
For address family: IPv4 Unicast
BGP table version 197, neighbor version 197/0
Output queue size : 0
Index 1, Offset 0, Mask 0x2
1 update-group member
Incoming update prefix filter list is NVKE-DEMO-VIP-IN
Outgoing update prefix filter list is NVKE-DEMO-DENY-OUT
Sent Rcvd
Prefix activity: ---- ----
Prefixes Current: 0 1 (Consumes 52 bytes)
Prefixes Total: 0 1
Implicit Withdraw: 0 0
Explicit Withdraw: 0 0
Used as bestpath: n/a 1
Used as multipath: n/a 1
Outbound Inbound
Local Policy Denied Prefixes: -------- -------
prefix-list 6 0
Total: 6 0
Number of NLRIs in the update sent: max 0, min 0
Address tracking is enabled, the RIB does have a route to 192.168.0.116
Address tracking requires at least a /24 route to the peer
Connections established 35; dropped 34
Last reset 03:45:12, due to Peer closed the session
Transport(tcp) path-mtu-discovery is enabled
Connection state is ESTAB, I/O status: 1, unread input bytes: 0
Mininum incoming TTL 0, Outgoing TTL 1
Local host: 192.168.0.1, Local port: 179
Foreign host: 192.168.0.116, Foreign port: 39293
Enqueued packets for retransmit: 0, input: 0 mis-ordered: 0 (0 bytes)
Event Timers (current time is 0x2E4F2FC2E):
Timer Starts Wakeups Next
Retrans 1323 0 0x0
TimeWait 0 0 0x0
AckHold 1354 1309 0x0
SendWnd 0 0 0x0
KeepAlive 0 0 0x0
GiveUp 0 0 0x0
PmtuAger 0 0 0x0
DeadWait 0 0 0x0
iss: 3757656863 snduna: 3757682008 sndnxt: 3757682008 sndwnd: 64062
irs: 2772535007 rcvnxt: 2772560808 rcvwnd: 15453 delrcvwnd: 931
SRTT: 300 ms, RTTO: 303 ms, RTV: 3 ms, KRTT: 0 ms
minRTT: 0 ms, maxRTT: 300 ms, ACK hold: 200 ms
Flags: passive open, nagle, path mtu capable, gen tcbs
Datagrams (max data segment is 1460 bytes):
Rcvd: 2677 (out of order: 0), with data: 1354, total data bytes: 25800
Sent: 2649 (retransmit: 0), with data: 1322, total data bytes: 25144
If everything looks good we can move onto saving the work.
Saving the Configuration
A running configuration on a Catalyst vanishes at the next reload, so the last step writes it to memory. The switch builds the configuration and reports OK.
cisco3750#wr mem Building configuration... [OK]
Finally we can look at being able to access the 10.150.0.10 VIP and we can do that from another host somewhere on the network. First we can ping the IP address and see a response but we can also try and access the Kubernetes cluster as well where our kubeconfig is using the 10.150.0.10 address.
$ ping 10.250.0.10
PING 10.250.0.10 (10.250.0.10) 56(84) bytes of data.
64 bytes from 10.250.0.10: icmp_seq=1 ttl=64 time=0.592 ms
From 192.168.0.1: icmp_seq=1 Redirect Host(New nexthop: 192.168.0.116)
64 bytes from 10.250.0.10: icmp_seq=2 ttl=64 time=0.459 ms
From 192.168.0.1: icmp_seq=2 Redirect Host(New nexthop: 192.168.0.116)
64 bytes from 10.250.0.10: icmp_seq=3 ttl=64 time=0.573 ms
From 192.168.0.1: icmp_seq=3 Redirect Host(New nexthop: 192.168.0.116)
^C
--- 10.250.0.10 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 0.459/0.541/0.592/0.058 ms
$ cat /home/bschmaus/nvke-demo.kubeconfig | grep server
server: https://10.250.0.10:6443
$ export KUBECONFIG=/home/bschmaus/nvke-demo.kubeconfig
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
nvke-demo-cp-4c8bbf0f Ready control-plane 5d19h v1.36.3 192.168.0.117 <none> Ubuntu 24.04.5 LTS 6.8.0-1064-nvidia (amd64) containerd://2.3.0
nvke-demo-cp-90c9dcc2 Ready control-plane 5d19h v1.36.3 192.168.0.118 <none> Ubuntu 24.04.5 LTS 6.8.0-1064-nvidia (amd64) containerd://2.3.0
nvke-demo-cp-f46d7a88 Ready control-plane 5d19h v1.36.3 192.168.0.116 <none> Ubuntu 24.04.5 LTS 6.8.0-1064-nvidia (amd64) containerd://2.3.0
At this point I feel pretty confident the switch side is ready. The prefix lists only allow the VIP in, the three neighbors are defined with the correct ASNs and the switch shows the nodes to connected. Because the cluster nodes are announcing 10.250.0.10/32, the neighbors show Established and the prefix count for each is one.
Hopefully this provided a clear example of what NVKE needs from the network when its control plane VIP is announced over BGP, and how little it takes to prepare an older Cisco 3750 to be the other end of that conversation.
