I have spent a fair amount of time over the years standing up GPU clusters and then, almost immediately, asking myself "okay, but how do I actually know this thing is healthy?" We can check that the GPU Operator is running, that Data Center GPU Manager(DCGM) is reporting metrics, and that nvidia-smi shows the right device count, but none of that tells us whether the NVLink and network fabric can actually move data at the bandwidth training jobs are going to need. That gap is exactly what the NVIDIA Cluster Readiness Engine (CRE) is built to close, and this post walks through standing it up on an example EKS cluster, working through a couple of hurdles along the way, and running an actual NCCL certification against real H100 nodes.
What is the Cluster Readiness Engine
The Cluster Readiness Engine is an OSS Kubernetes-native tool for validating that a GPU cluster is actually ready to run distributed training workloads before hand it off to a data science team. It does this by running certifications, which are Kubernetes custom resources that describe a category of test (communication, diagnostics, or training) and a target set of nodes, and then orchestrating real workloads like NCCL all-reduce tests or small-scale training runs against those nodes to prove the fabric and GPUs behave as expected. CRE builds on Kubeflow Trainer to actually schedule and run these workloads, and it ships its own controller, CRDs, and log profiles to interpret the results into a pass/fail report.
The reason this matters is that GPU cluster failures are rarely obvious. A node can report healthy, pass every liveness probe, and still have a degraded NVLink connection or a misconfigured NCCL environment variable that tanks all-reduce bandwidth. Multiply that by a few hundred nodes and a training job that stalls at 2am, and we start to appreciate having a repeatable, automated way to certify hardware before it becomes someone else's middle of the night problem. With that said let's get started.
Refreshing GitHub Credentials for the Image Pull Secret
CRE's controller image is currently distributed from GitHub Container Registry, which means the first thing we needed was a valid GitHub token with read:packages scope to use as an image pull secret.
gh auth login --web
gh auth refresh -s read:packages
gh auth token
The token we obtain from the output above can be exported into the GITHUB_TOKEN variable.
export GITHUB_TOKEN=<gh token from output above>
Now that we have an auth token we can proceed to the next steps.
Enabling DCGM in the GPU Operator's ClusterPolicy
CRE has an optional DCGM diagnostics category, and to use it we needed DCGM enabled in the GPU Operator's ClusterPolicy custom resource. By default the GPU Operator has this set to false.
% kubectl patch clusterpolicy cluster-policy --type=merge -p '{"spec":{"dcgm":{"enabled":true}}}'
clusterpolicy.nvidia.com/cluster-policy patched
With that in place, DCGM shows up later as an installed, optional component when we check CRE's setup status.
Installing the CRE
CRE ships an installer script that drops a kubectl plugin binary onto our path, which is what gives us the kubectl nvcre and kubectl ncre commands used throughout the rest of this post. The plugin is available for x86 and arm64.
curl -sSL https://github.com/NVIDIA/cluster-readiness-engine/releases/latest/download/installer | bash
First Attempt at Setup Init
With the CLI installed, the next step is kubectl nvcre setup init, which handles installing Kubeflow Trainer as a dependency and then the CRE Helm chart itself. We passed GitHub token in as the image pull secret so the controller image could actually be pulled.
% kubectl nvcre setup init --image-pull-secret $GITHUB_TOKEN
[preflight] Checking prerequisites...
Target cluster
Context: arn:aws:eks:us-east-1:590183901766:cluster/nvidia-cloud-tme-eks
Server: https://E156C2041AEE7DC39F97C38F65D795B0.gr7.us-east-1.eks.amazonaws.com
Phases:
- [deps ] Kubeflow Trainer v2.2.1
- [helm ] CRE Helm chart (ghcr.io/nvidia/cluster-readiness-engine/manager:v0.1.0)
Do you want to proceed? Only 'yes' will be accepted to confirm.
Enter a value: yes
[deps] Installing Kubeflow Trainer v2.2.1...
[deps] Installing Kubeflow Trainer Helm release "kubeflow-trainer" in namespace kubeflow-system...
[namespace] Creating namespace cluster-readiness-engine...
[helm] Created image pull secret "ncrectl-pull-secret" in namespace cluster-readiness-engine.
[helm] Installing CRE Helm release "cluster-readiness-engine" in namespace cluster-readiness-engine...
Release "cluster-readiness-engine" does not exist. Installing it now.
Pulled: ghcr.io/nvidia/cluster-readiness-engine:v0.1.0
Digest: sha256:e749fccca83f43b4107e2adec245f6951f8ffe439edc918bc2512fa622f10a17
Error: unable to build kubernetes objects from release manifest: resource mapping not found for name: "cluster-readiness-engine-metrics-monitor" namespace: "cluster-readiness-engine" from "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"
ensure CRDs are installed first
This is a good example of the kind of things we can run into with any Helm chart that ships a ServiceMonitor resource: the CRE chart includes a metrics monitor that depends on the Prometheus Operator's CRDs, and our EKS cluster didn't have the Prometheus Operator installed at all. The install rolled partway through before failing.
Cleaning Up After the Failed Install
Rather than trying to hand-patch a half-applied Helm release, I reset the environment back to a clean slate so the next install would be a proper first attempt.
kubectl nvcre setup reset
Installing the Prometheus Operator CRDs
The actual fix for the ServiceMonitor error is straightforward: install the Prometheus Operator's CRDs so the monitoring.coreos.com/v1 API group exists on the cluster, even if not running the full Prometheus Operator itself.
% kubectl create -f https://github.com/prometheus-operator/prometheus-operator/releases/download/v0.82.2/stripped-down-crds.yaml
customresourcedefinition.apiextensions.k8s.io/alertmanagerconfigs.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/podmonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/probes.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheusagents.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/scrapeconfigs.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/thanosrulers.monitoring.coreos.com created
The stripped-down-crds.yaml variant is worth calling out here, since it installs just the CRDs without the operator's webhook configurations, which is exactly what we want if all we need is the API types to satisfy a chart's ServiceMonitor object.
Scheduling the Manager onto a GPU Node
While the install was proceeding, we hit a second, unrelated snag: the CRE manager pod was stuck pending because the default scheduling didn't have any affinity toward GPU nodes on my cluster. In this example cluster all the nodes were GPU nodes and there were no available CPU nodes. We can write a small loop to catch the pod in its pending state and patch the deployment with a node affinity requiring nvidia.com/gpu.present as soon as it showed up. This will allow us to get the pod to a running state in our environment. We need to run this loop in another terminal before we start the installation.
% while true; do POD=$(kubectl get pods -n cluster-readiness-engine --field-selector status.phase=Pending -o jsonpath='{.items[0].metadata.name}' 2>/dev/null); if [ ! -z "$POD" ]; then echo "Found pending pod: $POD. Patching deployment..."; kubectl patch deployment cluster-readiness-engine-manager -n cluster-readiness-engine --type='strategic' -p '{"spec":{"template":{"spec":{"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"nvidia.com/gpu.present","operator":"Exists"}]}]}}}}}}}'; break; fi; sleep 2; done
Found pending pod: cluster-readiness-engine-manager-7fd75dfc9-vszks. Patching deployment...
deployment.apps/cluster-readiness-engine-manager patched
This isn't something we need to script for every environment, it was specific to how node selection was configured on this particular EKS cluster, but it's a good illustration of how mixed-node-pool clusters can trip up a controller that has an affinity about where it should run.
Re-running Setup Init
With the Prometheus CRDs in place, the pull secret already cached from the failed attempt, and our one liner patch script running in another terminal, the running setup init again completed cleanly.
% kubectl nvcre setup init --image-pull-secret $GITHUB_TOKEN
[preflight] Checking prerequisites...
Target cluster
Context: arn:aws:eks:us-east-1:590183901766:cluster/nvidia-cloud-tme-eks
Server: https://E156C2041AEE7DC39F97C38F65D795B0.gr7.us-east-1.eks.amazonaws.com
Phases:
- [deps ] Kubeflow Trainer v2.2.1
- [helm ] CRE Helm chart (ghcr.io/nvidia/cluster-readiness-engine/manager:v0.1.0)
Do you want to proceed? Only 'yes' will be accepted to confirm.
Enter a value: yes
[deps] Installing Kubeflow Trainer v2.2.1...
[deps] Installing Kubeflow Trainer Helm release "kubeflow-trainer" in namespace kubeflow-system...
[helm] Created image pull secret "ncrectl-pull-secret" in namespace cluster-readiness-engine.
[helm] Installing CRE Helm release "cluster-readiness-engine" in namespace cluster-readiness-engine...
CRE v0.1.0 initialized successfully.
At this point I feel pretty confident CRE is actually installed correctly.
Validating the Installation
CRE gives us a dedicated status command that checks each component it manages, rather than making us go hunt down CRDs, deployments, and Helm releases individually.
% kubectl nvcre setup status
Component Status
───────────────────────── ──────────
CRE CRDs ✓ installed
CRE Controller ✓ installed
Kubeflow Trainer ✓ installed
Log Profiles ✓ installed
GPU Operator ✓ installed
DCGM (optional) ✓ installed
Status: ready
Every component reports installed, including the optional DCGM row, which confirms that earlier GPU ClusterPolicy patch actually took effect. If everything looks good we can move onto actually asking CRE what it sees on the cluster.
Inspecting Cluster Topology
Before running any certifications, it's worth checking what CRE has discovered about the cluster's hardware and network topology, since that's what determines which certification categories even make sense to run.
% kubectl nvcre cluster info
Platform: aws
GPU: NVIDIA-H100-80GB-HBM3 (h100, 8 GPUs/node)
Nodes: 3 ready
Total GPUs: 24
Three nodes, eight H100s each, 24 GPUs total, all on a single network topology group. That single topology group matters for certification purposes, since it means all three nodes should be able to participate in the same all-reduce test without crossing a slower network boundary.
Listing Available Certification Categories
CRE organizes its tests into categories, each pairing a domain like communication, diagnostics, or training with a specific variant or variety of test to run. CRE also allows for custom test runs for those who have their own tests.
% kubectl nvcre certification list-categories
DOMAIN VARIANT
communication nccl-all-gather
communication nccl-all-reduce
communication nccl-alltoall
communication nccl-loopback
communication nccl-loopback-nvswitch
diagnostics dcgm-level4
training nemotron5-56b
training nemotron5-8b
The communication domain covers NCCL collective operations, which is what actually exercises the NVLink and network fabric. The diagnostics domain runs DCGM's built-in health checks. The training domain runs scaled-down versions of real model training jobs, in this case Nemotron variants, to catch issues that only surface under an actual training workload.
Running a Basic Certification
The simplest way to kick off a certification is the certification run command, pointed at a category, which handles creating the namespace and Certification resource.
% kubectl nvcre certification run --category communication/nccl-all-reduce
Discovered GPU nodes with product: NVIDIA-H100-80GB-HBM3
[namespace] Creating namespace ncrectl-20260820-125722...
Certification ncrectl-20260820-125722 created in namespace ncrectl-20260820-125722.
Categories:
- communication/nccl-all-reduce
To check status:
kubectl get certification ncrectl-20260820-125722 -n ncrectl-20260820-125722
Because this returns immediately, checking progress means separately polling the Certification resource, which is fine but a little clunky if we're just watching a single run.
Running a Certification with Watch
For a more interactive experience, the --wait flag streams the certification's progress in real time until it completes.
% kubectl nvcre certification run --category communication/nccl-all-reduce --wait
Discovered GPU nodes with product: NVIDIA-H100-80GB-HBM3
[namespace] Creating namespace ncrectl-20260819-134139...
Certification ncrectl-20260819-134139 created in namespace ncrectl-20260819-134139.
Categories:
- communication/nccl-all-reduce
[watch] Watching certification progress...
[watch] communication/nccl-all-reduce: InProgress (0s)
[watch] communication/nccl-all-reduce: InProgress (15s)
[watch] communication/nccl-all-reduce: InProgress (30s)
[watch] communication/nccl-all-reduce: InProgress (45s)
[watch] communication/nccl-all-reduce: InProgress (1m0s)
[watch] communication/nccl-all-reduce: InProgress (1m15s)
[watch] communication/nccl-all-reduce: InProgress (1m30s)
[watch] communication/nccl-all-reduce: InProgress (1m45s)
[watch] communication/nccl-all-reduce: InProgress (2m0s)
[watch] communication/nccl-all-reduce: InProgress (2m15s)
[watch] communication/nccl-all-reduce: InProgress (2m30s)
[watch] communication/nccl-all-reduce: InProgress (2m45s)
[watch] communication/nccl-all-reduce: InProgress (3m0s)
[watch] communication/nccl-all-reduce: InProgress (3m15s)
[watch] communication/nccl-all-reduce: InProgress (3m30s)
[watch] communication/nccl-all-reduce: InProgress (3m45s)
[watch] communication/nccl-all-reduce: InProgress (4m0s)
[watch] communication/nccl-all-reduce: InProgress (4m15s)
[watch] communication/nccl-all-reduce: InProgress (4m30s)
[watch] communication/nccl-all-reduce: InProgress (4m45s)
[watch] communication/nccl-all-reduce: InProgress (5m0s)
[watch] communication/nccl-all-reduce: InProgress (5m15s)
[watch] communication/nccl-all-reduce: InProgress (5m30s)
[watch] communication/nccl-all-reduce: InProgress (5m45s)
[watch] communication/nccl-all-reduce: InProgress (6m0s)
[watch] communication/nccl-all-reduce: InProgress (6m15s)
[watch] communication/nccl-all-reduce: InProgress (6m30s)
[watch] communication/nccl-all-reduce: InProgress (6m45s)
[watch] communication/nccl-all-reduce: InProgress (7m0s)
[watch] communication/nccl-all-reduce: InProgress (7m15s)
[watch] communication/nccl-all-reduce: InProgress (7m30s)
[watch] communication/nccl-all-reduce: InProgress (7m45s)
[watch] communication/nccl-all-reduce: InProgress (8m0s)
[watch] communication/nccl-all-reduce: InProgress (8m15s)
[watch] communication/nccl-all-reduce: InProgress (8m30s)
[watch] communication/nccl-all-reduce: InProgress (8m45s)
[watch] communication/nccl-all-reduce: InProgress (9m0s)
[watch] communication/nccl-all-reduce: InProgress (9m15s)
[watch] communication/nccl-all-reduce: InProgress (9m30s)
[watch] communication/nccl-all-reduce: InProgress (9m45s)
[watch] communication/nccl-all-reduce: InProgress (10m0s)
[watch] communication/nccl-all-reduce: InProgress (10m15s)
[watch] communication/nccl-all-reduce: InProgress (10m30s)
[watch] communication/nccl-all-reduce: InProgress (10m45s)
[watch] communication/nccl-all-reduce: InProgress (11m0s)
[watch] communication/nccl-all-reduce: InProgress (11m15s)
[watch] communication/nccl-all-reduce: InProgress (11m30s)
[watch] communication/nccl-all-reduce: InProgress (11m45s)
[watch] communication/nccl-all-reduce: InProgress (12m0s)
[watch] communication/nccl-all-reduce: InProgress (12m15s)
[watch] communication/nccl-all-reduce: InProgress (12m30s)
[watch] communication/nccl-all-reduce: InProgress (12m45s)
[watch] communication/nccl-all-reduce: InProgress (13m0s)
[watch] communication/nccl-all-reduce: InProgress (13m15s)
[watch] communication/nccl-all-reduce: Succeeded (13m22s)
[watch] Certification succeeded. (13m22s)
╔════════════════════════════════════════════════════════════════╗
║ Certification Report ║
╚════════════════════════════════════════════════════════════════╝
Name: ncrectl-20260819-134139
Platform: aws
GPU: h100
Nodes: 3
┌────────────────────────────────────────────────────────────────┐
│ communication/nccl-all-reduce │
├────────────────────────────────────────────────────────────────┤
│ Status: Succeeded │
└────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────┐
│ Summary │
├────────────────────────────────────────────────────────────────┤
│ Categories: 1/1 passed │
│ Failed Nodes: none │
│ Result: PASSED │
└────────────────────────────────────────────────────────────────┘
After thirteen minutes for a full NCCL all-reduce sweep across all three nodes, the ending shows a clean PASSED result with zero failed nodes. That's the whole point of this exercise. A certification like this is what tells us the fabric between these three nodes is actually production-ready, not just that the nodes are individually healthy.
Pulling a Report After the Fact
If we want to look at the detailed report again later, or hand it to someone else without re-running the whole certification, the certification report command re-renders it from the stored resource.
% kubectl nvcre certification report ncrectl-20260819-134139 -n ncrectl-20260819-134139
╔════════════════════════════════════════════════════════════════╗
║ Certification Report ║
╚════════════════════════════════════════════════════════════════╝
Name: ncrectl-20260819-134139
Platform: aws
GPU: h100
Nodes: 3
┌────────────────────────────────────────────────────────────────┐
│ communication/nccl-all-reduce │
├────────────────────────────────────────────────────────────────┤
│ Status: Succeeded │
│ Runtime: 13m 21s │
│ Scale: full-scale │
│ Nodes/Job: 3 │
│ Jobs: 1 │
│ │
│ Bandwidth: │
│ Size AlgBW BusBW Samples │
│ 16 GB 180.33 GB/s 345.61 GB/s 13 │
└────────────────────────────────────────────────────────────────┘
┌────────────────────────────────────────────────────────────────┐
│ Summary │
├────────────────────────────────────────────────────────────────┤
│ Categories: 1/1 passed │
│ Failed Nodes: none │
│ Result: PASSED │
└────────────────────────────────────────────────────────────────┘
This time the report includes the actual bandwidth numbers, 180.33 GB/s algorithmic bandwidth and 345.61 GB/s bus bandwidth on a 16 GB message size. Those are the numbers I'd actually want to compare against expected H100 NVLink and network fabric specs, and against future certification runs, to catch any regression before a training job does.
Writing a Custom Certification Manifest
While the built-in categories cover the common cases, CRE also lets us compose our own Certification resource targeting specific categories and a node selector. I put together a manifest targeting a set of nodes I'd tagged for NCCL debugging, running all three loopback and all-reduce variants together.
% cat test-render.yaml
apiVersion: cre.nvidia.com/v1alpha1
kind: Certification
metadata:
name: nccl-debug-19-08
namespace: default
spec:
categories:
- domain: communication
variant: nccl-loopback
- domain: communication
variant: nccl-loopback-nvswitch
- domain: communication
variant: nccl-all-reduce
target:
nodeSelector:
excalibur.nvidia.com/badnode: nccl
nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3
I always prefer yaml manifests so the following steps will show that method, rather than composing everything through CLI flags.
Rendering the Certification Without Running It
Before actually applying a custom certification, CRE lets us render it to see exactly what Workflow resources it would generate underneath the hood, which is a great way to sanity-check a manifest without spinning up real jobs.
% kubectl nvcre certification render test-render.yaml
apiVersion: cre.nvidia.com/v1alpha1
kind: Workflow
metadata:
labels:
app.kubernetes.io/managed-by: cluster-readiness-engine
cre.nvidia.com/category-domain: communication
cre.nvidia.com/category-variant: nccl-loopback
cre.nvidia.com/certification: nccl-debug-19-08
name: nccl-debug-19-08-communic-6b70d
spec:
dependencies:
- apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainingRuntime
metadata:
labels:
app: nccl-loopback
name: nccl-loopback-runtime
spec:
mlPolicy:
numNodes: 1
torch:
numProcPerNode: 1
template:
spec:
replicatedJobs:
- name: node
template:
metadata:
labels:
trainer.kubeflow.org/trainjob-ancestor-step: trainer
spec:
template:
spec:
containers:
- image: nvcr.io/nvidia/pytorch:26.01-py3
name: node
resources:
limits:
nvidia.com/gpu: "8"
requests:
nvidia.com/gpu: "8"
securityContext:
capabilities:
add:
- IPC_LOCK
restartPolicy: OnFailure
tolerations:
- operator: Exists
jobTemplate:
spec:
bandwidthMeasurement:
logProfileRef: nccl-loopback
sampleInterval: 30s
testType: loopback_all_reduce
nodeHealthMonitor:
cel:
expression: node.spec.unschedulable == true
workload:
trainJob:
runtimePatches:
- manager: cre.nvidia.com/catalog
trainingRuntimeSpec:
template:
spec:
replicatedJobs:
- name: node
template:
spec:
template:
spec:
containers:
- name: node
volumeMounts:
- mountPath: /dev/shm
name: dshm
volumes:
- emptyDir:
medium: Memory
name: dshm
runtimeRef:
kind: TrainingRuntime
name: nccl-loopback-runtime
trainer:
args:
- -b
- "8"
- -e
- 16G
- -f
- "2"
- -g
- "8"
- -n
- "10"
- -N
- "2"
command:
- /usr/local/bin/all_reduce_perf_mpi
env:
- name: NCCL_DEBUG
value: INFO
- name: NCCL_SHM_DISABLE
value: "1"
- name: NCCL_P2P_DISABLE
value: "1"
- name: NCCL_MNNVL_ENABLE
value: "0"
image: nvcr.io/nvidia/pytorch:26.01-py3
numNodes: 1
numProcPerNode: 1
orchestration:
execution: {}
iterations: 1
target:
nodeSelector:
excalibur.nvidia.com/badnode: nccl
nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3
---
apiVersion: cre.nvidia.com/v1alpha1
kind: Workflow
metadata:
labels:
app.kubernetes.io/managed-by: cluster-readiness-engine
cre.nvidia.com/category-domain: communication
cre.nvidia.com/category-variant: nccl-loopback-nvswitch
cre.nvidia.com/certification: nccl-debug-19-08
name: nccl-debug-19-08-communic-beb95
spec:
dependencies:
- apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainingRuntime
metadata:
labels:
app: nccl-loopback-nvswitch
name: nccl-loopback-nvswitch-runtime
spec:
mlPolicy:
numNodes: 1
torch:
numProcPerNode: 1
template:
spec:
replicatedJobs:
- name: node
template:
metadata:
labels:
trainer.kubeflow.org/trainjob-ancestor-step: trainer
spec:
template:
spec:
containers:
- image: nvcr.io/nvidia/pytorch:26.01-py3
name: node
resources:
limits:
nvidia.com/gpu: "8"
requests:
nvidia.com/gpu: "8"
securityContext:
capabilities:
add:
- IPC_LOCK
restartPolicy: OnFailure
tolerations:
- operator: Exists
jobTemplate:
spec:
bandwidthMeasurement:
logProfileRef: nccl-loopback
sampleInterval: 30s
testType: loopback_all_reduce
nodeHealthMonitor:
cel:
expression: node.spec.unschedulable == true
workload:
trainJob:
runtimePatches:
- manager: cre.nvidia.com/catalog
trainingRuntimeSpec:
template:
spec:
replicatedJobs:
- name: node
template:
spec:
template:
spec:
containers:
- name: node
volumeMounts:
- mountPath: /dev/shm
name: dshm
volumes:
- emptyDir:
medium: Memory
name: dshm
runtimeRef:
kind: TrainingRuntime
name: nccl-loopback-nvswitch-runtime
trainer:
args:
- -b
- "8"
- -e
- 16G
- -f
- "2"
- -g
- "8"
- -n
- "100"
- -N
- "10"
command:
- /usr/local/bin/all_reduce_perf_mpi
env:
- name: NCCL_DEBUG
value: INFO
- name: NCCL_MNNVL_ENABLE
value: "0"
image: nvcr.io/nvidia/pytorch:26.01-py3
numNodes: 1
numProcPerNode: 1
orchestration:
execution: {}
iterations: 1
target:
nodeSelector:
excalibur.nvidia.com/badnode: nccl
nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3
---
apiVersion: cre.nvidia.com/v1alpha1
kind: Workflow
metadata:
labels:
app.kubernetes.io/managed-by: cluster-readiness-engine
cre.nvidia.com/category-domain: communication
cre.nvidia.com/category-variant: nccl-all-reduce
cre.nvidia.com/certification: nccl-debug-19-08
name: nccl-debug-19-08-communic-c2ad8
spec:
dependencies:
- apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainingRuntime
metadata:
labels:
app: nccl-all-reduce
trainer.kubeflow.org/framework: mpi
name: nccl-all-reduce-runtime
spec:
mlPolicy:
mpi:
mpiImplementation: OpenMPI
numProcPerNode: 8
sshAuthMountPath: /tmp/mpi-ssh-raw
numNodes: 1
template:
spec:
network:
publishNotReadyAddresses: true
replicatedJobs:
- name: node
template:
spec:
template:
spec:
containers:
- args:
- |-
set -x &&
apt-get update &&
apt-get install -y --no-install-recommends openssh-server &&
mkdir -p /var/run/sshd &&
chmod 0755 /var/run/sshd &&
mkdir -p /root/.ssh &&
chmod 700 /root/.ssh &&
cp /tmp/mpi-ssh-raw/* /root/.ssh/ &&
chmod 600 /root/.ssh/id_rsa &&
chmod 644 /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys &&
/usr/sbin/sshd -De
command:
- sh
- -c
image: nvcr.io/nvidia/pytorch:26.01-py3
name: node
readinessProbe:
initialDelaySeconds: 5
tcpSocket:
port: 22
resources:
limits:
nvidia.com/gpu: "8"
requests:
nvidia.com/gpu: "8"
securityContext:
capabilities:
add:
- IPC_LOCK
volumeMounts:
- mountPath: /tmp/mpi-ssh-raw
name: mpi-ssh-auth
readOnly: true
restartPolicy: OnFailure
- dependsOn:
- name: node
status: Ready
name: launcher
template:
metadata:
labels:
trainer.kubeflow.org/trainjob-ancestor-step: trainer
spec:
template:
spec:
containers:
- image: nvcr.io/nvidia/pytorch:26.01-py3
name: node
resources:
limits:
cpu: "2"
memory: 1Gi
volumeMounts:
- mountPath: /tmp/mpi-ssh-raw
name: mpi-ssh-auth
readOnly: true
- mountPath: /root/.ssh
name: ssh-keys
initContainers:
- args:
- |-
set -x &&
cp /tmp/mpi-ssh-raw/* /root/.ssh/ &&
chmod 600 /root/.ssh/id_rsa &&
chmod 644 /root/.ssh/id_rsa.pub /root/.ssh/authorized_keys
command:
- sh
- -c
image: nvcr.io/nvidia/pytorch:26.01-py3
name: fix-ssh-permissions
volumeMounts:
- mountPath: /tmp/mpi-ssh-raw
name: mpi-ssh-auth
readOnly: true
- mountPath: /root/.ssh
name: ssh-keys
restartPolicy: OnFailure
volumes:
- emptyDir: {}
name: ssh-keys
successPolicy:
operator: All
targetReplicatedJobs:
- launcher
jobTemplate:
spec:
bandwidthMeasurement:
logProfileRef: nccl-bandwidth
sampleInterval: 30s
testType: all_reduce
nodeHealthMonitor:
cel:
expression: node.spec.unschedulable == true
workload:
trainJob:
runtimePatches:
- manager: cre.nvidia.com/catalog
trainingRuntimeSpec:
template:
spec:
replicatedJobs:
- name: node
template:
spec:
template:
spec:
containers:
- name: node
volumeMounts:
- mountPath: /dev/shm
name: dshm
volumes:
- emptyDir:
medium: Memory
name: dshm
- name: launcher
runtimeRef:
kind: TrainingRuntime
name: nccl-all-reduce-runtime
trainer:
args:
- -N
- "8"
- --allow-run-as-root
- --mca
- plm_rsh_args
- -o StrictHostKeyChecking=no
- -x
- NCCL_DEBUG=INFO
- -x
- NCCL_MNNVL_ENABLE=0
- /usr/local/bin/all_reduce_perf_mpi
- -b
- "8"
- -e
- 16G
- -f
- "2"
- -n
- "100"
- -N
- "10"
command:
- timeout
- "3600"
- /usr/local/mpi/bin/mpirun
image: nvcr.io/nvidia/pytorch:26.01-py3
numNodes: 1
numProcPerNode: 8
orchestration:
execution:
timeoutPerJob: 1h0m0s
iterations: 1
target:
nodeSelector:
excalibur.nvidia.com/badnode: nccl
nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3
This is genuinely useful to look at even if we never run it, because it shows exactly how much CRE is doing on our behalf. The nccl-all-reduce variant, for instance, spins up an MPI cluster from scratch, including generating SSH keys, fixing permissions in an init container, and starting sshd in the worker pods so mpirun can launch across nodes, all before it ever gets to running all_reduce_perf_mpi. Writing that boilerplate by hand for every certification run is exactly the kind of toil CRE exists to eliminate.
Applying a Custom Certification and Watching It Fail
I also had another gpu-cluster-cert.yaml manifest targeting any node with nvidia.com/gpu.present: "true" but only 2 nodes used.
% cat gpu-cluster-cert.yaml
apiVersion: cre.nvidia.com/v1alpha1
kind: Certification
metadata:
name: gpu-cluster-cert
namespace: default
spec:
target:
nodeSelector:
nvidia.com/gpu.present: "true"
nodesPerJob: 2
categories:
- domain: communication
variant: nccl-all-reduce
% kubectl create -f gpu-cluster-cert.yaml
certification.cre.nvidia.com/gpu-cluster-cert created
We can monitor the process by looking at the status conditions.
% kubectl get certification gpu-cluster-cert -o jsonpath='{.status.conditions}' | jq .
[
{
"lastTransitionTime": "2026-08-20T17:29:30Z",
"message": "Certification in progress",
"observedGeneration": 1,
"reason": "WorkflowRunning",
"status": "True",
"type": "InProgress"
},
{
"lastTransitionTime": "2026-08-20T17:29:30Z",
"message": "",
"observedGeneration": 1,
"reason": "NotApplicable",
"status": "False",
"type": "Succeeded"
},
{
"lastTransitionTime": "2026-08-20T17:29:30Z",
"message": "",
"observedGeneration": 1,
"reason": "NotApplicable",
"status": "False",
"type": "Failed"
}
]
This is a nice example of CRE's status conditions doing exactly what Kubernetes status conditions are supposed to do.
Checking Pods and Cleaning Up a Certification
While the certification was in progress, I checked in on the actual pods it spun up, and then confirmed there were no failed nodes before tearing it down.
% kubectl get pods
NAME READY STATUS RESTARTS AGE
gpu-cluster-cert-communic-8b9-78c54-workload-launcher-0-0-d868t 1/1 Running 0 68s
gpu-cluster-cert-communic-8b9-78c54-workload-node-0-0-xp5cw 1/1 Running 0 81s
gpu-cluster-cert-communic-8b9-78c54-workload-node-0-1-sc9fl 1/1 Running 0 81s
gpu-devicequery 0/1 Completed 0 91d
bschmaus@FVXW4M2W1Y aws-cluster % kubectl get certification gpu-cluster-cert -o jsonpath='{.status.failedNodes}'
bschmaus@FVXW4M2W1Y aws-cluster % kubectl delete certification gpu-cluster-cert
certification.cre.nvidia.com "gpu-cluster-cert" deleted from default namespace
We can see the launcher and two node pods that the MPI-based nccl-all-reduce workflow generates, matching exactly what we saw earlier in the rendered manifest. The empty response from failedNodes confirms nothing was flagged as unhealthy, so I deleted the Certification resource once I was satisfied, which cleans up the underlying Workflow, Job, and pod resources along with it.
Tearing Down CRE Entirely
Once we were done experimenting, we can reset the whole environment back to a clean state, which removes CRE's custom resources, the Helm release and CRDs, and the Kubeflow Trainer dependency it installed along the way.
% kubectl ncre setup reset
[preflight] Checking prerequisites...
Target cluster
Context: arn:aws:eks:us-east-1:590183901766:cluster/nvidia-cloud-tme-eks
Server: https://E156C2041AEE7DC39F97C38F65D795B0.gr7.us-east-1.eks.amazonaws.com
Phases:
- [cr ] CRE custom resources
- [helm ] CRE Helm release (CRDs, controller, LogProfiles)
- [deps ] Kubeflow Trainer v2.2.1
Do you want to proceed? Only 'yes' will be accepted to confirm.
Enter a value: yes
[cr] Deleting CRE custom resources...
BandwidthMeasurement/gpu-cluster-cert-communic-8b9-78c54-bandwidth (namespace: default) deleted
BandwidthMeasurement/gpu-cluster-cert-communic-8b9-f2d8e-bandwidth (namespace: default) deleted
BandwidthMeasurement/ncrectl-20260820-115745-c-81371-job-bandwidth (namespace: ncrectl-20260820-115745) deleted
BandwidthMeasurement/ncrectl-20260820-125722-c-ec346-job-bandwidth (namespace: ncrectl-20260820-125722) deleted
Certification/gpu-cluster-cert (namespace: default) deleted
Certification/ncrectl-20260820-115745 (namespace: ncrectl-20260820-115745) deleted
Certification/ncrectl-20260820-125722 (namespace: ncrectl-20260820-125722) deleted
Job/ncrectl-20260820-125722-c-ec346-job (namespace: ncrectl-20260820-125722) already terminating
Workflow/ncrectl-20260820-125722-c-ec346 (namespace: ncrectl-20260820-125722) already terminating
All CRE custom resources removed.
[helm] Removing CRE Helm release "cluster-readiness-engine" from namespace cluster-readiness-engine...
[helm] Removing CRE CRDs...
Waiting for CRDs to be fully removed...
All CRE CRDs removed.
[deps] Removing Kubeflow Trainer v2.2.1...
[deps] Removing Helm release "kubeflow-trainer" from namespace kubeflow-system...
[deps] Removing Kubeflow Trainer CRDs...
Waiting for CRDs to be fully removed...
All Kubeflow Trainer CRDs removed.
[deps] Removing JobSet CRDs...
Waiting for CRDs to be fully removed...
All JobSet CRDs removed.
CRE reset successfully.
Notice the phase list here is different from the install: setup reset explicitly enumerates every leftover custom resource it finds, including a couple of BandwidthMeasurement and Certification objects from earlier runs I'd forgotten were still lying around, before working its way back down through the Helm release, CRE's CRDs, and finally Kubeflow Trainer's own CRDs including JobSet. That's a genuinely clean teardown, and it's reassuring that the tool tracks its own dependency chain well enough to unwind it in the right order.
Wrapping Up
Between the GitHub token refresh, the Prometheus Operator CRDs, and the node affinity patch for the manager pod, this wasn't a completely frictionless install, but none of those issues were CRE's fault so much as normal artifacts of running on an EKS cluster that wasn't purpose-built for it. Once past setup, the actual certification workflow, from listing categories to running a full NCCL all-reduce test to reading real bandwidth numbers out of the report, was exactly the kind of repeatable validation I'd want between a new cluster and a production training job. Hopefully this write-up gives a good sense of what the Cluster Readiness Engine does, why validating fabric health matters as much as validating hardware inventory, and what to expect the first time it runs on a cluster.
For more information please refer to the official documentation.
