Anyone who has run Kubernetes for more than a release or two knows that standing a cluster up is the easy part. The hard part is the lifecycle that follows. A new Kubernetes minor release arrives a few times a year, and each one drags along matching versions of the container runtime, the Container Network Interface (CNI) plugin, the node operating system image and every add-on that watches or manages the cluster. When those versions drift apart from node to node, or from the last time the cluster was built, upgrades turn into archaeology and a rebuild turns into a guess. Versioning has to be a property of how the cluster is built, not something reconstructed from memory after the fact.
That is the gap NVIDIA Kubernetes Engine (NVKE) and NVIDIA Kubernetes Recipes (NVKR) are aimed at filling. Judging from what this walk through exercises, NVKR produces the versioned build artifacts and keeps a lifecycle record for every run that made them, while NVKE consumes those artifacts to turn a set of pre-provisioned nodes into a cluster described by a site spec that pins the Kubernetes version, the CNI version, the network ranges and the identity trust domain. Nothing about the result depends on what happened to be installed on a node the day before. I am describing these two from the angle of this one lab, not as a complete product overview.
In this post I will take the output of a complete run and explain what each command is doing. The goal is to take a set of NVKR build outputs (a node disk image, a set of NVSentinel container images and a Helm chart), bake them into an automatic install ISO, boot three virtual machines from it, bootstrap a three node NVKE cluster and finally deploy NVSentinel onto it using only content that already lives on the nodes. A word of caution before we start. The ISO build output itself labels this as experimental, and the node configuration runs in a lab-plaintext mode with mutual TLS disabled for the bootstrap, so this is a lab exercise and not a production recipe. The command output below is exactly as it came off the systems, and I have not changed any of it.
Starting Point
Let's get started. The first block sets every path and version the rest of the walk through depends on. The three ATTEMPT variables name NVKR runs that had already completed: one that built the Ubuntu 24 node disk image, one that built the NVSentinel container images and one that packaged the NVSentinel Helm chart. IMAGE_NVKE_COMMIT records the exact NVKE source commit the disk image was built from, and BASE_QCOW and CHART_TAR point at the files those runs produced. Setting all of this up front means nothing later has to be typed by hand, and every path can be traced back to a specific run.
$ export NVKR_REPO=/home/bschmaus/nvkr
$ export NVKE_REPO=/home/bschmaus/nvke
$ export AUTO_KIT=/home/bschmaus/nvke-qcow2-auto-iso
$ export WORK=/home/bschmaus/nvke-nvsentinel-iso-002
$ export DISK_ATTEMPT=ubuntu24-lab-a9a0537b-001
$ export IMAGES_ATTEMPT=nvsentinel-images-amd64-002
$ export CHART_ATTEMPT=nvsentinel-chart-amd64-001
$ export BASE_QCOW="$NVKR_REPO/.nvkr/runs/$DISK_ATTEMPT/outputs/nvke-node-image-ubuntu24-amd64.qcow2"
$ export IMAGE_NVKE_COMMIT=a9a0537b4418d08ca86d758ef8e3ba275501150e
$ export SENTINEL_INPUT_REL=.nvkr/lab-inputs/nvsentinel-images-amd64-002
$ export SENTINEL_INPUT_ROOT="$NVKR_REPO/$SENTINEL_INPUT_REL"
$ export CHART_TAR="$NVKR_REPO/.nvkr/runs/$CHART_ATTEMPT/nvsentinel-chart/out/nvsentinel-chart/nvsentinel-chart.tar"
$ export BOOTSTRAP_BUNDLE=/home/bschmaus/nvke-nvsentinel-iso-001/bootstrap-host-bundle.tar
With the variables in place we need somewhere to put things. The next block creates a working tree with a payload directory for the disk image, an nvsentinel directory for the chart and images, and a records directory for evidence. It then writes the NVKR repository commit into that records directory, which gives us a permanent note of which version of the recipe tooling was in use when this build was assembled.
$ mkdir -p "$WORK/payload" "$WORK/nvsentinel/images" "$WORK/records"
$ git -C "$NVKR_REPO" rev-parse HEAD >"$WORK/records/nvkr-commit.txt"
Confirming the NVKR Image Builds
Before anything gets packaged we want to know that every container image NVSentinel could need has been built. This loop reads a list of selected outputs and, for each name, checks the name for illegal characters and looks for an existing NVKR attempt directory. If one exists, it runs nvkr run inspect against that attempt's lifecycle record and asks jq whether the status is complete, whether cleanup finished and whether the image output was actually observed. A completed attempt that still has its OCI archive on disk is skipped, an unfinished one stops the loop so nothing is retried blindly, and anything else would get a fresh retry attempt with a plan and a local build. Here every one of the 28 images reports SKIP, which means each was already built and retained. A few of them, such as gpu-health-monitor-dcgm4-image and preflight-nccl-allreduce-image, point at a -retry3 attempt, which tells me they needed more than one try before they completed.
$ cd "$NVKR_REPO"
$ while IFS= read -r image_name; do
test -n "$image_name" || continue
case "$image_name" in
*[!a-z0-9-]*) echo "Invalid output name: $image_name"; exit 1 ;;
esac
definition="$SENTINEL_INPUT_REL/$image_name.json"
test -f "$definition"
attempt_base="$IMAGES_ATTEMPT-$image_name"
attempt="$attempt_base"
retry=1
while test -d ".nvkr/runs/$attempt"; do
inspection=$(bin/nvkr run inspect \
--record ".nvkr/runs/$attempt/lifecycle.json" --json)
if jq -e --arg name "$image_name" '
.record.status == "complete" and
.record.cleanupIncomplete == false and
any(.record.outputs[]; .name == $name and .state == "observed")
' <<<"$inspection" >/dev/null; then
archive=$(find ".nvkr/runs/$attempt" -type f \
-name '*.oci.tar' -size +0c -print -quit)
if test -n "$archive"; then
printf 'SKIP %s: completed output retained in %s\n' "$image_name" "$attempt"
continue 2
fi
fi
if ! jq -e '
(.record.status == "complete" or .record.status == "failed") and
.record.cleanupIncomplete == false
' <<<"$inspection" >/dev/null; then
echo "STOP: inspect unfinished/incomplete attempt $attempt before retrying"
exit 1
fi
retry=$((retry + 1))
attempt="$attempt_base-retry$retry"
done
plan="$SENTINEL_INPUT_ROOT/$image_name.plan.json"
bin/nvkr run plan --definition "$definition" \
--environment inputs/environments/nvsentinel/image-amd64.yaml --json >"$plan"
plan_bytes=$(wc -c <"$plan")
if test "$plan_bytes" -ge 4194304; then
echo "STOP: $image_name Plan is still too large: $plan_bytes bytes"
exit 1
fi
printf 'BUILD %s (attempt %s)\n' "$image_name" "$attempt"
bin/nvkr run local --definition "$definition" \
--environment inputs/environments/nvsentinel/image-amd64.yaml \
--attempt "$attempt" --runner-source . --jobs 1 --runtime docker
done <"$SENTINEL_INPUT_ROOT/selected-outputs.txt"
SKIP csp-health-monitor-image: completed output retained in nvsentinel-images-amd64-002-csp-health-monitor-image
SKIP event-exporter-image: completed output retained in nvsentinel-images-amd64-002-event-exporter-image
SKIP fault-quarantine-image: completed output retained in nvsentinel-images-amd64-002-fault-quarantine-image
SKIP fault-remediation-image: completed output retained in nvsentinel-images-amd64-002-fault-remediation-image
SKIP file-server-cleanup-image: completed output retained in nvsentinel-images-amd64-002-file-server-cleanup-image
SKIP gpu-health-monitor-dcgm3-image: completed output retained in nvsentinel-images-amd64-002-gpu-health-monitor-dcgm3-image
SKIP gpu-health-monitor-dcgm4-image: completed output retained in nvsentinel-images-amd64-002-gpu-health-monitor-dcgm4-image-retry3
SKIP gpu-reset-image: completed output retained in nvsentinel-images-amd64-002-gpu-reset-image
SKIP health-events-analyzer-image: completed output retained in nvsentinel-images-amd64-002-health-events-analyzer-image
SKIP janitor-image: completed output retained in nvsentinel-images-amd64-002-janitor-image
SKIP janitor-provider-image: completed output retained in nvsentinel-images-amd64-002-janitor-provider-image
SKIP kubernetes-object-monitor-image: completed output retained in nvsentinel-images-amd64-002-kubernetes-object-monitor-image
SKIP labeler-image: completed output retained in nvsentinel-images-amd64-002-labeler-image
SKIP lifecycle-manager-image: completed output retained in nvsentinel-images-amd64-002-lifecycle-manager-image
SKIP log-collector-image: completed output retained in nvsentinel-images-amd64-002-log-collector-image
SKIP maintenance-notifier-image: completed output retained in nvsentinel-images-amd64-002-maintenance-notifier-image
SKIP metadata-collector-image: completed output retained in nvsentinel-images-amd64-002-metadata-collector-image
SKIP nic-health-monitor-image: completed output retained in nvsentinel-images-amd64-002-nic-health-monitor-image
SKIP node-drainer-image: completed output retained in nvsentinel-images-amd64-002-node-drainer-image
SKIP nvcre-certification-monitor-image: completed output retained in nvsentinel-images-amd64-002-nvcre-certification-monitor-image
SKIP platform-connectors-image: completed output retained in nvsentinel-images-amd64-002-platform-connectors-image
SKIP plugins-slinky-drainer-image: completed output retained in nvsentinel-images-amd64-002-plugins-slinky-drainer-image
SKIP preflight-dcgm-diag-image: completed output retained in nvsentinel-images-amd64-002-preflight-dcgm-diag-image
SKIP preflight-image: completed output retained in nvsentinel-images-amd64-002-preflight-image
SKIP preflight-nccl-allreduce-image: completed output retained in nvsentinel-images-amd64-002-preflight-nccl-allreduce-image-retry3
SKIP preflight-nccl-loopback-image: completed output retained in nvsentinel-images-amd64-002-preflight-nccl-loopback-image
SKIP slurm-drain-monitor-image: completed output retained in nvsentinel-images-amd64-002-slurm-drain-monitor-image
SKIP syslog-health-monitor-image: completed output retained in nvsentinel-images-amd64-002-syslog-health-monitor-image
Unpacking and Validating the NVSentinel Chart
At this point the images are accounted for, so the next piece is the Helm chart. This block checks that the chart tarball produced by the NVKR chart run exists and is not empty, lists its contents and extracts it into the working tree. It then confirms that Chart.yaml and Chart.lock came out of the archive and runs helm dependency list to show the subcharts. All 22 of them report unpacked, which means the chart was packaged with its dependencies already vendored and nothing has to be downloaded from a repository later. The last command saves the chart's default values into the records directory for reference.
$ export CHART_TAR="$NVKR_REPO/.nvkr/runs/$CHART_ATTEMPT/nvsentinel-chart/out/nvsentinel-chart/nvsentinel-chart.tar"
$ test -s "$CHART_TAR"
$ tar -tf "$CHART_TAR" | less
$ tar -xf "$CHART_TAR" -C "$WORK/nvsentinel"
$ test -f "$WORK/nvsentinel/nvsentinel/Chart.yaml"
$ test -f "$WORK/nvsentinel/nvsentinel/Chart.lock"
$ helm dependency list "$WORK/nvsentinel/nvsentinel"
$ helm show values "$WORK/nvsentinel/nvsentinel" >"$WORK/records/nvsentinel-default-values.yaml"
NAME VERSION REPOSITORY STATUS
gpu-health-monitor 0.1.0 unpacked
mongodb-store 0.1.0 unpacked
postgresql 15.5.38 unpacked
fault-quarantine 0.1.0 unpacked
node-drainer 0.1.0 unpacked
fault-remediation 0.1.0 unpacked
health-events-analyzer 0.1.0 unpacked
csp-health-monitor 0.1.0 unpacked
syslog-health-monitor 0.1.0 unpacked
incluster-file-server 0.1.0 unpacked
labeler 0.1.0 unpacked
janitor 0.1.0 unpacked
lifecycle-manager 0.1.0 unpacked
janitor-provider 0.1.0 unpacked
metadata-collector 0.1.0 unpacked
kubernetes-object-monitor 0.1.0 unpacked
nvcre-certification-monitor 0.1.0 unpacked
event-exporter 0.1.0 unpacked
preflight 0.1.0 unpacked
k8sdatastore-crds 0.1.0 unpacked
slurm-drain-monitor 0.1.0 unpacked
nic-health-monitor 0.1.0 unpacked
Now we bring in the configuration that decides how much of NVSentinel actually gets deployed. The lab values file comes from the auto-install kit, and helm lint runs the chart against it. The only message is an informational note that the chart has no icon, and the summary shows one chart linted with zero failures.
$ cp "$AUTO_KIT/examples/nvsentinel/values-vm-smoke.yaml" "$WORK/nvsentinel/values-lab.yaml"
$ helm lint "$WORK/nvsentinel/nvsentinel" -f "$WORK/nvsentinel/values-lab.yaml"
==> Linting /home/bschmaus/nvke-nvsentinel-iso-002/nvsentinel/nvsentinel
[INFO] Chart.yaml: icon is recommended
1 chart(s) linted, 0 chart(s) failed
We can verify what that values file does by reading it. It is a smoke-test profile pinned to a specific NVSentinel commit, and it sets the image tag to lab-c15934773572 so the tag in the chart matches the images NVKR built. Nearly every component is switched off, including the GPU health monitor, fault quarantine, node drainer and the MongoDB store. What stays on is the syslog health monitor in STORE_ONLY mode and the platform connector as a DaemonSet with the Prometheus connector enabled. For a three node VM lab with no GPUs, that is a sensible slice that still proves the chart and its images deploy.
$ cat $WORK/nvsentinel/values-lab.yaml
# VM smoke-test profile for NVIDIA/NVSentinel commit
# c1593477357299ab93dd5246f5b2cd236f8f07c8.
# Merge with that chart's defaults; this is not a complete production profile.
global:
image:
tag: lab-c15934773572
gpuHealthMonitor:
enabled: false
healthEventsAnalyzer:
enabled: false
faultQuarantine:
enabled: false
nodeDrainer:
enabled: false
faultRemediation:
enabled: false
janitor:
enabled: false
lifecycleManager:
enabled: false
janitorProvider:
enabled: false
cspHealthMonitor:
enabled: false
syslogHealthMonitor:
enabled: true
labeler:
enabled: false
metadataCollector:
enabled: false
inclusterFileServer:
enabled: false
mongodbStore:
enabled: false
kubernetesObjectMonitor:
enabled: false
nvcreCertificationMonitor:
enabled: false
eventExporter:
enabled: false
preflight:
enabled: false
k8sdatastoreCrds:
enabled: false
slurmDrainMonitor:
enabled: false
nicHealthMonitor:
enabled: false
platformConnector:
daemonset:
enabled: true
deployment:
enabled: false
k8sConnector:
enabled: false
grpcSinkConnector:
enabled: false
promConnector:
enabled: true
syslog-health-monitor:
processingStrategy: STORE_ONLY
postgresql:
enabled: false
podMonitor:
enabled: false
nodeConditionCleanup:
enabled: false
Identifying Which Images the Chart Needs
Rather than trust a hand-written list, this block lets Helm say which images the profile actually uses. helm template renders the whole chart, CRDs included, into a file. A short Python script then walks every document in that file and collects each string stored under an image key, writing the sorted result to images-required.txt. The output lists two images, platform-connectors and syslog-health-monitor, both at the lab-c15934773572 tag. The second script ties those names back to NVKR by finding the matching OCI archive for each one, asserting that the required set has not changed and that each archive exists and is not empty, and then writing a tab separated image-map.tsv with the archive path on the left and the image reference on the right. The two rows printed at the end are that map.
$ helm template nvsentinel "$WORK/nvsentinel/nvsentinel" --namespace nvsentinel --include-crds -f "$WORK/nvsentinel/values-lab.yaml" >"$WORK/records/nvsentinel-rendered.yaml"
$ python3 - "$WORK/records/nvsentinel-rendered.yaml" >"$WORK/records/images-required.txt" <<'PY'
import sys
import yaml
images = set()
def walk(value):
if isinstance(value, dict):
if isinstance(value.get('image'), str):
images.add(value['image'])
for child in value.values():
walk(child)
elif isinstance(value, list):
for child in value:
walk(child)
for document in yaml.safe_load_all(open(sys.argv[1])):
walk(document)
print('\n'.join(sorted(images)))
PY
cat "$WORK/records/images-required.txt"
ghcr.io/nvidia/nvsentinel/platform-connectors:lab-c15934773572
ghcr.io/nvidia/nvsentinel/syslog-health-monitor:lab-c15934773572
bschmaus@sno2:~/nvkr$ python3 - "$NVKR_REPO" "$IMAGES_ATTEMPT" "$WORK" <<'PY'
from pathlib import Path
import sys
repo, attempt, work = Path(sys.argv[1]), sys.argv[2], Path(sys.argv[3])
names = ('platform-connectors', 'syslog-health-monitor')
references = {name: f'ghcr.io/nvidia/nvsentinel/{name}:lab-c15934773572'
for name in names}
required = set((work / 'records/images-required.txt').read_text().splitlines())
assert required == set(references.values()), 'Image selection changed; review the map'
rows = []
for name in names:
component = name + '-image'
archive = (repo / '.nvkr/runs' / f'{attempt}-{component}' / component
/ 'out' / f'nvsentinel-{component}' / f'{name}.oci.tar')
assert archive.is_file() and archive.stat().st_size > 0, f'Missing archive: {archive}'
rows.append(f'{archive.resolve()}\t{references[name]}\n')
output = work / 'nvsentinel/image-map.tsv'
output.write_text(''.join(rows))
print(output.read_text(), end='')
PY
/home/bschmaus/nvkr/.nvkr/runs/nvsentinel-images-amd64-002-platform-connectors-image/platform-connectors-image/out/nvsentinel-platform-connectors-image/platform-connectors.oci.tar ghcr.io/nvidia/nvsentinel/platform-connectors:lab-c15934773572
/home/bschmaus/nvkr/.nvkr/runs/nvsentinel-images-amd64-002-syslog-health-monitor-image/syslog-health-monitor-image/out/nvsentinel-syslog-health-monitor-image/syslog-health-monitor.oci.tar ghcr.io/nvidia/nvsentinel/syslog-health-monitor:lab-c15934773572
With the map in hand, we should look inside the archives before trusting them. This loop runs skopeo inspect against each OCI archive and prints its metadata. Both archives report amd64 on linux, which is what the nodes need. The platform connector carries labels showing it is a Chainguard static base image built from the NVIDIA NVSentinel source at revision c1593477357299ab93dd5246f5b2cd236f8f07c8, and that revision matches the commit recorded in the values file. The syslog health monitor has no labels at all, but its architecture and layers check out the same way.
$ while IFS=$'\t' read -r archive reference; do
printf 'Checking %s -> %s\n' "$archive" "$reference"
skopeo inspect "oci-archive:$archive"
done <"$WORK/nvsentinel/image-map.tsv"
Checking /home/bschmaus/nvkr/.nvkr/runs/nvsentinel-images-amd64-002-platform-connectors-image/platform-connectors-image/out/nvsentinel-platform-connectors-image/platform-connectors.oci.tar -> ghcr.io/nvidia/nvsentinel/platform-connectors:lab-c15934773572
{
"Digest": "sha256:44587a605c0093969a4f1e8773b36c85a8891591868b8608c2e217a3bfe927a4",
"RepoTags": [],
"Created": "2026-10-13T15:39:47Z",
"DockerVersion": "",
"Labels": {
"dev.chainguard.image.title": "static",
"dev.chainguard.package.main": "",
"org.opencontainers.image.authors": "Chainguard Team https://www.chainguard.dev/",
"org.opencontainers.image.created": "2026-10-13T15:39:47Z",
"org.opencontainers.image.licenses": "Apache-2.0",
"org.opencontainers.image.revision": "c1593477357299ab93dd5246f5b2cd236f8f07c8",
"org.opencontainers.image.source": "https://github.com/NVIDIA/NVSentinel",
"org.opencontainers.image.title": "NVSentinel Platform Connectors",
"org.opencontainers.image.url": "https://images.chainguard.dev/directory/image/static/overview",
"org.opencontainers.image.vendor": "Chainguard",
"org.opencontainers.image.version": "c1593477357299ab93dd5246f5b2cd236f8f07c8"
},
"Architecture": "amd64",
"Os": "linux",
"Layers": [
"sha256:bb8fa58ec978821768266f0c728f5deffbceba0afcd8f819e906150f6ff268f1",
"sha256:e22d79b0cd6ecbd66882a4fb9d6a48b1ff52547b7b97c19a8feef87220d29861"
],
"LayersData": [
{
"MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip",
"Digest": "sha256:bb8fa58ec978821768266f0c728f5deffbceba0afcd8f819e906150f6ff268f1",
"Size": 607648,
"Annotations": null
},
{
"MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip",
"Digest": "sha256:e22d79b0cd6ecbd66882a4fb9d6a48b1ff52547b7b97c19a8feef87220d29861",
"Size": 18978065,
"Annotations": {
"buildkit/rewritten-timestamp": "1791905987"
}
}
],
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin",
"SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt"
]
}
Checking /home/bschmaus/nvkr/.nvkr/runs/nvsentinel-images-amd64-002-syslog-health-monitor-image/syslog-health-monitor-image/out/nvsentinel-syslog-health-monitor-image/syslog-health-monitor.oci.tar -> ghcr.io/nvidia/nvsentinel/syslog-health-monitor:lab-c15934773572
{
"Digest": "sha256:5a22e1eb0e30a4ee06978ff595396398dab79dfcd742b59dd79ff264e97a1e97",
"RepoTags": [],
"Created": "2026-10-13T15:39:47Z",
"DockerVersion": "",
"Labels": null,
"Architecture": "amd64",
"Os": "linux",
"Layers": [
"sha256:ecc510c1e359bdc007b39802570bb7e4dec7d6ebccf357c3201a74299113e05e",
"sha256:d07e089535f23f6d8effe9356056f2074383926bc15856bc298a74ca182f1e94",
"sha256:b1326670d5094c16a8dd5485be0892592b5a1a6872131c571be5ec3008c918f8"
],
"LayersData": [
{
"MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip",
"Digest": "sha256:ecc510c1e359bdc007b39802570bb7e4dec7d6ebccf357c3201a74299113e05e",
"Size": 29843350,
"Annotations": null
},
{
"MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip",
"Digest": "sha256:d07e089535f23f6d8effe9356056f2074383926bc15856bc298a74ca182f1e94",
"Size": 1218439,
"Annotations": null
},
{
"MIMEType": "application/vnd.oci.image.layer.v1.tar+gzip",
"Digest": "sha256:b1326670d5094c16a8dd5485be0892592b5a1a6872131c571be5ec3008c918f8",
"Size": 13142725,
"Annotations": null
}
],
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
]
}
Next the OCI archives get converted into a format the node can import with its container tooling. For each line in the map, the loop confirms both fields are present and the archive exists, refuses any reference that uses a digest, and numbers the destination file. skopeo copy then writes a Docker archive with the amd64 override and tags it with the full image reference. The output shows the blobs and manifest being written for each of the two images.
$ index=0
$ while IFS=$'\t' read -r archive reference; do
test -n "$archive" && test -n "$reference"
test -f "$archive"
case "$reference" in *@*) echo 'Resolve digest references explicitly'; exit 1 ;; esac
index=$((index + 1))
destination=$(printf '%s/images/%03d.tar' "$WORK/nvsentinel" "$index")
test ! -e "$destination"
skopeo copy --override-arch amd64 \
"oci-archive:$archive" "docker-archive:$destination:$reference"
done <"$WORK/nvsentinel/image-map.tsv"
test "$index" -gt 0
Getting image source signatures
Copying blob bb8fa58ec978 done |
Copying blob e22d79b0cd6e done |
Copying config 27f31ee27f done |
Writing manifest to image destination
Getting image source signatures
Copying blob ecc510c1e359 done |
Copying blob d07e089535f2 done |
Copying blob b1326670d509 done |
Copying config 1a0b53262a done |
Writing manifest to image destination
Packaging the Add-On Bundle
Now that we have identified which images we need, we can move onto packaging. Our first check here just prints the image map again with cat. It is a quick confirmation that the map still holds the two entries we expect.
$ cat $WORK/nvsentinel/image-map.tsv
/home/bschmaus/nvkr/.nvkr/runs/nvsentinel-images-amd64-002-platform-connectors-image/platform-connectors-image/out/nvsentinel-platform-connectors-image/platform-connectors.oci.tar ghcr.io/nvidia/nvsentinel/platform-connectors:lab-c15934773572
/home/bschmaus/nvkr/.nvkr/runs/nvsentinel-images-amd64-002-syslog-health-monitor-image/syslog-health-monitor-image/out/nvsentinel-syslog-health-monitor-image/syslog-health-monitor.oci.tar ghcr.io/nvidia/nvsentinel/syslog-health-monitor:lab-c15934773572
The required image list and the NVKR commit are copied next to the chart, and a file named nvsentinel-source-commit.txt gets the NVSentinel commit written into it. These small files travel with the bundle, so anyone who finds it on a node later can see what it was built from.
$ cp "$WORK/records/images-required.txt" "$WORK/nvsentinel/"
$ cp "$WORK/records/nvkr-commit.txt" "$WORK/nvsentinel/"
$ printf '%s\n' c1593477357299ab93dd5246f5b2cd236f8f07c8 >"$WORK/nvsentinel/nvsentinel-source-commit.txt"
Finally the whole nvsentinel directory is sealed. A subshell with set -euo pipefail hashes every file except the checksum file itself into a SHA256SUMS manifest, sorting the names so the manifest is stable, and then immediately verifies that manifest with sha256sum -c. Every file, from the two image tarballs to each template in the chart, reports OK. The du at the end shows the bundle comes to 188 MB. I will come back to this manifest later, because it gets checked again on the node itself.
$ (
set -euo pipefail
cd "$WORK/nvsentinel"
find . -type f ! -name SHA256SUMS -print0 | sort -z | \
xargs -0 sha256sum >SHA256SUMS
sha256sum -c SHA256SUMS
)
du -sh "$WORK/nvsentinel"
./image-map.tsv: OK
./images-required.txt: OK
./images/001.tar: OK
./images/002.tar: OK
./nvkr-commit.txt: OK
./nvsentinel-source-commit.txt: OK
./nvsentinel/.helmignore: OK
./nvsentinel/Chart.lock: OK
./nvsentinel/Chart.yaml: OK
./nvsentinel/charts/csp-health-monitor/.helmignore: OK
./nvsentinel/charts/csp-health-monitor/Chart.yaml: OK
./nvsentinel/charts/csp-health-monitor/templates/_helpers.tpl: OK
./nvsentinel/charts/csp-health-monitor/templates/clusterrole.yaml: OK
./nvsentinel/charts/csp-health-monitor/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/csp-health-monitor/templates/configmap.yaml: OK
./nvsentinel/charts/csp-health-monitor/templates/deployment.yaml: OK
./nvsentinel/charts/csp-health-monitor/templates/serviceaccount.yaml: OK
./nvsentinel/charts/csp-health-monitor/values.yaml: OK
./nvsentinel/charts/event-exporter/Chart.yaml: OK
./nvsentinel/charts/event-exporter/templates/_helpers.tpl: OK
./nvsentinel/charts/event-exporter/templates/configmap.yaml: OK
./nvsentinel/charts/event-exporter/templates/deployment.yaml: OK
./nvsentinel/charts/event-exporter/values.yaml: OK
./nvsentinel/charts/fault-quarantine/.helmignore: OK
./nvsentinel/charts/fault-quarantine/Chart.yaml: OK
./nvsentinel/charts/fault-quarantine/templates/_helpers.tpl: OK
./nvsentinel/charts/fault-quarantine/templates/clusterrole.yaml: OK
./nvsentinel/charts/fault-quarantine/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/fault-quarantine/templates/configmap.yaml: OK
./nvsentinel/charts/fault-quarantine/templates/deployment.yaml: OK
./nvsentinel/charts/fault-quarantine/templates/serviceaccount.yaml: OK
./nvsentinel/charts/fault-quarantine/values.yaml: OK
./nvsentinel/charts/fault-remediation/.helmignore: OK
./nvsentinel/charts/fault-remediation/Chart.yaml: OK
./nvsentinel/charts/fault-remediation/files/log-collector-job.yaml: OK
./nvsentinel/charts/fault-remediation/templates/_helpers.tpl: OK
./nvsentinel/charts/fault-remediation/templates/clusterrole.yaml: OK
./nvsentinel/charts/fault-remediation/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/fault-remediation/templates/configmap.yaml: OK
./nvsentinel/charts/fault-remediation/templates/deployment.yaml: OK
./nvsentinel/charts/fault-remediation/templates/log-collector-rbac.yaml: OK
./nvsentinel/charts/fault-remediation/templates/serviceaccount.yaml: OK
./nvsentinel/charts/fault-remediation/values.yaml: OK
./nvsentinel/charts/gpu-health-monitor/.helmignore: OK
./nvsentinel/charts/gpu-health-monitor/Chart.yaml: OK
./nvsentinel/charts/gpu-health-monitor/files/dcgmerrorsmapping.csv: OK
./nvsentinel/charts/gpu-health-monitor/templates/NOTES.txt: OK
./nvsentinel/charts/gpu-health-monitor/templates/_helpers.tpl: OK
./nvsentinel/charts/gpu-health-monitor/templates/configmap.yaml: OK
./nvsentinel/charts/gpu-health-monitor/templates/daemonset-dcgm-3.x.yaml: OK
./nvsentinel/charts/gpu-health-monitor/templates/daemonset-dcgm-4.x.yaml: OK
./nvsentinel/charts/gpu-health-monitor/tests/dcgm_runtime_debounce_test.yaml: OK
./nvsentinel/charts/gpu-health-monitor/tests/dcgm_startup_gate_test.yaml: OK
./nvsentinel/charts/gpu-health-monitor/tests/gpu_deploy_client_affinity_test.yaml: OK
./nvsentinel/charts/gpu-health-monitor/tests/pc_auth_strictness_test.yaml: OK
./nvsentinel/charts/gpu-health-monitor/tests/priorityclassname_test.yaml: OK
./nvsentinel/charts/gpu-health-monitor/tests/publish_wiring_test.yaml: OK
./nvsentinel/charts/gpu-health-monitor/values.yaml: OK
./nvsentinel/charts/health-events-analyzer/.helmignore: OK
./nvsentinel/charts/health-events-analyzer/Chart.yaml: OK
./nvsentinel/charts/health-events-analyzer/templates/_helpers.tpl: OK
./nvsentinel/charts/health-events-analyzer/templates/configmap.yaml: OK
./nvsentinel/charts/health-events-analyzer/templates/deployment.yaml: OK
./nvsentinel/charts/health-events-analyzer/templates/node-recovery-rbac.yaml: OK
./nvsentinel/charts/health-events-analyzer/templates/role.yaml: OK
./nvsentinel/charts/health-events-analyzer/templates/rolebinding.yaml: OK
./nvsentinel/charts/health-events-analyzer/templates/serviceaccount.yaml: OK
./nvsentinel/charts/health-events-analyzer/tests/node_recovery_rbac_test.yaml: OK
./nvsentinel/charts/health-events-analyzer/values.yaml: OK
./nvsentinel/charts/incluster-file-server/Chart.yaml: OK
./nvsentinel/charts/incluster-file-server/templates/_helpers.tpl: OK
./nvsentinel/charts/incluster-file-server/templates/configmap-nginx.yaml: OK
./nvsentinel/charts/incluster-file-server/templates/deployment.yaml: OK
./nvsentinel/charts/incluster-file-server/templates/service.yaml: OK
./nvsentinel/charts/incluster-file-server/values.yaml: OK
./nvsentinel/charts/janitor-provider/Chart.yaml: OK
./nvsentinel/charts/janitor-provider/templates/_helpers.tpl: OK
./nvsentinel/charts/janitor-provider/templates/certificate.yaml: OK
./nvsentinel/charts/janitor-provider/templates/clusterrole.yaml: OK
./nvsentinel/charts/janitor-provider/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/janitor-provider/templates/deployment.yaml: OK
./nvsentinel/charts/janitor-provider/templates/networkpolicy.yaml: OK
./nvsentinel/charts/janitor-provider/templates/role.yaml: OK
./nvsentinel/charts/janitor-provider/templates/rolebinding.yaml: OK
./nvsentinel/charts/janitor-provider/templates/service.yaml: OK
./nvsentinel/charts/janitor-provider/templates/serviceaccount.yaml: OK
./nvsentinel/charts/janitor-provider/values.yaml: OK
./nvsentinel/charts/janitor/.helmignore: OK
./nvsentinel/charts/janitor/Chart.yaml: OK
./nvsentinel/charts/janitor/crds/external_remediation.crd.yaml: OK
./nvsentinel/charts/janitor/crds/janitor.dgxc.nvidia.com_gpuresets.yaml: OK
./nvsentinel/charts/janitor/crds/janitor.dgxc.nvidia.com_rebootnodes.yaml: OK
./nvsentinel/charts/janitor/crds/janitor.dgxc.nvidia.com_terminatenodes.yaml: OK
./nvsentinel/charts/janitor/templates/_helpers.tpl: OK
./nvsentinel/charts/janitor/templates/certificate-metrics.yaml: OK
./nvsentinel/charts/janitor/templates/clusterrole.yaml: OK
./nvsentinel/charts/janitor/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/janitor/templates/configmap.yaml: OK
./nvsentinel/charts/janitor/templates/deployment.yaml: OK
./nvsentinel/charts/janitor/templates/issuer.yaml: OK
./nvsentinel/charts/janitor/templates/networkpolicy.yaml: OK
./nvsentinel/charts/janitor/templates/service.yaml: OK
./nvsentinel/charts/janitor/templates/serviceaccount.yaml: OK
./nvsentinel/charts/janitor/templates/webhook.yaml: OK
./nvsentinel/charts/janitor/values.schema.json: OK
./nvsentinel/charts/janitor/values.yaml: OK
./nvsentinel/charts/k8s-datastore/Chart.yaml: OK
./nvsentinel/charts/k8s-datastore/crds/health_event.crd.yaml: OK
./nvsentinel/charts/k8s-datastore/values.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/Chart.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/templates/_helpers.tpl: OK
./nvsentinel/charts/kubernetes-object-monitor/templates/clusterrole.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/templates/configmap.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/templates/deployment.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/templates/serviceaccount.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/tests/pc_auth_strictness_test.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/tests/publish_wiring_test.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/values.yaml: OK
./nvsentinel/charts/labeler/.helmignore: OK
./nvsentinel/charts/labeler/Chart.yaml: OK
./nvsentinel/charts/labeler/templates/_helpers.tpl: OK
./nvsentinel/charts/labeler/templates/clusterrole.yaml: OK
./nvsentinel/charts/labeler/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/labeler/templates/configmap.yaml: OK
./nvsentinel/charts/labeler/templates/deployment.yaml: OK
./nvsentinel/charts/labeler/templates/serviceaccount.yaml: OK
./nvsentinel/charts/labeler/tests/priorityclassname_test.yaml: OK
./nvsentinel/charts/labeler/values.yaml: OK
./nvsentinel/charts/lifecycle-manager/.helmignore: OK
./nvsentinel/charts/lifecycle-manager/Chart.yaml: OK
./nvsentinel/charts/lifecycle-manager/crds/maintenance_request.crd.yaml: OK
./nvsentinel/charts/lifecycle-manager/crds/nvsentinel.nvidia.com_validationrequests.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/_helpers.tpl: OK
./nvsentinel/charts/lifecycle-manager/templates/certificate-metrics.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/clusterrole.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/configmap-templates.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/configmap.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/deployment.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/issuer.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/networkpolicy.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/service.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/serviceaccount.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/webhook.yaml: OK
./nvsentinel/charts/lifecycle-manager/tests/deployment_test.yaml: OK
./nvsentinel/charts/lifecycle-manager/tests/pc_auth_strictness_test.yaml: OK
./nvsentinel/charts/lifecycle-manager/tests/publish_wiring_test.yaml: OK
./nvsentinel/charts/lifecycle-manager/tests/rbac_test.yaml: OK
./nvsentinel/charts/lifecycle-manager/tests/webhook_test.yaml: OK
./nvsentinel/charts/lifecycle-manager/values.yaml: OK
./nvsentinel/charts/metadata-collector/Chart.yaml: OK
./nvsentinel/charts/metadata-collector/templates/_helpers.tpl: OK
./nvsentinel/charts/metadata-collector/templates/clusterrole.yaml: OK
./nvsentinel/charts/metadata-collector/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/metadata-collector/templates/daemonset.yaml: OK
./nvsentinel/charts/metadata-collector/templates/resourceclaimtemplate.yaml: OK
./nvsentinel/charts/metadata-collector/templates/serviceaccount.yaml: OK
./nvsentinel/charts/metadata-collector/tests/daemonset_test.yaml: OK
./nvsentinel/charts/metadata-collector/tests/gpu_deploy_client_affinity_test.yaml: OK
./nvsentinel/charts/metadata-collector/tests/resourceclaimtemplate_test.yaml: OK
./nvsentinel/charts/metadata-collector/values.yaml: OK
./nvsentinel/charts/mongodb-store/.helmignore: OK
./nvsentinel/charts/mongodb-store/Chart.lock: OK
./nvsentinel/charts/mongodb-store/Chart.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/.helmignore: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/Chart.lock: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/Chart.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/README.md: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/.helmignore: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/Chart.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/README.md: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_affinities.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_capabilities.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_compatibility.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_errors.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_images.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_ingress.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_labels.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_names.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_resources.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_secrets.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_storage.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_tplvalues.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_utils.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_warnings.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/validations/_cassandra.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/validations/_mariadb.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/validations/_mongodb.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/validations/_mysql.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/validations/_postgresql.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/validations/_redis.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/validations/_validations.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/values.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/NOTES.txt: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/_helpers.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/arbiter/configmap.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/arbiter/headless-svc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/arbiter/pdb.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/arbiter/statefulset.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/backup/cronjob.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/backup/pvc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/common-scripts-cm.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/configmap.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/extra-list.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/hidden/configmap.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/hidden/external-access-svc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/hidden/headless-svc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/hidden/pdb.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/hidden/statefulset.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/initialization-configmap.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/metrics-svc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/networkpolicy.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/prometheusrule.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/psp.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/replicaset/external-access-svc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/replicaset/headless-svc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/replicaset/pdb.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/replicaset/scripts-configmap.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/replicaset/statefulset.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/replicaset/svc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/role.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/rolebinding.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/secrets-ca.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/secrets.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/serviceaccount.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/servicemonitor.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/standalone/dep-sts.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/standalone/pdb.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/standalone/pvc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/standalone/svc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/values.schema.json: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/values.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/.helmignore: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/Chart.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/README.md: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/templates/NOTES.txt: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/templates/_helpers.tpl: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/templates/cluster-secret.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/templates/cluster.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/templates/clustersync-secret.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/templates/clustersync.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/values.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/.helmignore: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/Chart.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/LICENSE.txt: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/README.md: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/crds/crd.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/templates/NOTES.txt: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/templates/_helpers.tpl: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/templates/deployment.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/templates/namespace.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/templates/role-binding.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/templates/role.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/values.yaml: OK
./nvsentinel/charts/mongodb-store/templates/_helpers.tpl: OK
./nvsentinel/charts/mongodb-store/templates/_init-eval.tpl: OK
./nvsentinel/charts/mongodb-store/templates/certmanager.yaml: OK
./nvsentinel/charts/mongodb-store/templates/configmap.yaml: OK
./nvsentinel/charts/mongodb-store/templates/jobs.yaml: OK
./nvsentinel/charts/mongodb-store/templates/metrics-service.yaml: OK
./nvsentinel/charts/mongodb-store/templates/networkpolicy.yaml: OK
./nvsentinel/charts/mongodb-store/templates/pv.yaml: OK
./nvsentinel/charts/mongodb-store/templates/role.yaml: OK
./nvsentinel/charts/mongodb-store/templates/rolebinding.yaml: OK
./nvsentinel/charts/mongodb-store/tests/cluster_domain_test.yaml: OK
./nvsentinel/charts/mongodb-store/tests/networkpolicy_test.yaml: OK
./nvsentinel/charts/mongodb-store/tests/psmdb_versions_test.yaml: OK
./nvsentinel/charts/mongodb-store/values.yaml: OK
./nvsentinel/charts/nic-health-monitor/.helmignore: OK
./nvsentinel/charts/nic-health-monitor/Chart.yaml: OK
./nvsentinel/charts/nic-health-monitor/templates/_helpers.tpl: OK
./nvsentinel/charts/nic-health-monitor/templates/configmap.yaml: OK
./nvsentinel/charts/nic-health-monitor/templates/daemonset.yaml: OK
./nvsentinel/charts/nic-health-monitor/tests/daemonset_test.yaml: OK
./nvsentinel/charts/nic-health-monitor/tests/pc_auth_strictness_test.yaml: OK
./nvsentinel/charts/nic-health-monitor/tests/publish_wiring_test.yaml: OK
./nvsentinel/charts/nic-health-monitor/values.yaml: OK
./nvsentinel/charts/node-drainer/.helmignore: OK
./nvsentinel/charts/node-drainer/Chart.yaml: OK
./nvsentinel/charts/node-drainer/templates/_helpers.tpl: OK
./nvsentinel/charts/node-drainer/templates/clusterrole.yaml: OK
./nvsentinel/charts/node-drainer/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/node-drainer/templates/configmap.yaml: OK
./nvsentinel/charts/node-drainer/templates/deployment.yaml: OK
./nvsentinel/charts/node-drainer/templates/serviceaccount.yaml: OK
./nvsentinel/charts/node-drainer/tests/pod_drain_policies_test.yaml: OK
./nvsentinel/charts/node-drainer/values.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/Chart.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/templates/_helpers.tpl: OK
./nvsentinel/charts/nvcre-certification-monitor/templates/clusterrole.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/templates/configmap.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/templates/deployment.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/templates/serviceaccount.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/tests/pc_auth_strictness_test.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/tests/publish_wiring_test.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/values.yaml: OK
./nvsentinel/charts/postgresql/.helmignore: OK
./nvsentinel/charts/postgresql/Chart.lock: OK
./nvsentinel/charts/postgresql/Chart.yaml: OK
./nvsentinel/charts/postgresql/README.md: OK
./nvsentinel/charts/postgresql/charts/common/.helmignore: OK
./nvsentinel/charts/postgresql/charts/common/Chart.yaml: OK
./nvsentinel/charts/postgresql/charts/common/README.md: OK
./nvsentinel/charts/postgresql/charts/common/templates/_affinities.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_capabilities.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_compatibility.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_errors.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_images.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_ingress.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_labels.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_names.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_resources.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_secrets.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_storage.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_tplvalues.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_utils.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_warnings.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/validations/_cassandra.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/validations/_mariadb.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/validations/_mongodb.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/validations/_mysql.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/validations/_postgresql.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/validations/_redis.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/validations/_validations.tpl: OK
./nvsentinel/charts/postgresql/charts/common/values.yaml: OK
./nvsentinel/charts/postgresql/templates/NOTES.txt: OK
./nvsentinel/charts/postgresql/templates/_helpers.tpl: OK
./nvsentinel/charts/postgresql/templates/backup/cronjob.yaml: OK
./nvsentinel/charts/postgresql/templates/backup/networkpolicy.yaml: OK
./nvsentinel/charts/postgresql/templates/backup/pvc.yaml: OK
./nvsentinel/charts/postgresql/templates/extra-list.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/configmap.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/extended-configmap.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/initialization-configmap.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/metrics-configmap.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/metrics-svc.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/networkpolicy.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/pdb.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/preinitialization-configmap.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/servicemonitor.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/statefulset.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/svc-headless.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/svc.yaml: OK
./nvsentinel/charts/postgresql/templates/prometheusrule.yaml: OK
./nvsentinel/charts/postgresql/templates/psp.yaml: OK
./nvsentinel/charts/postgresql/templates/read/extended-configmap.yaml: OK
./nvsentinel/charts/postgresql/templates/read/metrics-configmap.yaml: OK
./nvsentinel/charts/postgresql/templates/read/metrics-svc.yaml: OK
./nvsentinel/charts/postgresql/templates/read/networkpolicy.yaml: OK
./nvsentinel/charts/postgresql/templates/read/pdb.yaml: OK
./nvsentinel/charts/postgresql/templates/read/servicemonitor.yaml: OK
./nvsentinel/charts/postgresql/templates/read/statefulset.yaml: OK
./nvsentinel/charts/postgresql/templates/read/svc-headless.yaml: OK
./nvsentinel/charts/postgresql/templates/read/svc.yaml: OK
./nvsentinel/charts/postgresql/templates/role.yaml: OK
./nvsentinel/charts/postgresql/templates/rolebinding.yaml: OK
./nvsentinel/charts/postgresql/templates/secrets.yaml: OK
./nvsentinel/charts/postgresql/templates/serviceaccount.yaml: OK
./nvsentinel/charts/postgresql/templates/tls-secrets.yaml: OK
./nvsentinel/charts/postgresql/values.schema.json: OK
./nvsentinel/charts/postgresql/values.yaml: OK
./nvsentinel/charts/preflight/.helmignore: OK
./nvsentinel/charts/preflight/Chart.yaml: OK
./nvsentinel/charts/preflight/crds/preflight.nvsentinel.nvidia.com_preflightconfigs.yaml: OK
./nvsentinel/charts/preflight/files/ndv4-topo.xml: OK
./nvsentinel/charts/preflight/files/ndv5-topo.xml: OK
./nvsentinel/charts/preflight/templates/_helpers.tpl: OK
./nvsentinel/charts/preflight/templates/ca-certificate.yaml: OK
./nvsentinel/charts/preflight/templates/ca-issuer.yaml: OK
./nvsentinel/charts/preflight/templates/certificate.yaml: OK
./nvsentinel/charts/preflight/templates/clusterrole.yaml: OK
./nvsentinel/charts/preflight/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/preflight/templates/configmap.yaml: OK
./nvsentinel/charts/preflight/templates/deployment.yaml: OK
./nvsentinel/charts/preflight/templates/image-cache-daemonset.yaml: OK
./nvsentinel/charts/preflight/templates/mutatingwebhook.yaml: OK
./nvsentinel/charts/preflight/templates/networkpolicy.yaml: OK
./nvsentinel/charts/preflight/templates/selfsigned-issuer.yaml: OK
./nvsentinel/charts/preflight/templates/service.yaml: OK
./nvsentinel/charts/preflight/templates/serviceaccount.yaml: OK
./nvsentinel/charts/preflight/tests/dra_rbac_test.yaml: OK
./nvsentinel/charts/preflight/tests/mutatingwebhook_test.yaml: OK
./nvsentinel/charts/preflight/tests/pc_auth_strictness_test.yaml: OK
./nvsentinel/charts/preflight/tests/publish_wiring_test.yaml: OK
./nvsentinel/charts/preflight/values.schema.json: OK
./nvsentinel/charts/preflight/values.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/Chart.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/templates/_helpers.tpl: OK
./nvsentinel/charts/slurm-drain-monitor/templates/clusterrole.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/templates/configmap.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/templates/deployment.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/templates/serviceaccount.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/tests/pc_auth_strictness_test.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/tests/publish_wiring_test.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/values.yaml: OK
./nvsentinel/charts/syslog-health-monitor/.helmignore: OK
./nvsentinel/charts/syslog-health-monitor/Chart.yaml: OK
./nvsentinel/charts/syslog-health-monitor/templates/_helpers.tpl: OK
./nvsentinel/charts/syslog-health-monitor/templates/configmap.yaml: OK
./nvsentinel/charts/syslog-health-monitor/templates/daemonset-kata.yaml: OK
./nvsentinel/charts/syslog-health-monitor/templates/daemonset-regular.yaml: OK
./nvsentinel/charts/syslog-health-monitor/tests/pc_auth_strictness_test.yaml: OK
./nvsentinel/charts/syslog-health-monitor/tests/publish_wiring_test.yaml: OK
./nvsentinel/charts/syslog-health-monitor/values.yaml: OK
./nvsentinel/files/node-condition-cleanup.sh: OK
./nvsentinel/policies/README.md: OK
./nvsentinel/policies/must-have-sbom.yaml: OK
./nvsentinel/policies/must-have-slsa.yaml: OK
./nvsentinel/templates/NOTES.txt: OK
./nvsentinel/templates/_external-mongo-init-eval.tpl: OK
./nvsentinel/templates/_helpers.tpl: OK
./nvsentinel/templates/_platform-connector-config.tpl: OK
./nvsentinel/templates/certmanager-postgresql.yaml: OK
./nvsentinel/templates/clusterrole.yaml: OK
./nvsentinel/templates/clusterrolebinding.yaml: OK
./nvsentinel/templates/configmap-datastore.yaml: OK
./nvsentinel/templates/configmap.yaml: OK
./nvsentinel/templates/daemonset.yaml: OK
./nvsentinel/templates/job-external-mongodb-setup.yaml: OK
./nvsentinel/templates/networkpolicy.yaml: OK
./nvsentinel/templates/node-condition-cleanup-hook.yaml: OK
./nvsentinel/templates/platform-connector-deployment-certificate.yaml: OK
./nvsentinel/templates/platform-connector-deployment-configmap.yaml: OK
./nvsentinel/templates/platform-connector-deployment-deployment.yaml: OK
./nvsentinel/templates/platform-connector-deployment-networkpolicy.yaml: OK
./nvsentinel/templates/platform-connector-deployment-pdb.yaml: OK
./nvsentinel/templates/platform-connector-deployment-service.yaml: OK
./nvsentinel/templates/platform-connector-deployment-serviceaccount.yaml: OK
./nvsentinel/templates/platform-connector-deployment-validation.yaml: OK
./nvsentinel/templates/podMonitor.yaml: OK
./nvsentinel/templates/serviceaccount.yaml: OK
./nvsentinel/tests/cluster_domain_test.yaml: OK
./nvsentinel/tests/csp_health_monitor_publish_wiring_test.yaml: OK
./nvsentinel/tests/datastore_credentials_test.yaml: OK
./nvsentinel/tests/deployed_crversion_test.yaml: OK
./nvsentinel/tests/external_mongodb_setup_job_test.yaml: OK
./nvsentinel/tests/health_events_analyzer_publish_wiring_test.yaml: OK
./nvsentinel/tests/pc_auth_cross_node_test.yaml: OK
./nvsentinel/tests/platform_connector_deployment_networkpolicy_test.yaml: OK
./nvsentinel/tests/platform_connector_deployment_server_test.yaml: OK
./nvsentinel/tests/platform_connector_deployment_test.yaml: OK
./nvsentinel/tests/platform_connector_retry_test.yaml: OK
./nvsentinel/tests/strict_render_test.yaml: OK
./nvsentinel/tests/values/publishers-on-deployment.yaml: OK
./nvsentinel/values-aicr-aws-uat.yaml: OK
./nvsentinel/values-aicr-gcp-uat.yaml: OK
./nvsentinel/values-aicr-uat.yaml: OK
./nvsentinel/values-atlas-gcp.yaml: OK
./nvsentinel/values-aws-docdb.yaml: OK
./nvsentinel/values-cosmosdb-test.yaml: OK
./nvsentinel/values-full.yaml: OK
./nvsentinel/values-npd-gpu-services.yaml: OK
./nvsentinel/values-npd-remediation.yaml: OK
./nvsentinel/values-nvcre.yaml: OK
./nvsentinel/values-postgresql.yaml: OK
./nvsentinel/values-preflight-kube.yaml: OK
./nvsentinel/values-remediation.yaml: OK
./nvsentinel/values-tilt-arm64.yaml: OK
./nvsentinel/values-tilt-deployment.yaml: OK
./nvsentinel/values-tilt-mongodb-percona.yaml: OK
./nvsentinel/values-tilt-mongodb-tls-disabled-percona.yaml: OK
./nvsentinel/values-tilt-mongodb-tls-disabled.yaml: OK
./nvsentinel/values-tilt-mongodb.yaml: OK
./nvsentinel/values-tilt-nvcre.yaml: OK
./nvsentinel/values-tilt-postgresql.yaml: OK
./nvsentinel/values-tilt-socket.yaml: OK
./nvsentinel/values-tilt.yaml: OK
./nvsentinel/values.yaml: OK
./values-lab.yaml: OK
188M /home/bschmaus/nvke-nvsentinel-iso-002/nvsentinel
Preparing the Node Disk Image
At this point the add-on content is ready and our attention moves to the node image. Before resizing the base image we should know how it is laid out. virt-filesystems shows a roughly 12 GB disk with an EFI partition, a small boot partition and a third partition holding an LVM volume group named ubuntu-vg. The root filesystem lives on a 9.4 GB logical volume inside that group, which is far too small for container images and a Kubernetes control plane.
$ virt-filesystems --format=qcow2 -a "$BASE_QCOW" --all --long -h
Name Type VFS Label MBR Size Parent
/dev/sda1 filesystem vfat - - 571M -
/dev/sda2 filesystem ext4 - - 1.7G -
/dev/ubuntu-vg/ubuntu-lv filesystem ext4 cloudimg-rootfs - 9.4G -
/dev/ubuntu-vg/ubuntu-lv lv - - - 9.7G /dev/ubuntu-vg
/dev/ubuntu-vg vg - - - 9.7G /dev/sda3
/dev/sda3 pv - - - 9.7G -
/dev/sda1 partition - - - 572M /dev/sda
/dev/sda2 partition - - - 1.8G /dev/sda
/dev/sda3 partition - - - 9.7G /dev/sda
/dev/sda device - - - 12G -
Thus we have to grow the root filesystem. The first command creates an empty 128 GB qcow2 file, and virt-resize copies the base image into it. The summary at the top spells out the plan: the first two partitions are left alone, the third is expanded from 9.7 GB to 125.7 GB, the LVM physical volume is resized to match and the root logical volume is expanded to its maximum with resize2fs. It finishes with no errors, and the tool adds the sensible reminder to boot the resized disk before throwing the old one away.
$ qemu-img create -f qcow2 "$WORK/payload/node.qcow2" 128G
Formatting '/home/bschmaus/nvke-nvsentinel-iso-002/payload/node.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=137438953472 lazy_refcounts=off refcount_bits=16
$ virt-resize --format qcow2 --output-format qcow2 --expand /dev/sda3 --LV-expand /dev/ubuntu-vg/ubuntu-lv "$BASE_QCOW" "$WORK/payload/node.qcow2"
[ 0.0] Examining /home/bschmaus/nvkr/.nvkr/runs/ubuntu24-lab-a9a0537b-001/outputs/nvke-node-image-ubuntu24-amd64.qcow2
**********
Summary of changes:
virt-resize: /dev/sda1: This partition will be left alone.
virt-resize: /dev/sda2: This partition will be left alone.
virt-resize: /dev/sda3: This partition will be resized from 9.7G to 125.7G.
The LVM PV on /dev/sda3 will be expanded using the ‘pvresize’ method.
virt-resize: /dev/ubuntu-vg/ubuntu-lv: This logical volume will be expanded
to maximum size. The filesystem ext4 on /dev/ubuntu-vg/ubuntu-lv will be
expanded using the ‘resize2fs’ method.
**********
[ 5.0] Setting up initial partition table on /home/bschmaus/nvke-nvsentinel-iso-002/payload/node.qcow2
[ 16.2] Copying /dev/sda1
100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
[ 19.5] Copying /dev/sda2
100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
[ 24.2] Copying /dev/sda3
100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ --:--
[ 51.4] Expanding /dev/sda3 using the ‘pvresize’ method
[ 51.4] Expanding /dev/ubuntu-vg/ubuntu-lv using the ‘resize2fs’ method
virt-resize: Resize operation completed with no errors. Before deleting the old disk, carefully check that the resized disk boots and works correctly.
Staging the Bootstrap Host Tooling
The node that runs the cluster bootstrap needs nvkectl and a few helpers, and the plan is to have them already sitting on the disk. This block makes sure a bootstrap host bundle exists, building one only if the file is missing, and then reads bundle.json out of the tarball. The source commit it prints matches the NVKE commit exported at the start, and it lists two images, the NVKE controller and the cluster API provider for pre-provisioned machines. No build output appears, which tells me the bundle from an earlier run (nvke-nvsentinel-iso-001) was reused as is.
$ export BOOTSTRAP_BUNDLE="${BOOTSTRAP_BUNDLE:-$WORK/bootstrap-host-bundle.tar}"
(
set -euo pipefail
if ! test -s "$BOOTSTRAP_BUNDLE"; then
export OCI_BIN=nerdctl
export BUILDKIT_HOST=unix:///run/buildkit/buildkitd.sock
python3 "$AUTO_KIT/prepare-bootstrap-bundle.py" --source "$NVKE_REPO" --output "$BOOTSTRAP_BUNDLE" --controller-image nke-controller:lab-as65000 --runtime-sudo
fi
tar -xOf "$BOOTSTRAP_BUNDLE" bundle.json | python3 -c 'import json,sys; m=json.load(sys.stdin); print("Source:", m["source_commit"]); print("Images:", m["images"])'
)
Source: a9a0537b4418d08ca86d758ef8e3ba275501150e
Images: {'controller': 'nke-controller:lab-as65000', 'provider': 'cluster-api-provider-provisioned:latest'}
Next the bootstrap host files are staged without touching this machine. bootstrap_host.py --stage-only writes its files into a temporary root, and the result is archived with numeric ownership set to root so the files land on the node with the right owner. The message at the end is worth reading carefully: the service is staged and disabled, and it should be started manually only on node one. That keeps the other nodes from also trying to become a bootstrap host.
$ (
set -euo pipefail
test -s "${BOOTSTRAP_BUNDLE:?BOOTSTRAP_BUNDLE is not set}"
test ! -e "$WORK/bootstrap-host-stage.tar"
bootstrap_stage=$(mktemp -d "$WORK/bootstrap-stage.XXXXXX")
sudo python3 "$AUTO_KIT/bootstrap_host.py" --stage-only --root "$bootstrap_stage" --bundle "$BOOTSTRAP_BUNDLE"
sudo tar --numeric-owner --owner=0 --group=0 -C "$bootstrap_stage" -cf "$WORK/bootstrap-host-stage.tar" opt usr etc
sudo chown "$(id -u):$(id -g)" "$WORK/bootstrap-host-stage.tar"
printf 'Retained staging tree: %s\n' "$bootstrap_stage"
)
[sudo] password for bschmaus:
nvke-bootstrap-host-setup.service staged and disabled; start it manually ONLY on node 1.
Retained staging tree: /home/bschmaus/nvke-nvsentinel-iso-002/bootstrap-stage.VfGfHA
Injecting the Payload into the Disk Image
Now everything gets baked into the disk. The NVSentinel directory is archived, and virt-customize opens the resized qcow2, creates /opt/nvke-addons and unpacks both the NVSentinel bundle and the staged bootstrap host files into the image. The log lines show each tar file unpacking, an SELinux relabel and a clean finish, all without booting the image.
$ tar -C "$WORK" -cf "$WORK/nvsentinel-bundle.tar" nvsentinel
virt-customize --format qcow2 -a "$WORK/payload/node.qcow2" \
--mkdir /opt/nvke-addons \
--tar-in "$WORK/nvsentinel-bundle.tar:/opt/nvke-addons" \
--tar-in "$WORK/bootstrap-host-stage.tar:/"
[ 0.0] Examining the guest ...
[ 15.3] Setting a random seed
[ 15.3] Setting the machine ID in /etc/machine-id
[ 15.3] Making directory: /opt/nvke-addons
[ 15.3] Unpack tar file: /home/bschmaus/nvke-nvsentinel-iso-002/nvsentinel-bundle.tar to /opt/nvke-addons
[ 16.1] Unpack tar file: /home/bschmaus/nvke-nvsentinel-iso-002/bootstrap-host-stage.tar to /
[ 19.0] SELinux relabelling
[ 20.0] Finishing off
We can verify this by looking inside the image from the outside. These commands read the recorded NVSentinel commit, print the systemd unit that was staged and list the bootstrap directory. The unit is a oneshot service that requires containerd, only runs if a marker file is absent and is meant for manual starts. The listing shows the 296 MB bundle.tar, and qemu-img check reports no errors with the image about 3.5 percent allocated. qemu-img info confirms a 128 GiB virtual size that occupies only 4.43 GiB on disk. Two libguestfs complaints about a path not starting with a slash show up in the output, but every file I asked for still came back, so I treated them as noise from the tooling and not a problem with the image.
$ virt-cat --format qcow2 -a "$WORK/payload/node.qcow2" /opt/nvke-addons/nvsentinel/nvsentinel-source-commit.txt
$ virt-cat --format qcow2 -a "$WORK/payload/node.qcow2" /etc/systemd/system/nvke-bootstrap-host-setup.service
$ virt-ls --format=qcow2 -a "$WORK/payload/node.qcow2" -l /opt/nvke-bootstrap
$ qemu-img check -f qcow2 "$WORK/payload/node.qcow2"
$ qemu-img info --backing-chain "$WORK/payload/node.qcow2"
libguestfs: error: download: download_stub: path must start with a / character
c1593477357299ab93dd5246f5b2cd236f8f07c8
libguestfs: error: download: download_stub: path must start with a / character
[Unit]
Description=Prepare this node as an NVKE bootstrap host (manual start only)
Requires=containerd.service
After=containerd.service network-online.target cloud-final.service nvke-node-firstboot.service
Wants=network-online.target
ConditionPathExists=!/var/lib/nvke-iso/bootstrap-host-prepared
[Service]
Type=oneshot
ExecStart=/usr/bin/python3 /usr/local/lib/nvke/bootstrap_host.py
TimeoutStartSec=1h
RemainAfterExit=yes
StandardOutput=journal+console
StandardError=journal+console
[Install]
WantedBy=cloud-init.target
total 289232
drwxr-xr-x 2 0 0 4096 Nov 5 18:34 .
drwxr-xr-x 4 0 0 4096 Nov 5 18:34 ..
-rw------- 1 0 0 296161280 Nov 5 18:34 bundle.tar
No errors were found on the image.
72427/2097152 = 3.45% allocated, 4.82% fragmented, 0.00% compressed clusters
Image end offset: 4752539648
image: /home/bschmaus/nvke-nvsentinel-iso-002/payload/node.qcow2
file format: qcow2
virtual size: 128 GiB (137438953472 bytes)
disk size: 4.43 GiB
cluster_size: 65536
Format specific information:
compat: 1.1
compression type: zlib
lazy refcounts: false
refcount bits: 16
corrupt: false
extended l2: false
Child node '/file':
filename: /home/bschmaus/nvke-nvsentinel-iso-002/payload/node.qcow2
protocol type: file
file length: 4.43 GiB (4752539648 bytes)
disk size: 4.43 GiB
The disk gets one last stamp. The NVKE source commit is written next to it, and a checksum is generated for node.qcow2 and then verified. The final node.qcow2: OK is the line that matters, because the image that goes into the ISO is now one we can prove has not changed.
$ (
set -euo pipefail
test -n "${IMAGE_NVKE_COMMIT:?Restore the actual source commit selected by the completed disk Run}"
printf '%s\n' "$IMAGE_NVKE_COMMIT" >"$WORK/payload/nvke-source-commit.txt"
cd "$WORK/payload"
sha256sum node.qcow2 >node.qcow2.sha256
sha256sum -c node.qcow2.sha256
)
node.qcow2: OK
Building the Automatic Install ISO
The disk image is ready to be wrapped in an installer. However this ISO wrapped installer is not part of the NVKR or NVKE repos but rather one I came up with because instead of iPXE booting the qcow2 image I wanted a method of installation via an ISO I can mount to my virtual machines.
The example node file from the auto-install kit defines the shape of that installer. It sets a lab-plaintext mode and package preparation on first boot, and then it lists three nodes. Each node entry carries the MAC address to match, the target disk and its serial number, an erase_confirmation string that must be typed exactly before a disk will be wiped, and the SSH user and authorized key.
$ cat $AUTO_KIT/nodes.example.json
{
"schema_version": 1,
"mode": "lab-plaintext",
"prepare_packages": true,
"nodes": [
{
"name": "asus3-vm1",
"instance_id": "asus3-vm1",
"match_mac": "52:54:00:e7:05:72",
"disk": "/dev/sda",
"disk_serial": "asus3vm1",
"erase_confirmation": "ERASE:asus3vm1",
"ssh_user": "bschmaus",
"ssh_authorized_keys": ["ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN8yjl5h1+CNAAoaKL5NmDFvI2Owbpp4Q11onBmXaFaR bschmaus@FVXW4M2W1Y"]
},
{
"name": "asus3-vm2",
"instance_id": "asus3-vm2",
"match_mac": "52:54:00:95:fd:f3",
"disk": "/dev/sda",
"disk_serial": "asus3vm2",
"erase_confirmation": "ERASE:asus3vm2",
"ssh_user": "bschmaus",
"ssh_authorized_keys": ["ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN8yjl5h1+CNAAoaKL5NmDFvI2Owbpp4Q11onBmXaFaR bschmaus@FVXW4M2W1Y"]
},
{
"name": "asus3-vm3",
"instance_id": "asus3-vm3",
"match_mac": "52:54:00:e8:b9:18",
"disk": "/dev/sda",
"disk_serial": "asus3vm3",
"erase_confirmation": "ERASE:asus3vm3",
"ssh_user": "bschmaus",
"ssh_authorized_keys": ["ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN8yjl5h1+CNAAoaKL5NmDFvI2Owbpp4Q11onBmXaFaR bschmaus@FVXW4M2W1Y"]
}
]
}
Now we build it. The block sets the node config path, the stock Ubuntu 24.04.5 live server ISO and the output name, copies the example config into place if one does not exist, and validates it with imager.py. The validation prints a profile for each of the three VMs and states that no hardware operations were performed. build-iso.py then takes the NVKE source, the Ubuntu ISO, the qcow2 payload and the node config, and xorriso rewrites the ISO with the payload added. The progress lines count up to 99.2 percent, it reports the finished ISO at about 8.4 GB with its El Torito boot record and EFI image intact, and it writes a checksum file. The build also prints a warning that it is experimental and should be validated on a fresh blank VM first, and it notes that no target disks were written. The sha256sum -c at the end reports the ISO as OK.
$ export NODE_CONFIG="$WORK/nodes-auto.json"
export UBUNTU_ISO="$NVKE_REPO/os-image/packer/iso-cache/ubuntu-24.04.5-live-server-amd64.iso"
export OUTPUT_ISO="$WORK/nvke-nvsentinel-automatic-amd64.iso"
if test ! -e "$NODE_CONFIG"; then
cp "$AUTO_KIT/nodes.example.json" "$NODE_CONFIG"
fi
python3 "$AUTO_KIT/imager.py" --validate-config "$NODE_CONFIG"
sudo python3 "$AUTO_KIT/build-iso.py" --source "$NVKE_REPO" --ubuntu-iso "$UBUNTU_ISO" --qcow "$WORK/payload/node.qcow2" --config "$NODE_CONFIG" --output "$OUTPUT_ISO" --acknowledge-disk-erasure
(
cd "$WORK"
sha256sum -c nvke-nvsentinel-automatic-amd64.iso.sha256
)
Node profiles valid (no hardware operations performed).
Profile: asus3-vm1 MAC=52:54:00:e7:05:72 disk=/dev/sda serial=asus3vm1
Profile: asus3-vm2 MAC=52:54:00:95:fd:f3 disk=/dev/sda serial=asus3vm2
Profile: asus3-vm3 MAC=52:54:00:e8:b9:18 disk=/dev/sda serial=asus3vm3
First-boot package preparation: True
xorriso 1.5.6 : RockRidge filesystem manipulator, libburnia project.
Copying of file objects from ISO image to disk filesystem is: Enabled
xorriso : NOTE : Loading ISO image tree from LBA 0
xorriso : UPDATE : 1195 nodes read in 1 seconds
libisofs: NOTE : Found hidden El-Torito image for EFI.
libisofs: NOTE : EFI image start and size: 1989719 * 2048 , 10160 * 512
xorriso : NOTE : Detected El-Torito boot information which currently is set to be discarded
Drive current: -indev '/home/bschmaus/nvke/os-image/packer/iso-cache/ubuntu-24.04.5-live-server-amd64.iso'
Media current: stdio file, overwriteable
Media status : is written , is appendable
Boot record : El Torito , MBR protective-msdos-label grub2-mbr cyl-align-off GPT
Media summary: 1 session, 1992425 data blocks, 3891m data, 82.2g free
Volume id : 'Ubuntu-Server 24.04.5 LTS amd64'
xorriso : UPDATE : 1 files restored ( 573b) in 1 seconds = 0.0xD
Extracted from ISO image: file '/boot/grub/grub.cfg'='/tmp/nvke-auto-iso-mf4zr5mn/original-grub.cfg'
xorriso : UPDATE : 1 files restored ( 39027b) in 1 seconds = 0.0xD
Extracted from ISO image: file '/md5sum.txt'='/tmp/nvke-auto-iso-mf4zr5mn/original-md5sum.txt'
xorriso 1.5.6 : RockRidge filesystem manipulator, libburnia project.
xorriso : NOTE : Loading ISO image tree from LBA 0
xorriso : UPDATE : 1195 nodes read in 1 seconds
libisofs: NOTE : Found hidden El-Torito image for EFI.
libisofs: NOTE : EFI image start and size: 1989719 * 2048 , 10160 * 512
xorriso : NOTE : Detected El-Torito boot information which currently is set to be discarded
Drive current: -indev '/home/bschmaus/nvke/os-image/packer/iso-cache/ubuntu-24.04.5-live-server-amd64.iso'
Media current: stdio file, overwriteable
Media status : is written , is appendable
Boot record : El Torito , MBR protective-msdos-label grub2-mbr cyl-align-off GPT
Media summary: 1 session, 1992425 data blocks, 3891m data, 82.2g free
Volume id : 'Ubuntu-Server 24.04.5 LTS amd64'
Drive current: -outdev '/home/bschmaus/nvke-nvsentinel-iso-002/nvke-nvsentinel-automatic-amd64.iso'
Media current: stdio file, overwriteable
Media status : is blank
Media summary: 0 sessions, 0 data blocks, 0 data, 82.2g free
xorriso : UPDATE : 1 files added in 1 seconds
Added to ISO image: file '/nvke/node.qcow2'='/home/bschmaus/nvke-nvsentinel-iso-002/payload/node.qcow2'
xorriso : UPDATE : 9 files added in 1 seconds
Added to ISO image: directory '/nvke-auto'='/tmp/nvke-auto-iso-mf4zr5mn/nvke-auto'
xorriso : UPDATE : 1 files added in 1 seconds
Added to ISO image: file '/autoinstall.yaml'='/home/bschmaus/nvke-qcow2-auto-iso/autoinstall.yaml'
xorriso : UPDATE : 1 files added in 1 seconds
Added to ISO image: file '/boot/grub/grub.cfg'='/tmp/nvke-auto-iso-mf4zr5mn/grub.cfg'
xorriso : UPDATE : 1 files added in 1 seconds
Added to ISO image: file '/md5sum.txt'='/tmp/nvke-auto-iso-mf4zr5mn/md5sum.txt'
xorriso : WARNING : -volid text problematic as automatic mount point name
xorriso : WARNING : -volid text does not comply to ISO 9660 / ECMA 119 rules
xorriso : NOTE : Replayed 22 boot related commands
xorriso : NOTE : Copying to System Area: 32768 bytes from file '--interval:imported_iso:0s-15s:zero_mbrpt,zero_gpt:/home/bschmaus/nvke/os-image/packer/iso-cache/ubuntu-24.04.5-live-server-amd64.iso'
xorriso : UPDATE : Writing: 8192s 0.2% fifo 100% buf 50%
xorriso : UPDATE : Writing: 16384s 0.4% fifo 100% buf 50% 20.1xD
xorriso : UPDATE : Writing: 155648s 3.6% fifo 100% buf 50% 204.9xD
xorriso : UPDATE : Writing: 294912s 6.8% fifo 100% buf 50% 205.5xD
xorriso : UPDATE : Writing: 434176s 10.1% fifo 100% buf 50% 205.5xD
xorriso : UPDATE : Writing: 581632s 13.5% fifo 100% buf 50% 217.6xD
xorriso : UPDATE : Writing: 720896s 16.7% fifo 100% buf 50% 205.4xD
xorriso : UPDATE : Writing: 860160s 19.9% fifo 100% buf 50% 205.5xD
xorriso : UPDATE : Writing: 999424s 23.2% fifo 100% buf 50% 205.5xD
xorriso : UPDATE : Writing: 1130496s 26.2% fifo 100% buf 50% 193.4xD
xorriso : UPDATE : Writing: 1269760s 29.4% fifo 100% buf 50% 205.5xD
xorriso : UPDATE : Writing: 1392640s 32.3% fifo 100% buf 50% 200.9xD
xorriso : UPDATE : Writing: 1531904s 35.5% fifo 100% buf 50% 205.5xD
xorriso : UPDATE : Writing: 1638400s 38.0% fifo 100% buf 50% 157.0xD
xorriso : UPDATE : Writing: 1720320s 39.9% fifo 100% buf 50% 120.5xD
xorriso : UPDATE : Writing: 1810432s 42.0% fifo 100% buf 50% 132.9xD
xorriso : UPDATE : Writing: 1890480s 43.8% fifo 99% buf 50% 118.1xD
xorriso : UPDATE : Writing: 1974272s 45.8% fifo 100% buf 50% 123.5xD
xorriso : UPDATE : Writing: 2040688s 47.3% fifo 99% buf 50% 97.9xD
xorriso : UPDATE : Writing: 2113536s 49.0% fifo 100% buf 50% 107.2xD
xorriso : UPDATE : Writing: 2195456s 50.9% fifo 100% buf 50% 120.7xD
xorriso : UPDATE : Writing: 2285568s 53.0% fifo 100% buf 50% 132.9xD
xorriso : UPDATE : Writing: 2359296s 54.7% fifo 100% buf 50% 108.7xD
xorriso : UPDATE : Writing: 2433024s 56.4% fifo 58% buf 50% 108.8xD
xorriso : UPDATE : Writing: 2498560s 57.9% fifo 100% buf 50% 96.7xD
xorriso : UPDATE : Writing: 2564096s 59.5% fifo 100% buf 50% 96.6xD
xorriso : UPDATE : Writing: 2639296s 61.2% fifo 83% buf 50% 110.9xD
xorriso : UPDATE : Writing: 2782688s 64.5% fifo 54% buf 50% 211.7xD
xorriso : UPDATE : Writing: 2927840s 67.9% fifo 71% buf 50% 214.2xD
xorriso : UPDATE : Writing: 3087280s 71.6% fifo 36% buf 50% 235.4xD
xorriso : UPDATE : Writing: 3325808s 77.1% fifo 15% buf 50% 352.2xD
xorriso : UPDATE : Writing: 3555328s 82.4% fifo 100% buf 50% 338.9xD
xorriso : UPDATE : Writing: 3792896s 88.0% fifo 100% buf 50% 350.8xD
xorriso : UPDATE : Writing: 4032000s 93.5% fifo 99% buf 50% 353.1xD
xorriso : UPDATE : Writing: 4276224s 99.2% fifo 95% buf 50% 360.7xD
ISO image produced: 4312321 sectors
Written to medium : 4312496 sectors at LBA 48
Writing to '/home/bschmaus/nvke-nvsentinel-iso-002/nvke-nvsentinel-automatic-amd64.iso' completed successfully.
xorriso : NOTE : Re-assessing -outdev '/home/bschmaus/nvke-nvsentinel-iso-002/nvke-nvsentinel-automatic-amd64.iso'
xorriso : NOTE : Loading ISO image tree from LBA 0
xorriso : UPDATE : 1208 nodes read in 1 seconds
libisofs: NOTE : Found hidden El-Torito image for EFI.
libisofs: NOTE : EFI image start and size: 4309813 * 2048 , 10160 * 512
xorriso : NOTE : Detected El-Torito boot information which currently is set to be discarded
Drive current: -dev '/home/bschmaus/nvke-nvsentinel-iso-002/nvke-nvsentinel-automatic-amd64.iso'
Media current: stdio file, overwriteable
Media status : is written , is appendable
Boot record : El Torito , MBR protective-msdos-label grub2-mbr cyl-align-off GPT
Media summary: 1 session, 4312369 data blocks, 8423m data, 74.0g free
Volume id : 'Ubuntu-Server 24.04.5 LTS amd64'
Built /home/bschmaus/nvke-nvsentinel-iso-002/nvke-nvsentinel-automatic-amd64.iso
Checksum: /home/bschmaus/nvke-nvsentinel-iso-002/nvke-nvsentinel-automatic-amd64.iso.sha256
Experimental: validate on a fresh blank VM before other nodes. No target disks were written by this build command.
nvke-nvsentinel-automatic-amd64.iso: OK
Booting the First Node
With a verified ISO, it gets copied to the hypervisor's libvirt image directory with scp, where a blank VM can boot from it. The hypervisor I am moving it to has three virtual machines we will use.
$ scp /home/bschmaus/nvke-nvsentinel-iso-002/nvke-nvsentinel-automatic-amd64.iso root@192.168.0.22:/var/lib/libvirt/images
root@192.168.0.22's password:
nvke-nvsentinel-automatic-amd64.iso 100% 8423MB 103.5MB/s 01:21
The install itself is not captured here because console capture isn't my thing, so we pick up once the first VM has come up. From my workstation, I connect to asus3-vm1 at 192.168.0.116 and switch to root. The checks here confirm that first boot did its work. cloud-init status --wait reports done, and the nvke-node-firstboot.service unit shows it ran to completion and installed the staged NVKE packages, including the OVN CNI images and charts for version 2.4.2. The remaining commands show a kubeadm version of v1.36.3, containerd, openvswitch-switch and nvke-agent all reporting active, a root filesystem with 179 GB free and the SHA256SUMS manifest from the add-on bundle verifying cleanly on the node. At this point I feel pretty confident the image delivered exactly what was put into it.
bschmaus@FVXW4M2W1Y ~ % ssh bschmaus@192.168.0.116
bschmaus@asus3-vm1:~$ sudo bash
root@asus3-vm1:/home/bschmaus# cloud-init status --wait
systemctl status nvke-node-firstboot.service --no-pager
test -f /var/lib/nvke-iso/packages-prepared
kubeadm version -o short
systemctl is-active containerd openvswitch-switch nvke-agent
df -h /
cd /opt/nvke-addons/nvsentinel
sha256sum -c SHA256SUMS
status: done
● nvke-node-firstboot.service - Install NVKE packages staged in the reference node image
Loaded: loaded (/etc/systemd/system/nvke-node-firstboot.service; enabled; preset: enabled)
Active: active (exited) since Thu 2026-11-05 20:22:08 UTC; 47min ago
Process: 1683 ExecStart=/bin/bash /usr/local/sbin/nvke-node-firstboot.sh (code=exited, status=0/SUCCESS)
Main PID: 1683 (code=exited, status=0/SUCCESS)
CPU: 35.156s
Nov 05 20:21:46 asus3-vm1 bash[2379]: Unpacking nvke-cni-ovn-images-2.4.2 (1+git.8af51273d52d) ...
Nov 05 20:21:48 asus3-vm1 bash[2379]: Setting up nvke-cni-ovn-images-2.4.2 (1+git.8af51273d52d) ...
Nov 05 20:21:49 asus3-vm1 bash[2379]: ovn-images_2.4.2_linux_amd64.tgz: OK
Nov 05 20:22:01 asus3-vm1 bash[2379]: Setting up nvke-cni-ovn-charts-2.4.2 (1+git.8af51273d52d) ...
Nov 05 20:22:02 asus3-vm1 bash[2522]: Running kernel seems to be up-to-date.
Nov 05 20:22:02 asus3-vm1 bash[2522]: No services need to be restarted.
Nov 05 20:22:02 asus3-vm1 bash[2522]: No containers need to be restarted.
Nov 05 20:22:02 asus3-vm1 bash[2522]: No user sessions are running outdated binaries.
Nov 05 20:22:02 asus3-vm1 bash[2522]: No VM guests are running outdated hypervisor (qemu) binaries on this host.
Nov 05 20:22:08 asus3-vm1 systemd[1]: Finished nvke-node-firstboot.service - Install NVKE packages staged in the reference node image.
v1.36.3
active
active
active
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv 195G 7.1G 179G 4% /
./image-map.tsv: OK
./images-required.txt: OK
./images/001.tar: OK
./images/002.tar: OK
./nvkr-commit.txt: OK
./nvsentinel-source-commit.txt: OK
./nvsentinel/.helmignore: OK
./nvsentinel/Chart.lock: OK
./nvsentinel/Chart.yaml: OK
./nvsentinel/charts/csp-health-monitor/.helmignore: OK
./nvsentinel/charts/csp-health-monitor/Chart.yaml: OK
./nvsentinel/charts/csp-health-monitor/templates/_helpers.tpl: OK
./nvsentinel/charts/csp-health-monitor/templates/clusterrole.yaml: OK
./nvsentinel/charts/csp-health-monitor/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/csp-health-monitor/templates/configmap.yaml: OK
./nvsentinel/charts/csp-health-monitor/templates/deployment.yaml: OK
./nvsentinel/charts/csp-health-monitor/templates/serviceaccount.yaml: OK
./nvsentinel/charts/csp-health-monitor/values.yaml: OK
./nvsentinel/charts/event-exporter/Chart.yaml: OK
./nvsentinel/charts/event-exporter/templates/_helpers.tpl: OK
./nvsentinel/charts/event-exporter/templates/configmap.yaml: OK
./nvsentinel/charts/event-exporter/templates/deployment.yaml: OK
./nvsentinel/charts/event-exporter/values.yaml: OK
./nvsentinel/charts/fault-quarantine/.helmignore: OK
./nvsentinel/charts/fault-quarantine/Chart.yaml: OK
./nvsentinel/charts/fault-quarantine/templates/_helpers.tpl: OK
./nvsentinel/charts/fault-quarantine/templates/clusterrole.yaml: OK
./nvsentinel/charts/fault-quarantine/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/fault-quarantine/templates/configmap.yaml: OK
./nvsentinel/charts/fault-quarantine/templates/deployment.yaml: OK
./nvsentinel/charts/fault-quarantine/templates/serviceaccount.yaml: OK
./nvsentinel/charts/fault-quarantine/values.yaml: OK
./nvsentinel/charts/fault-remediation/.helmignore: OK
./nvsentinel/charts/fault-remediation/Chart.yaml: OK
./nvsentinel/charts/fault-remediation/files/log-collector-job.yaml: OK
./nvsentinel/charts/fault-remediation/templates/_helpers.tpl: OK
./nvsentinel/charts/fault-remediation/templates/clusterrole.yaml: OK
./nvsentinel/charts/fault-remediation/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/fault-remediation/templates/configmap.yaml: OK
./nvsentinel/charts/fault-remediation/templates/deployment.yaml: OK
./nvsentinel/charts/fault-remediation/templates/log-collector-rbac.yaml: OK
./nvsentinel/charts/fault-remediation/templates/serviceaccount.yaml: OK
./nvsentinel/charts/fault-remediation/values.yaml: OK
./nvsentinel/charts/gpu-health-monitor/.helmignore: OK
./nvsentinel/charts/gpu-health-monitor/Chart.yaml: OK
./nvsentinel/charts/gpu-health-monitor/files/dcgmerrorsmapping.csv: OK
./nvsentinel/charts/gpu-health-monitor/templates/NOTES.txt: OK
./nvsentinel/charts/gpu-health-monitor/templates/_helpers.tpl: OK
./nvsentinel/charts/gpu-health-monitor/templates/configmap.yaml: OK
./nvsentinel/charts/gpu-health-monitor/templates/daemonset-dcgm-3.x.yaml: OK
./nvsentinel/charts/gpu-health-monitor/templates/daemonset-dcgm-4.x.yaml: OK
./nvsentinel/charts/gpu-health-monitor/tests/dcgm_runtime_debounce_test.yaml: OK
./nvsentinel/charts/gpu-health-monitor/tests/dcgm_startup_gate_test.yaml: OK
./nvsentinel/charts/gpu-health-monitor/tests/gpu_deploy_client_affinity_test.yaml: OK
./nvsentinel/charts/gpu-health-monitor/tests/pc_auth_strictness_test.yaml: OK
./nvsentinel/charts/gpu-health-monitor/tests/priorityclassname_test.yaml: OK
./nvsentinel/charts/gpu-health-monitor/tests/publish_wiring_test.yaml: OK
./nvsentinel/charts/gpu-health-monitor/values.yaml: OK
./nvsentinel/charts/health-events-analyzer/.helmignore: OK
./nvsentinel/charts/health-events-analyzer/Chart.yaml: OK
./nvsentinel/charts/health-events-analyzer/templates/_helpers.tpl: OK
./nvsentinel/charts/health-events-analyzer/templates/configmap.yaml: OK
./nvsentinel/charts/health-events-analyzer/templates/deployment.yaml: OK
./nvsentinel/charts/health-events-analyzer/templates/node-recovery-rbac.yaml: OK
./nvsentinel/charts/health-events-analyzer/templates/role.yaml: OK
./nvsentinel/charts/health-events-analyzer/templates/rolebinding.yaml: OK
./nvsentinel/charts/health-events-analyzer/templates/serviceaccount.yaml: OK
./nvsentinel/charts/health-events-analyzer/tests/node_recovery_rbac_test.yaml: OK
./nvsentinel/charts/health-events-analyzer/values.yaml: OK
./nvsentinel/charts/incluster-file-server/Chart.yaml: OK
./nvsentinel/charts/incluster-file-server/templates/_helpers.tpl: OK
./nvsentinel/charts/incluster-file-server/templates/configmap-nginx.yaml: OK
./nvsentinel/charts/incluster-file-server/templates/deployment.yaml: OK
./nvsentinel/charts/incluster-file-server/templates/service.yaml: OK
./nvsentinel/charts/incluster-file-server/values.yaml: OK
./nvsentinel/charts/janitor-provider/Chart.yaml: OK
./nvsentinel/charts/janitor-provider/templates/_helpers.tpl: OK
./nvsentinel/charts/janitor-provider/templates/certificate.yaml: OK
./nvsentinel/charts/janitor-provider/templates/clusterrole.yaml: OK
./nvsentinel/charts/janitor-provider/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/janitor-provider/templates/deployment.yaml: OK
./nvsentinel/charts/janitor-provider/templates/networkpolicy.yaml: OK
./nvsentinel/charts/janitor-provider/templates/role.yaml: OK
./nvsentinel/charts/janitor-provider/templates/rolebinding.yaml: OK
./nvsentinel/charts/janitor-provider/templates/service.yaml: OK
./nvsentinel/charts/janitor-provider/templates/serviceaccount.yaml: OK
./nvsentinel/charts/janitor-provider/values.yaml: OK
./nvsentinel/charts/janitor/.helmignore: OK
./nvsentinel/charts/janitor/Chart.yaml: OK
./nvsentinel/charts/janitor/crds/external_remediation.crd.yaml: OK
./nvsentinel/charts/janitor/crds/janitor.dgxc.nvidia.com_gpuresets.yaml: OK
./nvsentinel/charts/janitor/crds/janitor.dgxc.nvidia.com_rebootnodes.yaml: OK
./nvsentinel/charts/janitor/crds/janitor.dgxc.nvidia.com_terminatenodes.yaml: OK
./nvsentinel/charts/janitor/templates/_helpers.tpl: OK
./nvsentinel/charts/janitor/templates/certificate-metrics.yaml: OK
./nvsentinel/charts/janitor/templates/clusterrole.yaml: OK
./nvsentinel/charts/janitor/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/janitor/templates/configmap.yaml: OK
./nvsentinel/charts/janitor/templates/deployment.yaml: OK
./nvsentinel/charts/janitor/templates/issuer.yaml: OK
./nvsentinel/charts/janitor/templates/networkpolicy.yaml: OK
./nvsentinel/charts/janitor/templates/service.yaml: OK
./nvsentinel/charts/janitor/templates/serviceaccount.yaml: OK
./nvsentinel/charts/janitor/templates/webhook.yaml: OK
./nvsentinel/charts/janitor/values.schema.json: OK
./nvsentinel/charts/janitor/values.yaml: OK
./nvsentinel/charts/k8s-datastore/Chart.yaml: OK
./nvsentinel/charts/k8s-datastore/crds/health_event.crd.yaml: OK
./nvsentinel/charts/k8s-datastore/values.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/Chart.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/templates/_helpers.tpl: OK
./nvsentinel/charts/kubernetes-object-monitor/templates/clusterrole.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/templates/configmap.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/templates/deployment.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/templates/serviceaccount.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/tests/pc_auth_strictness_test.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/tests/publish_wiring_test.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/values.yaml: OK
./nvsentinel/charts/labeler/.helmignore: OK
./nvsentinel/charts/labeler/Chart.yaml: OK
./nvsentinel/charts/labeler/templates/_helpers.tpl: OK
./nvsentinel/charts/labeler/templates/clusterrole.yaml: OK
./nvsentinel/charts/labeler/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/labeler/templates/configmap.yaml: OK
./nvsentinel/charts/labeler/templates/deployment.yaml: OK
./nvsentinel/charts/labeler/templates/serviceaccount.yaml: OK
./nvsentinel/charts/labeler/tests/priorityclassname_test.yaml: OK
./nvsentinel/charts/labeler/values.yaml: OK
./nvsentinel/charts/lifecycle-manager/.helmignore: OK
./nvsentinel/charts/lifecycle-manager/Chart.yaml: OK
./nvsentinel/charts/lifecycle-manager/crds/maintenance_request.crd.yaml: OK
./nvsentinel/charts/lifecycle-manager/crds/nvsentinel.nvidia.com_validationrequests.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/_helpers.tpl: OK
./nvsentinel/charts/lifecycle-manager/templates/certificate-metrics.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/clusterrole.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/configmap-templates.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/configmap.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/deployment.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/issuer.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/networkpolicy.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/service.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/serviceaccount.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/webhook.yaml: OK
./nvsentinel/charts/lifecycle-manager/tests/deployment_test.yaml: OK
./nvsentinel/charts/lifecycle-manager/tests/pc_auth_strictness_test.yaml: OK
./nvsentinel/charts/lifecycle-manager/tests/publish_wiring_test.yaml: OK
./nvsentinel/charts/lifecycle-manager/tests/rbac_test.yaml: OK
./nvsentinel/charts/lifecycle-manager/tests/webhook_test.yaml: OK
./nvsentinel/charts/lifecycle-manager/values.yaml: OK
./nvsentinel/charts/metadata-collector/Chart.yaml: OK
./nvsentinel/charts/metadata-collector/templates/_helpers.tpl: OK
./nvsentinel/charts/metadata-collector/templates/clusterrole.yaml: OK
./nvsentinel/charts/metadata-collector/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/metadata-collector/templates/daemonset.yaml: OK
./nvsentinel/charts/metadata-collector/templates/resourceclaimtemplate.yaml: OK
./nvsentinel/charts/metadata-collector/templates/serviceaccount.yaml: OK
./nvsentinel/charts/metadata-collector/tests/daemonset_test.yaml: OK
./nvsentinel/charts/metadata-collector/tests/gpu_deploy_client_affinity_test.yaml: OK
./nvsentinel/charts/metadata-collector/tests/resourceclaimtemplate_test.yaml: OK
./nvsentinel/charts/metadata-collector/values.yaml: OK
./nvsentinel/charts/mongodb-store/.helmignore: OK
./nvsentinel/charts/mongodb-store/Chart.lock: OK
./nvsentinel/charts/mongodb-store/Chart.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/.helmignore: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/Chart.lock: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/Chart.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/README.md: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/.helmignore: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/Chart.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/README.md: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_affinities.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_capabilities.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_compatibility.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_errors.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_images.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_ingress.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_labels.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_names.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_resources.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_secrets.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_storage.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_tplvalues.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_utils.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_warnings.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/validations/_cassandra.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/validations/_mariadb.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/validations/_mongodb.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/validations/_mysql.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/validations/_postgresql.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/validations/_redis.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/validations/_validations.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/values.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/NOTES.txt: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/_helpers.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/arbiter/configmap.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/arbiter/headless-svc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/arbiter/pdb.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/arbiter/statefulset.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/backup/cronjob.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/backup/pvc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/common-scripts-cm.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/configmap.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/extra-list.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/hidden/configmap.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/hidden/external-access-svc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/hidden/headless-svc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/hidden/pdb.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/hidden/statefulset.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/initialization-configmap.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/metrics-svc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/networkpolicy.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/prometheusrule.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/psp.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/replicaset/external-access-svc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/replicaset/headless-svc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/replicaset/pdb.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/replicaset/scripts-configmap.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/replicaset/statefulset.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/replicaset/svc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/role.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/rolebinding.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/secrets-ca.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/secrets.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/serviceaccount.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/servicemonitor.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/standalone/dep-sts.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/standalone/pdb.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/standalone/pvc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/standalone/svc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/values.schema.json: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/values.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/.helmignore: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/Chart.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/README.md: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/templates/NOTES.txt: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/templates/_helpers.tpl: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/templates/cluster-secret.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/templates/cluster.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/templates/clustersync-secret.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/templates/clustersync.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/values.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/.helmignore: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/Chart.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/LICENSE.txt: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/README.md: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/crds/crd.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/templates/NOTES.txt: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/templates/_helpers.tpl: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/templates/deployment.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/templates/namespace.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/templates/role-binding.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/templates/role.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/values.yaml: OK
./nvsentinel/charts/mongodb-store/templates/_helpers.tpl: OK
./nvsentinel/charts/mongodb-store/templates/_init-eval.tpl: OK
./nvsentinel/charts/mongodb-store/templates/certmanager.yaml: OK
./nvsentinel/charts/mongodb-store/templates/configmap.yaml: OK
./nvsentinel/charts/mongodb-store/templates/jobs.yaml: OK
./nvsentinel/charts/mongodb-store/templates/metrics-service.yaml: OK
./nvsentinel/charts/mongodb-store/templates/networkpolicy.yaml: OK
./nvsentinel/charts/mongodb-store/templates/pv.yaml: OK
./nvsentinel/charts/mongodb-store/templates/role.yaml: OK
./nvsentinel/charts/mongodb-store/templates/rolebinding.yaml: OK
./nvsentinel/charts/mongodb-store/tests/cluster_domain_test.yaml: OK
./nvsentinel/charts/mongodb-store/tests/networkpolicy_test.yaml: OK
./nvsentinel/charts/mongodb-store/tests/psmdb_versions_test.yaml: OK
./nvsentinel/charts/mongodb-store/values.yaml: OK
./nvsentinel/charts/nic-health-monitor/.helmignore: OK
./nvsentinel/charts/nic-health-monitor/Chart.yaml: OK
./nvsentinel/charts/nic-health-monitor/templates/_helpers.tpl: OK
./nvsentinel/charts/nic-health-monitor/templates/configmap.yaml: OK
./nvsentinel/charts/nic-health-monitor/templates/daemonset.yaml: OK
./nvsentinel/charts/nic-health-monitor/tests/daemonset_test.yaml: OK
./nvsentinel/charts/nic-health-monitor/tests/pc_auth_strictness_test.yaml: OK
./nvsentinel/charts/nic-health-monitor/tests/publish_wiring_test.yaml: OK
./nvsentinel/charts/nic-health-monitor/values.yaml: OK
./nvsentinel/charts/node-drainer/.helmignore: OK
./nvsentinel/charts/node-drainer/Chart.yaml: OK
./nvsentinel/charts/node-drainer/templates/_helpers.tpl: OK
./nvsentinel/charts/node-drainer/templates/clusterrole.yaml: OK
./nvsentinel/charts/node-drainer/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/node-drainer/templates/configmap.yaml: OK
./nvsentinel/charts/node-drainer/templates/deployment.yaml: OK
./nvsentinel/charts/node-drainer/templates/serviceaccount.yaml: OK
./nvsentinel/charts/node-drainer/tests/pod_drain_policies_test.yaml: OK
./nvsentinel/charts/node-drainer/values.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/Chart.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/templates/_helpers.tpl: OK
./nvsentinel/charts/nvcre-certification-monitor/templates/clusterrole.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/templates/configmap.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/templates/deployment.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/templates/serviceaccount.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/tests/pc_auth_strictness_test.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/tests/publish_wiring_test.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/values.yaml: OK
./nvsentinel/charts/postgresql/.helmignore: OK
./nvsentinel/charts/postgresql/Chart.lock: OK
./nvsentinel/charts/postgresql/Chart.yaml: OK
./nvsentinel/charts/postgresql/README.md: OK
./nvsentinel/charts/postgresql/charts/common/.helmignore: OK
./nvsentinel/charts/postgresql/charts/common/Chart.yaml: OK
./nvsentinel/charts/postgresql/charts/common/README.md: OK
./nvsentinel/charts/postgresql/charts/common/templates/_affinities.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_capabilities.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_compatibility.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_errors.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_images.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_ingress.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_labels.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_names.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_resources.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_secrets.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_storage.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_tplvalues.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_utils.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_warnings.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/validations/_cassandra.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/validations/_mariadb.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/validations/_mongodb.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/validations/_mysql.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/validations/_postgresql.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/validations/_redis.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/validations/_validations.tpl: OK
./nvsentinel/charts/postgresql/charts/common/values.yaml: OK
./nvsentinel/charts/postgresql/templates/NOTES.txt: OK
./nvsentinel/charts/postgresql/templates/_helpers.tpl: OK
./nvsentinel/charts/postgresql/templates/backup/cronjob.yaml: OK
./nvsentinel/charts/postgresql/templates/backup/networkpolicy.yaml: OK
./nvsentinel/charts/postgresql/templates/backup/pvc.yaml: OK
./nvsentinel/charts/postgresql/templates/extra-list.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/configmap.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/extended-configmap.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/initialization-configmap.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/metrics-configmap.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/metrics-svc.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/networkpolicy.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/pdb.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/preinitialization-configmap.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/servicemonitor.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/statefulset.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/svc-headless.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/svc.yaml: OK
./nvsentinel/charts/postgresql/templates/prometheusrule.yaml: OK
./nvsentinel/charts/postgresql/templates/psp.yaml: OK
./nvsentinel/charts/postgresql/templates/read/extended-configmap.yaml: OK
./nvsentinel/charts/postgresql/templates/read/metrics-configmap.yaml: OK
./nvsentinel/charts/postgresql/templates/read/metrics-svc.yaml: OK
./nvsentinel/charts/postgresql/templates/read/networkpolicy.yaml: OK
./nvsentinel/charts/postgresql/templates/read/pdb.yaml: OK
./nvsentinel/charts/postgresql/templates/read/servicemonitor.yaml: OK
./nvsentinel/charts/postgresql/templates/read/statefulset.yaml: OK
./nvsentinel/charts/postgresql/templates/read/svc-headless.yaml: OK
./nvsentinel/charts/postgresql/templates/read/svc.yaml: OK
./nvsentinel/charts/postgresql/templates/role.yaml: OK
./nvsentinel/charts/postgresql/templates/rolebinding.yaml: OK
./nvsentinel/charts/postgresql/templates/secrets.yaml: OK
./nvsentinel/charts/postgresql/templates/serviceaccount.yaml: OK
./nvsentinel/charts/postgresql/templates/tls-secrets.yaml: OK
./nvsentinel/charts/postgresql/values.schema.json: OK
./nvsentinel/charts/postgresql/values.yaml: OK
./nvsentinel/charts/preflight/.helmignore: OK
./nvsentinel/charts/preflight/Chart.yaml: OK
./nvsentinel/charts/preflight/crds/preflight.nvsentinel.nvidia.com_preflightconfigs.yaml: OK
./nvsentinel/charts/preflight/files/ndv4-topo.xml: OK
./nvsentinel/charts/preflight/files/ndv5-topo.xml: OK
./nvsentinel/charts/preflight/templates/_helpers.tpl: OK
./nvsentinel/charts/preflight/templates/ca-certificate.yaml: OK
./nvsentinel/charts/preflight/templates/ca-issuer.yaml: OK
./nvsentinel/charts/preflight/templates/certificate.yaml: OK
./nvsentinel/charts/preflight/templates/clusterrole.yaml: OK
./nvsentinel/charts/preflight/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/preflight/templates/configmap.yaml: OK
./nvsentinel/charts/preflight/templates/deployment.yaml: OK
./nvsentinel/charts/preflight/templates/image-cache-daemonset.yaml: OK
./nvsentinel/charts/preflight/templates/mutatingwebhook.yaml: OK
./nvsentinel/charts/preflight/templates/networkpolicy.yaml: OK
./nvsentinel/charts/preflight/templates/selfsigned-issuer.yaml: OK
./nvsentinel/charts/preflight/templates/service.yaml: OK
./nvsentinel/charts/preflight/templates/serviceaccount.yaml: OK
./nvsentinel/charts/preflight/tests/dra_rbac_test.yaml: OK
./nvsentinel/charts/preflight/tests/mutatingwebhook_test.yaml: OK
./nvsentinel/charts/preflight/tests/pc_auth_strictness_test.yaml: OK
./nvsentinel/charts/preflight/tests/publish_wiring_test.yaml: OK
./nvsentinel/charts/preflight/values.schema.json: OK
./nvsentinel/charts/preflight/values.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/Chart.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/templates/_helpers.tpl: OK
./nvsentinel/charts/slurm-drain-monitor/templates/clusterrole.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/templates/configmap.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/templates/deployment.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/templates/serviceaccount.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/tests/pc_auth_strictness_test.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/tests/publish_wiring_test.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/values.yaml: OK
./nvsentinel/charts/syslog-health-monitor/.helmignore: OK
./nvsentinel/charts/syslog-health-monitor/Chart.yaml: OK
./nvsentinel/charts/syslog-health-monitor/templates/_helpers.tpl: OK
./nvsentinel/charts/syslog-health-monitor/templates/configmap.yaml: OK
./nvsentinel/charts/syslog-health-monitor/templates/daemonset-kata.yaml: OK
./nvsentinel/charts/syslog-health-monitor/templates/daemonset-regular.yaml: OK
./nvsentinel/charts/syslog-health-monitor/tests/pc_auth_strictness_test.yaml: OK
./nvsentinel/charts/syslog-health-monitor/tests/publish_wiring_test.yaml: OK
./nvsentinel/charts/syslog-health-monitor/values.yaml: OK
./nvsentinel/files/node-condition-cleanup.sh: OK
./nvsentinel/policies/README.md: OK
./nvsentinel/policies/must-have-sbom.yaml: OK
./nvsentinel/policies/must-have-slsa.yaml: OK
./nvsentinel/templates/NOTES.txt: OK
./nvsentinel/templates/_external-mongo-init-eval.tpl: OK
./nvsentinel/templates/_helpers.tpl: OK
./nvsentinel/templates/_platform-connector-config.tpl: OK
./nvsentinel/templates/certmanager-postgresql.yaml: OK
./nvsentinel/templates/clusterrole.yaml: OK
./nvsentinel/templates/clusterrolebinding.yaml: OK
./nvsentinel/templates/configmap-datastore.yaml: OK
./nvsentinel/templates/configmap.yaml: OK
./nvsentinel/templates/daemonset.yaml: OK
./nvsentinel/templates/job-external-mongodb-setup.yaml: OK
./nvsentinel/templates/networkpolicy.yaml: OK
./nvsentinel/templates/node-condition-cleanup-hook.yaml: OK
./nvsentinel/templates/platform-connector-deployment-certificate.yaml: OK
./nvsentinel/templates/platform-connector-deployment-configmap.yaml: OK
./nvsentinel/templates/platform-connector-deployment-deployment.yaml: OK
./nvsentinel/templates/platform-connector-deployment-networkpolicy.yaml: OK
./nvsentinel/templates/platform-connector-deployment-pdb.yaml: OK
./nvsentinel/templates/platform-connector-deployment-service.yaml: OK
./nvsentinel/templates/platform-connector-deployment-serviceaccount.yaml: OK
./nvsentinel/templates/platform-connector-deployment-validation.yaml: OK
./nvsentinel/templates/podMonitor.yaml: OK
./nvsentinel/templates/serviceaccount.yaml: OK
./nvsentinel/tests/cluster_domain_test.yaml: OK
./nvsentinel/tests/csp_health_monitor_publish_wiring_test.yaml: OK
./nvsentinel/tests/datastore_credentials_test.yaml: OK
./nvsentinel/tests/deployed_crversion_test.yaml: OK
./nvsentinel/tests/external_mongodb_setup_job_test.yaml: OK
./nvsentinel/tests/health_events_analyzer_publish_wiring_test.yaml: OK
./nvsentinel/tests/pc_auth_cross_node_test.yaml: OK
./nvsentinel/tests/platform_connector_deployment_networkpolicy_test.yaml: OK
./nvsentinel/tests/platform_connector_deployment_server_test.yaml: OK
./nvsentinel/tests/platform_connector_deployment_test.yaml: OK
./nvsentinel/tests/platform_connector_retry_test.yaml: OK
./nvsentinel/tests/strict_render_test.yaml: OK
./nvsentinel/tests/values/publishers-on-deployment.yaml: OK
./nvsentinel/values-aicr-aws-uat.yaml: OK
./nvsentinel/values-aicr-gcp-uat.yaml: OK
./nvsentinel/values-aicr-uat.yaml: OK
./nvsentinel/values-atlas-gcp.yaml: OK
./nvsentinel/values-aws-docdb.yaml: OK
./nvsentinel/values-cosmosdb-test.yaml: OK
./nvsentinel/values-full.yaml: OK
./nvsentinel/values-npd-gpu-services.yaml: OK
./nvsentinel/values-npd-remediation.yaml: OK
./nvsentinel/values-nvcre.yaml: OK
./nvsentinel/values-postgresql.yaml: OK
./nvsentinel/values-preflight-kube.yaml: OK
./nvsentinel/values-remediation.yaml: OK
./nvsentinel/values-tilt-arm64.yaml: OK
./nvsentinel/values-tilt-deployment.yaml: OK
./nvsentinel/values-tilt-mongodb-percona.yaml: OK
./nvsentinel/values-tilt-mongodb-tls-disabled-percona.yaml: OK
./nvsentinel/values-tilt-mongodb-tls-disabled.yaml: OK
./nvsentinel/values-tilt-mongodb.yaml: OK
./nvsentinel/values-tilt-nvcre.yaml: OK
./nvsentinel/values-tilt-postgresql.yaml: OK
./nvsentinel/values-tilt-socket.yaml: OK
./nvsentinel/values-tilt.yaml: OK
./nvsentinel/values.yaml: OK
./values-lab.yaml: OK
Boot the other two nodes with the same ISO and repeat the steps we did for the first node. These three nodes will now form the basis for our three node cluster.
Preparing the Bootstrap Host
Now we use the manual start that the staging step reserved for node one. systemctl start --no-block kicks off the bootstrap host service and returns right away, and journalctl -fu follows its log. Keep in mind the --no-block matters here because the unit is a oneshot, and without it the shell would sit waiting for the service to finish. The log shows nvkectl at v0.3.0-rc.2, kind at v0.32.0 and clusterctl at v1.13.6, followed by a message saying the host is prepared and that no seed cluster was created and no bootstrap was run. Nothing has been built yet, only the tooling is prepared and made ready.
root@asus3-vm1:/home/bschmaus# systemctl start --no-block nvke-bootstrap-host-setup.service
root@asus3-vm1:/home/bschmaus# journalctl -fu nvke-bootstrap-host-setup.service
Nov 05 21:12:37 asus3-vm1 python3[11667]: }
Nov 05 21:12:37 asus3-vm1 python3[11667]: ]
Nov 05 21:12:37 asus3-vm1 python3[11609]: 2026-11-05 21:12:37,227 INFO Run: /opt/nvke-bootstrap/bin/nvkectl version
Nov 05 21:12:37 asus3-vm1 python3[11679]: nvkectl v0.3.0-rc.2-18-ga9a0537b (commit: a9a0537b, built: 2026-10-29T22:05:19+00:00)
Nov 05 21:12:37 asus3-vm1 python3[11609]: 2026-11-05 21:12:37,243 INFO Run: /opt/nvke-bootstrap/bin/kind version
Nov 05 21:12:37 asus3-vm1 python3[11691]: kind v0.32.0 go1.26.3 linux/amd64
Nov 05 21:12:37 asus3-vm1 python3[11609]: 2026-11-05 21:12:37,247 INFO Run: /opt/nvke-bootstrap/bin/clusterctl version
Nov 05 21:12:37 asus3-vm1 python3[11696]: clusterctl version: &version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.6", GitCommit:"bd8162edfe045c1bb490d431b3acddd568bdb887", GitTreeState:"clean", BuildDate:"2026-09-08T16:11:41Z", GoVersion:"go1.25.13", Compiler:"gc", Platform:"linux/amd64"}
Nov 05 21:12:38 asus3-vm1 python3[11609]: 2026-11-05 21:12:38,041 INFO Bootstrap host prepared. No seed cluster was created and no cluster bootstrap was run.
Nov 05 21:12:38 asus3-vm1 systemd[1]: Finished nvke-bootstrap-host-setup.service - Prepare this node as an NVKE bootstrap host (manual start only).
Before moving on, here is the set of values this lab uses. These numbers show up again in the site spec, so having them in one place helps when reading the output further down.
| Setting | Reference lab value |
|---|---|
| Nodes | asus3-vm1, asus3-vm2, asus3-vm3 |
| Management IPs | 192.168.0.116, 192.168.0.117, 192.168.0.118 |
| Kubernetes / OVN | 1.36.3 / 2.4.2 |
| API VIP | 10.250.0.10 |
| Pod / service CIDRs | 10.244.0.0/16 / 10.96.0.0/12 |
| BGP node AS / gateway peer AS | 65100 / 65000 |
| Site / SPIRE trust domain | nvke-demo / example.com |
Generating and Validating the Site Spec
Everything for the bootstrap lives in one directory. The block creates /root/nvke-demo with mode 0700 and sets umask 077, so every file created from here on, including the kubeconfig and the etcd encryption key, is readable only by root.
root@asus3-vm1:/home/bschmaus# install -d -m 0700 /root/nvke-demo
root@asus3-vm1:/home/bschmaus# cd /root/nvke-demo
root@asus3-vm1:/home/bschmaus# umask 077
The node list comes next. A small hosts.csv holds each node's name and management IP, with the TPM endorsement key column left empty since no TPM attestation is used in this lab.
root@asus3-vm1:~/nvke-demo# cat >hosts.csv <<'EOF'
name,ip,tpm_ek_thumbprint
asus3-vm1,192.168.0.116,
asus3-vm2,192.168.0.117,
asus3-vm3,192.168.0.118,
EOF
Now the cluster definition gets generated. nvkectl generate sitespec takes the node list and a series of --set options that pin the Kubernetes version to 1.36.3, the CNI to ovn-kubernetes at 2.4.2, the control plane virtual IP, the pod and service CIDRs, the kubelet's reserved memory and the SPIRE trust domain. The template it writes is heavily commented, and the comments explain each field and how optional pieces like the TPM allowlist and a SPIRE CA supplied by the operator behave. The dry run is the useful check. nvkectl bootstrap site --dry-run loads the file, validates it and prints the effective spec, which includes the three nodes, the pinned versions, etcdEncryption: true and a node agent port of 15005. It ends with Dry run: site spec is valid. No workflow dispatched.
root@asus3-vm1:~/nvke-demo# nvkectl generate sitespec nvke-demo --nodes hosts.csv --set cluster.k8sVersion=1.36.3 --set cluster.cniName=ovn-kubernetes --set cluster.cniVersion=2.4.2 --set cluster.controlPlaneVIP=10.250.0.10 --set cluster.podCidr=10.244.0.0/16 --set cluster.serviceCidr=10.96.0.0/12 --set cluster.kubeletConfig.systemReservedMemory=4Gi --set spire.trustDomain=nvke-demo.example.com >sitespec.yaml
root@asus3-vm1:~/nvke-demo# cat sitespec.yaml
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
# Site spec template rendered by "nvkectl generate sitespec".
#
# The site spec describes the pre-provisioned nodes and the vanilla Kubernetes
# cluster to build on them. nvkectl loads and validates this file before it
# dispatches any work, so a typo or a missing field fails fast with a clear
# error. See docs/sitespec.md for the full field reference.
# Name of the Kubernetes cluster to create (lowercase RFC 1123 label).
clusterName: "nvke-demo"
cluster:
# Pre-provisioned control-plane nodes. The site spec does not select which
# node runs kubeadm init.
# tpmEkThumbprint is optional: the SHA-256 of the node's TPM endorsement key,
# read from the host before the bootstrap (spire-tpm-plugin's
# get_tpm_pubhash). Every thumbprint listed here is allowlisted on the SPIRE
# server nvkectl deploys, and nothing else is — a node without one cannot
# attest, and a file with none deploys no TPM attestor at all. It is a public
# key hash, not a secret.
nodes:
- name: "asus3-vm1"
ip: "192.168.0.116"
- name: "asus3-vm2"
ip: "192.168.0.117"
- name: "asus3-vm3"
ip: "192.168.0.118"
# Kubernetes version to install — plain semantic version, no leading "v".
k8sVersion: "1.36.3"
# CNI plugin. New management clusters use "ovn-kubernetes".
cniName: "ovn-kubernetes"
# Optional CNI plugin version. Empty uses the node-agent's built-in default.
# The value must be an OVN-Kubernetes version.
cniVersion: "2.4.2"
# Pod network CIDR. Multiple CIDRs may be given as a comma-separated string.
podCidr: "10.244.0.0/16"
# Service network CIDR.
serviceCidr: "10.96.0.0/12"
# Encryption at rest for the Kubernetes API's etcd data. On by default. It can
# only be chosen at bootstrap: a cluster built with it off cannot be encrypted
# afterwards. Set to false to opt out.
# etcdEncryption: true
# IPv4 API endpoint advertised by kube-vip. Required when nodes contains more
# than one entry. It needs BGP routing to the nodes and must not overlap
# podCidr or serviceCidr. It may equal a declared node IP. A single-node
# cluster may leave it empty and use that node's IP as its fixed endpoint.
controlPlaneVIP: "10.250.0.10"
# Optional: gRPC port nvke-agent listens on. Defaults to "15005".
# nodeAgentPort: "15005"
# Optional kubelet reservations applied before kubeadm init/join.
kubeletConfig:
systemReservedMemory: "4Gi"
# Identity plane. Optional.
#
# SPIRE can chain its CA to one you bring, so a node given that CA's
# certificate when it was provisioned verifies the SPIRE server on first
# contact. Set NVKECTL_SPIRE_CA_CERT_FILE and NVKECTL_SPIRE_CA_KEY_FILE to its
# certificate and key, and upstreamCA below. Without them SPIRE is self-signed.
spire:
trustDomain: "nvke-demo.example.com"
root@asus3-vm1:~/nvke-demo# nvkectl bootstrap site -f sitespec.yaml --disable-mtls --dry-run
Trace ID: b4db09bd24f39ef89fe287640fa04a88
time=2026-11-05T21:17:43.264Z level=INFO msg="loaded site spec" component=bootstrap-site path=sitespec.yaml cluster=nvke-demo trace_id=b4db09bd24f39ef89fe287640fa04a88 span_id=03f16e31047b4766
Loaded site spec:
clusterName: nvke-demo
infra:
provider: provisioned
cluster:
nodes:
- name: asus3-vm1
ip: 192.168.0.116
- name: asus3-vm2
ip: 192.168.0.117
- name: asus3-vm3
ip: 192.168.0.118
k8sVersion: 1.36.3
cniName: ovn-kubernetes
cniVersion: 2.4.2
controlPlaneVIP: 10.250.0.10
podCidr: 10.244.0.0/16
serviceCidr: 10.96.0.0/12
nodeAgentPort: "15005"
etcdEncryption: true
kubeletConfig:
systemReservedMemory: 4Gi
spire:
trustDomain: nvke-demo.example.com
Dry run: site spec is valid. No workflow dispatched.
Trace ID: b4db09bd24f39ef89fe287640fa04a88
Bootstrapping the Cluster
If everything looks good we can move onto the real deployment. This is the same command without --dry-run, plus a 45 minute timeout, the controller image from the bundle and output paths for the kubeconfig and the etcd encryption key. The log tells the story in order. The cluster is created and goes from Provisioning to Provisioned in about a minute and forty seconds, and then the control plane nodes join one at a time at roughly ten second intervals until all three are in. nvkectl writes the kubeconfig and the etcd encryption key and reminds the operator to move that key off the host, removes the control plane taint from all three nodes so they can run workloads, and tears down the temporary kind bootstrap cluster. From start to the ready message is a little over five minutes.
root@asus3-vm1:~/nvke-demo# nvkectl bootstrap site -f /root/nvke-demo/sitespec.yaml \
--disable-mtls --timeout 45m \
--nvke-controller-image nke-controller:lab-as65000 \
--kubeconfig-out /root/nvke-demo/nvke-demo.kubeconfig \
--encryption-key-out /root/nvke-demo/nvke-demo-etcd-encryption-key.json
Trace ID: 298f2d4b6bd715093ddab7614d2b5cf6
time=2026-11-05T21:23:38.212Z level=INFO msg="loaded site spec" component=bootstrap-site path=/root/nvke-demo/sitespec.yaml cluster=nvke-demo trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
Loaded site spec:
clusterName: nvke-demo
infra:
provider: provisioned
cluster:
nodes:
- name: asus3-vm1
ip: 192.168.0.116
- name: asus3-vm2
ip: 192.168.0.117
- name: asus3-vm3
ip: 192.168.0.118
k8sVersion: 1.36.3
cniName: ovn-kubernetes
cniVersion: 2.4.2
controlPlaneVIP: 10.250.0.10
podCidr: 10.244.0.0/16
serviceCidr: 10.96.0.0/12
nodeAgentPort: "15005"
etcdEncryption: true
kubeletConfig:
systemReservedMemory: 4Gi
spire:
trustDomain: nvke-demo.example.com
time=2026-11-05T21:23:38.274Z level=INFO msg="creating ephemeral bootstrap kind cluster" component=bootstrap-site cluster=nvkectl-bootstrap trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:23:38.291Z level=INFO msg="creating bootstrap kind cluster" component=bootstrap-site component=bootstrapengine cluster=nvkectl-bootstrap trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:24:46.752Z level=INFO msg="bootstrap kind cluster ready" component=bootstrap-site component=bootstrapengine cluster=nvkectl-bootstrap trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:24:46.780Z level=INFO msg="loading image into bootstrap kind cluster" component=bootstrap-site component=bootstrapengine cluster=nvkectl-bootstrap image=nke-controller:lab-as65000 trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:24:49.443Z level=INFO msg="loading image into bootstrap kind cluster" component=bootstrap-site component=bootstrapengine cluster=nvkectl-bootstrap image=cluster-api-provider-provisioned:latest trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:24:50.669Z level=INFO msg="deploying the controller onto bootstrap kind cluster" component=bootstrap-site cluster=nvkectl-bootstrap image=nke-controller:lab-as65000 trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:24:50.670Z level=INFO msg="installing postgres" component=bootstrap-site component=controllerdeploy namespace=postgres chart=/tmp/nvkectl-postgres-chart-263185207 trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:25:15.553Z level=INFO msg="installing temporal" component=bootstrap-site component=controllerdeploy namespace=temporal version=0.74.0 chart=temporal trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:26:07.548Z level=WARN msg="SPIFFE is disabled: skipping SPIRE, and the controller will dial node agents in plaintext" component=bootstrap-site component=controllerdeploy trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:26:07.589Z level=INFO msg="installing cluster api core and the kubeadm providers" component=bootstrap-site component=controllerdeploy version=v1.13.6 trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:26:39.856Z level=INFO msg="installing the provisioned infrastructure provider" component=bootstrap-site component=controllerdeploy namespace=provisioned-system image=cluster-api-provider-provisioned:latest trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:26:40.696Z level=INFO msg="installing the controller" component=bootstrap-site component=controllerdeploy namespace=nke-controller image=nke-controller:lab-as65000 chart=/tmp/nvkectl-controller-chart-1009388464 trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:26:52.751Z level=INFO msg="publishing the controller api for the host" component=bootstrap-site component=controllerdeploy namespace=nke-controller port=30080 trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:26:52.833Z level=INFO msg="waiting for the controller to report healthy" component=bootstrap-site timeout=5m0s trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:26:52.869Z level=INFO msg="controller reports healthy" component=bootstrap-site component=controllerdeploy checks="map[database:ok temporal:ok]" trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:26:53.870Z level=INFO msg="controller deployment healthy" component=bootstrap-site trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:26:53.870Z level=INFO msg="creating the site cluster" component=bootstrap-site cluster=nvke-demo kubernetesVersion=1.36.3 cni=ovn-kubernetes infraProvider=provisioned controlPlaneNodes=3 trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:26:53.884Z level=INFO msg="cluster created" component=bootstrap-site cluster=nvke-demo id=8f803855-e0e5-4d43-9ecd-770fbd43d517 trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:26:53.884Z level=INFO msg="waiting for the cluster to finish provisioning" component=bootstrap-site cluster=nvke-demo timeout=45m0s trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:26:53.886Z level=INFO msg="cluster state" component=bootstrap-site cluster=nvke-demo status=Provisioning trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:28:33.948Z level=INFO msg="cluster state" component=bootstrap-site cluster=nvke-demo status=Provisioned trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:28:33.948Z level=INFO msg="cluster provisioned" component=bootstrap-site cluster=nvke-demo id=8f803855-e0e5-4d43-9ecd-770fbd43d517 trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:28:33.948Z level=INFO msg="waiting for the control plane nodes to join" component=bootstrap-site cluster=8f803855-e0e5-4d43-9ecd-770fbd43d517 nodes=3 trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:28:33.950Z level=INFO msg="control plane nodes joined" component=bootstrap-site cluster=8f803855-e0e5-4d43-9ecd-770fbd43d517 provisioned=0 of=3 trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:28:43.961Z level=INFO msg="control plane nodes joined" component=bootstrap-site cluster=8f803855-e0e5-4d43-9ecd-770fbd43d517 provisioned=1 of=3 trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:28:53.964Z level=INFO msg="control plane nodes joined" component=bootstrap-site cluster=8f803855-e0e5-4d43-9ecd-770fbd43d517 provisioned=2 of=3 trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:29:03.975Z level=INFO msg="control plane nodes joined" component=bootstrap-site cluster=8f803855-e0e5-4d43-9ecd-770fbd43d517 provisioned=3 of=3 trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:29:03.975Z level=INFO msg="all control plane nodes joined" component=bootstrap-site cluster=nvke-demo nodes=3 trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:29:04.273Z level=INFO msg="wrote the cluster kubeconfig" component=bootstrap-site path=/root/nvke-demo/nvke-demo.kubeconfig trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:29:04.336Z level=INFO msg="wrote the etcd encryption key" component=bootstrap-site path=/root/nvke-demo/nvke-demo-etcd-encryption-key.json sha256=ff8eb01569081918145fa14e3fac58714152a7bf3a032dcb6d70e60561368d5a trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
The etcd encryption key of cluster "nvke-demo" is at /root/nvke-demo/nvke-demo-etcd-encryption-key.json. Move that file off this host and keep it with the cluster's other credentials.
time=2026-11-05T21:29:04.359Z level=INFO msg="removed the control-plane taint" component=bootstrap-site node=nvke-demo-cp-4c8bbf0f trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:29:04.377Z level=INFO msg="removed the control-plane taint" component=bootstrap-site node=nvke-demo-cp-90c9dcc2 trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:29:04.395Z level=INFO msg="removed the control-plane taint" component=bootstrap-site node=nvke-demo-cp-f46d7a88 trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:29:04.396Z level=INFO msg="control-plane nodes are schedulable" component=bootstrap-site untainted=3 trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
Cluster "nvke-demo" is ready. Talk to it with:
kubectl --kubeconfig /root/nvke-demo/nvke-demo.kubeconfig get nodes
time=2026-11-05T21:29:04.396Z level=INFO msg="tearing down bootstrap kind cluster" component=bootstrap-site cluster=nvkectl-bootstrap trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
time=2026-11-05T21:29:04.396Z level=INFO msg="deleting bootstrap kind cluster" component=bootstrap-site component=bootstrapengine cluster=nvkectl-bootstrap trace_id=298f2d4b6bd715093ddab7614d2b5cf6 span_id=ae36382a04c9ac45
Trace ID: 298f2d4b6bd715093ddab7614d2b5cf6
We can verify this by talking to the new cluster. After pointing KUBECONFIG at the file that was just written, kubectl get nodes shows three nodes, all Ready with the control-plane role at v1.36.3. The pod listing shows what NVKE actually deployed: CoreDNS, three each of etcd, kube-apiserver, kube-controller-manager and kube-scheduler, kube-vip for the virtual IP, node-local-dns and the OVN-Kubernetes control plane and per-node pods. Everything is Running, with one restart on one of the kube-vip pods, and the ages are all under nine minutes.
root@asus3-vm1:~/nvke-demo# export KUBECONFIG=/root/nvke-demo/nvke-demo.kubeconfig
root@asus3-vm1:~/nvke-demo# kubectl get nodes
NAME STATUS ROLES AGE VERSION
nvke-demo-cp-4c8bbf0f Ready control-plane 8m32s v1.36.3
nvke-demo-cp-90c9dcc2 Ready control-plane 7m14s v1.36.3
nvke-demo-cp-f46d7a88 Ready control-plane 7m3s v1.36.3
root@asus3-vm1:~/nvke-demo# kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-7846cffc78-w64jv 1/1 Running 0 8m25s
kube-system coredns-7846cffc78-wgz9g 1/1 Running 0 8m25s
kube-system etcd-nvke-demo-cp-4c8bbf0f 1/1 Running 0 8m27s
kube-system etcd-nvke-demo-cp-90c9dcc2 1/1 Running 0 7m16s
kube-system etcd-nvke-demo-cp-f46d7a88 1/1 Running 0 7m5s
kube-system kube-apiserver-nvke-demo-cp-4c8bbf0f 1/1 Running 0 8m27s
kube-system kube-apiserver-nvke-demo-cp-90c9dcc2 1/1 Running 0 7m16s
kube-system kube-apiserver-nvke-demo-cp-f46d7a88 1/1 Running 0 7m5s
kube-system kube-controller-manager-nvke-demo-cp-4c8bbf0f 1/1 Running 0 8m27s
kube-system kube-controller-manager-nvke-demo-cp-90c9dcc2 1/1 Running 0 7m16s
kube-system kube-controller-manager-nvke-demo-cp-f46d7a88 1/1 Running 0 7m5s
kube-system kube-scheduler-nvke-demo-cp-4c8bbf0f 1/1 Running 0 8m27s
kube-system kube-scheduler-nvke-demo-cp-90c9dcc2 1/1 Running 0 7m16s
kube-system kube-scheduler-nvke-demo-cp-f46d7a88 1/1 Running 0 7m5s
kube-system kube-vip-nvke-demo-cp-4c8bbf0f 1/1 Running 1 (8m28s ago) 8m36s
kube-system kube-vip-nvke-demo-cp-90c9dcc2 1/1 Running 0 7m12s
kube-system kube-vip-nvke-demo-cp-f46d7a88 1/1 Running 0 7m2s
kube-system node-local-dns-cdfq6 1/1 Running 0 7m18s
kube-system node-local-dns-h2h4p 1/1 Running 0 7m7s
kube-system node-local-dns-xwrdc 1/1 Running 0 8m25s
ovn-kubernetes ovnkube-control-plane-5f58f45986-5xcxg 1/1 Running 0 8m25s
ovn-kubernetes ovnkube-node-7hx2t 5/5 Running 0 8m25s
ovn-kubernetes ovnkube-node-8lsdn 5/5 Running 0 7m7s
ovn-kubernetes ovnkube-node-fzwrm 5/5 Running 0 7m18s
Loading and Deploying NVSentinel
With the cluster up, the NVSentinel images that were baked into the disk can be loaded. This block first verifies the add-on bundle against its manifest again, then imports each Docker archive into the k8s.io containerd namespace with the linux/amd64 platform and lists what containerd now holds. The long run of OK lines is the manifest check, and the image listing that follows includes the two NVSentinel images alongside the cluster's own images. A second loop uses crictl inspecti to prove the Kubernetes runtime can see both references, and it prints READY for each. I ran the import on node one in this capture, and the pod events further down show the image already present on all three nodes.
root@asus3-vm1:~/nvke-demo# (
set -euo pipefail
cd /opt/nvke-addons/nvsentinel
sha256sum -c SHA256SUMS
for archive in images/*.tar; do
test -f "$archive"
ctr -n k8s.io images import --platform linux/amd64 "$archive"
done
ctr -n k8s.io images list
)
./image-map.tsv: OK
./images-required.txt: OK
./images/001.tar: OK
./images/002.tar: OK
./nvkr-commit.txt: OK
./nvsentinel-source-commit.txt: OK
./nvsentinel/.helmignore: OK
./nvsentinel/Chart.lock: OK
./nvsentinel/Chart.yaml: OK
./nvsentinel/charts/csp-health-monitor/.helmignore: OK
./nvsentinel/charts/csp-health-monitor/Chart.yaml: OK
./nvsentinel/charts/csp-health-monitor/templates/_helpers.tpl: OK
./nvsentinel/charts/csp-health-monitor/templates/clusterrole.yaml: OK
./nvsentinel/charts/csp-health-monitor/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/csp-health-monitor/templates/configmap.yaml: OK
./nvsentinel/charts/csp-health-monitor/templates/deployment.yaml: OK
./nvsentinel/charts/csp-health-monitor/templates/serviceaccount.yaml: OK
./nvsentinel/charts/csp-health-monitor/values.yaml: OK
./nvsentinel/charts/event-exporter/Chart.yaml: OK
./nvsentinel/charts/event-exporter/templates/_helpers.tpl: OK
./nvsentinel/charts/event-exporter/templates/configmap.yaml: OK
./nvsentinel/charts/event-exporter/templates/deployment.yaml: OK
./nvsentinel/charts/event-exporter/values.yaml: OK
./nvsentinel/charts/fault-quarantine/.helmignore: OK
./nvsentinel/charts/fault-quarantine/Chart.yaml: OK
./nvsentinel/charts/fault-quarantine/templates/_helpers.tpl: OK
./nvsentinel/charts/fault-quarantine/templates/clusterrole.yaml: OK
./nvsentinel/charts/fault-quarantine/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/fault-quarantine/templates/configmap.yaml: OK
./nvsentinel/charts/fault-quarantine/templates/deployment.yaml: OK
./nvsentinel/charts/fault-quarantine/templates/serviceaccount.yaml: OK
./nvsentinel/charts/fault-quarantine/values.yaml: OK
./nvsentinel/charts/fault-remediation/.helmignore: OK
./nvsentinel/charts/fault-remediation/Chart.yaml: OK
./nvsentinel/charts/fault-remediation/files/log-collector-job.yaml: OK
./nvsentinel/charts/fault-remediation/templates/_helpers.tpl: OK
./nvsentinel/charts/fault-remediation/templates/clusterrole.yaml: OK
./nvsentinel/charts/fault-remediation/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/fault-remediation/templates/configmap.yaml: OK
./nvsentinel/charts/fault-remediation/templates/deployment.yaml: OK
./nvsentinel/charts/fault-remediation/templates/log-collector-rbac.yaml: OK
./nvsentinel/charts/fault-remediation/templates/serviceaccount.yaml: OK
./nvsentinel/charts/fault-remediation/values.yaml: OK
./nvsentinel/charts/gpu-health-monitor/.helmignore: OK
./nvsentinel/charts/gpu-health-monitor/Chart.yaml: OK
./nvsentinel/charts/gpu-health-monitor/files/dcgmerrorsmapping.csv: OK
./nvsentinel/charts/gpu-health-monitor/templates/NOTES.txt: OK
./nvsentinel/charts/gpu-health-monitor/templates/_helpers.tpl: OK
./nvsentinel/charts/gpu-health-monitor/templates/configmap.yaml: OK
./nvsentinel/charts/gpu-health-monitor/templates/daemonset-dcgm-3.x.yaml: OK
./nvsentinel/charts/gpu-health-monitor/templates/daemonset-dcgm-4.x.yaml: OK
./nvsentinel/charts/gpu-health-monitor/tests/dcgm_runtime_debounce_test.yaml: OK
./nvsentinel/charts/gpu-health-monitor/tests/dcgm_startup_gate_test.yaml: OK
./nvsentinel/charts/gpu-health-monitor/tests/gpu_deploy_client_affinity_test.yaml: OK
./nvsentinel/charts/gpu-health-monitor/tests/pc_auth_strictness_test.yaml: OK
./nvsentinel/charts/gpu-health-monitor/tests/priorityclassname_test.yaml: OK
./nvsentinel/charts/gpu-health-monitor/tests/publish_wiring_test.yaml: OK
./nvsentinel/charts/gpu-health-monitor/values.yaml: OK
./nvsentinel/charts/health-events-analyzer/.helmignore: OK
./nvsentinel/charts/health-events-analyzer/Chart.yaml: OK
./nvsentinel/charts/health-events-analyzer/templates/_helpers.tpl: OK
./nvsentinel/charts/health-events-analyzer/templates/configmap.yaml: OK
./nvsentinel/charts/health-events-analyzer/templates/deployment.yaml: OK
./nvsentinel/charts/health-events-analyzer/templates/node-recovery-rbac.yaml: OK
./nvsentinel/charts/health-events-analyzer/templates/role.yaml: OK
./nvsentinel/charts/health-events-analyzer/templates/rolebinding.yaml: OK
./nvsentinel/charts/health-events-analyzer/templates/serviceaccount.yaml: OK
./nvsentinel/charts/health-events-analyzer/tests/node_recovery_rbac_test.yaml: OK
./nvsentinel/charts/health-events-analyzer/values.yaml: OK
./nvsentinel/charts/incluster-file-server/Chart.yaml: OK
./nvsentinel/charts/incluster-file-server/templates/_helpers.tpl: OK
./nvsentinel/charts/incluster-file-server/templates/configmap-nginx.yaml: OK
./nvsentinel/charts/incluster-file-server/templates/deployment.yaml: OK
./nvsentinel/charts/incluster-file-server/templates/service.yaml: OK
./nvsentinel/charts/incluster-file-server/values.yaml: OK
./nvsentinel/charts/janitor-provider/Chart.yaml: OK
./nvsentinel/charts/janitor-provider/templates/_helpers.tpl: OK
./nvsentinel/charts/janitor-provider/templates/certificate.yaml: OK
./nvsentinel/charts/janitor-provider/templates/clusterrole.yaml: OK
./nvsentinel/charts/janitor-provider/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/janitor-provider/templates/deployment.yaml: OK
./nvsentinel/charts/janitor-provider/templates/networkpolicy.yaml: OK
./nvsentinel/charts/janitor-provider/templates/role.yaml: OK
./nvsentinel/charts/janitor-provider/templates/rolebinding.yaml: OK
./nvsentinel/charts/janitor-provider/templates/service.yaml: OK
./nvsentinel/charts/janitor-provider/templates/serviceaccount.yaml: OK
./nvsentinel/charts/janitor-provider/values.yaml: OK
./nvsentinel/charts/janitor/.helmignore: OK
./nvsentinel/charts/janitor/Chart.yaml: OK
./nvsentinel/charts/janitor/crds/external_remediation.crd.yaml: OK
./nvsentinel/charts/janitor/crds/janitor.dgxc.nvidia.com_gpuresets.yaml: OK
./nvsentinel/charts/janitor/crds/janitor.dgxc.nvidia.com_rebootnodes.yaml: OK
./nvsentinel/charts/janitor/crds/janitor.dgxc.nvidia.com_terminatenodes.yaml: OK
./nvsentinel/charts/janitor/templates/_helpers.tpl: OK
./nvsentinel/charts/janitor/templates/certificate-metrics.yaml: OK
./nvsentinel/charts/janitor/templates/clusterrole.yaml: OK
./nvsentinel/charts/janitor/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/janitor/templates/configmap.yaml: OK
./nvsentinel/charts/janitor/templates/deployment.yaml: OK
./nvsentinel/charts/janitor/templates/issuer.yaml: OK
./nvsentinel/charts/janitor/templates/networkpolicy.yaml: OK
./nvsentinel/charts/janitor/templates/service.yaml: OK
./nvsentinel/charts/janitor/templates/serviceaccount.yaml: OK
./nvsentinel/charts/janitor/templates/webhook.yaml: OK
./nvsentinel/charts/janitor/values.schema.json: OK
./nvsentinel/charts/janitor/values.yaml: OK
./nvsentinel/charts/k8s-datastore/Chart.yaml: OK
./nvsentinel/charts/k8s-datastore/crds/health_event.crd.yaml: OK
./nvsentinel/charts/k8s-datastore/values.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/Chart.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/templates/_helpers.tpl: OK
./nvsentinel/charts/kubernetes-object-monitor/templates/clusterrole.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/templates/configmap.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/templates/deployment.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/templates/serviceaccount.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/tests/pc_auth_strictness_test.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/tests/publish_wiring_test.yaml: OK
./nvsentinel/charts/kubernetes-object-monitor/values.yaml: OK
./nvsentinel/charts/labeler/.helmignore: OK
./nvsentinel/charts/labeler/Chart.yaml: OK
./nvsentinel/charts/labeler/templates/_helpers.tpl: OK
./nvsentinel/charts/labeler/templates/clusterrole.yaml: OK
./nvsentinel/charts/labeler/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/labeler/templates/configmap.yaml: OK
./nvsentinel/charts/labeler/templates/deployment.yaml: OK
./nvsentinel/charts/labeler/templates/serviceaccount.yaml: OK
./nvsentinel/charts/labeler/tests/priorityclassname_test.yaml: OK
./nvsentinel/charts/labeler/values.yaml: OK
./nvsentinel/charts/lifecycle-manager/.helmignore: OK
./nvsentinel/charts/lifecycle-manager/Chart.yaml: OK
./nvsentinel/charts/lifecycle-manager/crds/maintenance_request.crd.yaml: OK
./nvsentinel/charts/lifecycle-manager/crds/nvsentinel.nvidia.com_validationrequests.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/_helpers.tpl: OK
./nvsentinel/charts/lifecycle-manager/templates/certificate-metrics.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/clusterrole.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/configmap-templates.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/configmap.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/deployment.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/issuer.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/networkpolicy.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/service.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/serviceaccount.yaml: OK
./nvsentinel/charts/lifecycle-manager/templates/webhook.yaml: OK
./nvsentinel/charts/lifecycle-manager/tests/deployment_test.yaml: OK
./nvsentinel/charts/lifecycle-manager/tests/pc_auth_strictness_test.yaml: OK
./nvsentinel/charts/lifecycle-manager/tests/publish_wiring_test.yaml: OK
./nvsentinel/charts/lifecycle-manager/tests/rbac_test.yaml: OK
./nvsentinel/charts/lifecycle-manager/tests/webhook_test.yaml: OK
./nvsentinel/charts/lifecycle-manager/values.yaml: OK
./nvsentinel/charts/metadata-collector/Chart.yaml: OK
./nvsentinel/charts/metadata-collector/templates/_helpers.tpl: OK
./nvsentinel/charts/metadata-collector/templates/clusterrole.yaml: OK
./nvsentinel/charts/metadata-collector/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/metadata-collector/templates/daemonset.yaml: OK
./nvsentinel/charts/metadata-collector/templates/resourceclaimtemplate.yaml: OK
./nvsentinel/charts/metadata-collector/templates/serviceaccount.yaml: OK
./nvsentinel/charts/metadata-collector/tests/daemonset_test.yaml: OK
./nvsentinel/charts/metadata-collector/tests/gpu_deploy_client_affinity_test.yaml: OK
./nvsentinel/charts/metadata-collector/tests/resourceclaimtemplate_test.yaml: OK
./nvsentinel/charts/metadata-collector/values.yaml: OK
./nvsentinel/charts/mongodb-store/.helmignore: OK
./nvsentinel/charts/mongodb-store/Chart.lock: OK
./nvsentinel/charts/mongodb-store/Chart.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/.helmignore: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/Chart.lock: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/Chart.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/README.md: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/.helmignore: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/Chart.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/README.md: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_affinities.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_capabilities.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_compatibility.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_errors.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_images.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_ingress.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_labels.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_names.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_resources.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_secrets.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_storage.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_tplvalues.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_utils.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/_warnings.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/validations/_cassandra.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/validations/_mariadb.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/validations/_mongodb.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/validations/_mysql.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/validations/_postgresql.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/validations/_redis.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/templates/validations/_validations.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/charts/common/values.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/NOTES.txt: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/_helpers.tpl: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/arbiter/configmap.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/arbiter/headless-svc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/arbiter/pdb.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/arbiter/statefulset.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/backup/cronjob.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/backup/pvc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/common-scripts-cm.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/configmap.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/extra-list.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/hidden/configmap.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/hidden/external-access-svc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/hidden/headless-svc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/hidden/pdb.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/hidden/statefulset.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/initialization-configmap.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/metrics-svc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/networkpolicy.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/prometheusrule.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/psp.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/replicaset/external-access-svc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/replicaset/headless-svc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/replicaset/pdb.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/replicaset/scripts-configmap.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/replicaset/statefulset.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/replicaset/svc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/role.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/rolebinding.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/secrets-ca.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/secrets.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/serviceaccount.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/servicemonitor.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/standalone/dep-sts.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/standalone/pdb.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/standalone/pvc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/templates/standalone/svc.yaml: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/values.schema.json: OK
./nvsentinel/charts/mongodb-store/charts/mongodb/values.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/.helmignore: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/Chart.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/README.md: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/templates/NOTES.txt: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/templates/_helpers.tpl: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/templates/cluster-secret.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/templates/cluster.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/templates/clustersync-secret.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/templates/clustersync.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-db/values.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/.helmignore: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/Chart.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/LICENSE.txt: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/README.md: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/crds/crd.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/templates/NOTES.txt: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/templates/_helpers.tpl: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/templates/deployment.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/templates/namespace.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/templates/role-binding.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/templates/role.yaml: OK
./nvsentinel/charts/mongodb-store/charts/psmdb-operator/values.yaml: OK
./nvsentinel/charts/mongodb-store/templates/_helpers.tpl: OK
./nvsentinel/charts/mongodb-store/templates/_init-eval.tpl: OK
./nvsentinel/charts/mongodb-store/templates/certmanager.yaml: OK
./nvsentinel/charts/mongodb-store/templates/configmap.yaml: OK
./nvsentinel/charts/mongodb-store/templates/jobs.yaml: OK
./nvsentinel/charts/mongodb-store/templates/metrics-service.yaml: OK
./nvsentinel/charts/mongodb-store/templates/networkpolicy.yaml: OK
./nvsentinel/charts/mongodb-store/templates/pv.yaml: OK
./nvsentinel/charts/mongodb-store/templates/role.yaml: OK
./nvsentinel/charts/mongodb-store/templates/rolebinding.yaml: OK
./nvsentinel/charts/mongodb-store/tests/cluster_domain_test.yaml: OK
./nvsentinel/charts/mongodb-store/tests/networkpolicy_test.yaml: OK
./nvsentinel/charts/mongodb-store/tests/psmdb_versions_test.yaml: OK
./nvsentinel/charts/mongodb-store/values.yaml: OK
./nvsentinel/charts/nic-health-monitor/.helmignore: OK
./nvsentinel/charts/nic-health-monitor/Chart.yaml: OK
./nvsentinel/charts/nic-health-monitor/templates/_helpers.tpl: OK
./nvsentinel/charts/nic-health-monitor/templates/configmap.yaml: OK
./nvsentinel/charts/nic-health-monitor/templates/daemonset.yaml: OK
./nvsentinel/charts/nic-health-monitor/tests/daemonset_test.yaml: OK
./nvsentinel/charts/nic-health-monitor/tests/pc_auth_strictness_test.yaml: OK
./nvsentinel/charts/nic-health-monitor/tests/publish_wiring_test.yaml: OK
./nvsentinel/charts/nic-health-monitor/values.yaml: OK
./nvsentinel/charts/node-drainer/.helmignore: OK
./nvsentinel/charts/node-drainer/Chart.yaml: OK
./nvsentinel/charts/node-drainer/templates/_helpers.tpl: OK
./nvsentinel/charts/node-drainer/templates/clusterrole.yaml: OK
./nvsentinel/charts/node-drainer/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/node-drainer/templates/configmap.yaml: OK
./nvsentinel/charts/node-drainer/templates/deployment.yaml: OK
./nvsentinel/charts/node-drainer/templates/serviceaccount.yaml: OK
./nvsentinel/charts/node-drainer/tests/pod_drain_policies_test.yaml: OK
./nvsentinel/charts/node-drainer/values.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/Chart.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/templates/_helpers.tpl: OK
./nvsentinel/charts/nvcre-certification-monitor/templates/clusterrole.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/templates/configmap.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/templates/deployment.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/templates/serviceaccount.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/tests/pc_auth_strictness_test.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/tests/publish_wiring_test.yaml: OK
./nvsentinel/charts/nvcre-certification-monitor/values.yaml: OK
./nvsentinel/charts/postgresql/.helmignore: OK
./nvsentinel/charts/postgresql/Chart.lock: OK
./nvsentinel/charts/postgresql/Chart.yaml: OK
./nvsentinel/charts/postgresql/README.md: OK
./nvsentinel/charts/postgresql/charts/common/.helmignore: OK
./nvsentinel/charts/postgresql/charts/common/Chart.yaml: OK
./nvsentinel/charts/postgresql/charts/common/README.md: OK
./nvsentinel/charts/postgresql/charts/common/templates/_affinities.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_capabilities.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_compatibility.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_errors.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_images.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_ingress.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_labels.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_names.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_resources.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_secrets.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_storage.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_tplvalues.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_utils.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/_warnings.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/validations/_cassandra.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/validations/_mariadb.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/validations/_mongodb.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/validations/_mysql.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/validations/_postgresql.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/validations/_redis.tpl: OK
./nvsentinel/charts/postgresql/charts/common/templates/validations/_validations.tpl: OK
./nvsentinel/charts/postgresql/charts/common/values.yaml: OK
./nvsentinel/charts/postgresql/templates/NOTES.txt: OK
./nvsentinel/charts/postgresql/templates/_helpers.tpl: OK
./nvsentinel/charts/postgresql/templates/backup/cronjob.yaml: OK
./nvsentinel/charts/postgresql/templates/backup/networkpolicy.yaml: OK
./nvsentinel/charts/postgresql/templates/backup/pvc.yaml: OK
./nvsentinel/charts/postgresql/templates/extra-list.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/configmap.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/extended-configmap.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/initialization-configmap.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/metrics-configmap.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/metrics-svc.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/networkpolicy.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/pdb.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/preinitialization-configmap.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/servicemonitor.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/statefulset.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/svc-headless.yaml: OK
./nvsentinel/charts/postgresql/templates/primary/svc.yaml: OK
./nvsentinel/charts/postgresql/templates/prometheusrule.yaml: OK
./nvsentinel/charts/postgresql/templates/psp.yaml: OK
./nvsentinel/charts/postgresql/templates/read/extended-configmap.yaml: OK
./nvsentinel/charts/postgresql/templates/read/metrics-configmap.yaml: OK
./nvsentinel/charts/postgresql/templates/read/metrics-svc.yaml: OK
./nvsentinel/charts/postgresql/templates/read/networkpolicy.yaml: OK
./nvsentinel/charts/postgresql/templates/read/pdb.yaml: OK
./nvsentinel/charts/postgresql/templates/read/servicemonitor.yaml: OK
./nvsentinel/charts/postgresql/templates/read/statefulset.yaml: OK
./nvsentinel/charts/postgresql/templates/read/svc-headless.yaml: OK
./nvsentinel/charts/postgresql/templates/read/svc.yaml: OK
./nvsentinel/charts/postgresql/templates/role.yaml: OK
./nvsentinel/charts/postgresql/templates/rolebinding.yaml: OK
./nvsentinel/charts/postgresql/templates/secrets.yaml: OK
./nvsentinel/charts/postgresql/templates/serviceaccount.yaml: OK
./nvsentinel/charts/postgresql/templates/tls-secrets.yaml: OK
./nvsentinel/charts/postgresql/values.schema.json: OK
./nvsentinel/charts/postgresql/values.yaml: OK
./nvsentinel/charts/preflight/.helmignore: OK
./nvsentinel/charts/preflight/Chart.yaml: OK
./nvsentinel/charts/preflight/crds/preflight.nvsentinel.nvidia.com_preflightconfigs.yaml: OK
./nvsentinel/charts/preflight/files/ndv4-topo.xml: OK
./nvsentinel/charts/preflight/files/ndv5-topo.xml: OK
./nvsentinel/charts/preflight/templates/_helpers.tpl: OK
./nvsentinel/charts/preflight/templates/ca-certificate.yaml: OK
./nvsentinel/charts/preflight/templates/ca-issuer.yaml: OK
./nvsentinel/charts/preflight/templates/certificate.yaml: OK
./nvsentinel/charts/preflight/templates/clusterrole.yaml: OK
./nvsentinel/charts/preflight/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/preflight/templates/configmap.yaml: OK
./nvsentinel/charts/preflight/templates/deployment.yaml: OK
./nvsentinel/charts/preflight/templates/image-cache-daemonset.yaml: OK
./nvsentinel/charts/preflight/templates/mutatingwebhook.yaml: OK
./nvsentinel/charts/preflight/templates/networkpolicy.yaml: OK
./nvsentinel/charts/preflight/templates/selfsigned-issuer.yaml: OK
./nvsentinel/charts/preflight/templates/service.yaml: OK
./nvsentinel/charts/preflight/templates/serviceaccount.yaml: OK
./nvsentinel/charts/preflight/tests/dra_rbac_test.yaml: OK
./nvsentinel/charts/preflight/tests/mutatingwebhook_test.yaml: OK
./nvsentinel/charts/preflight/tests/pc_auth_strictness_test.yaml: OK
./nvsentinel/charts/preflight/tests/publish_wiring_test.yaml: OK
./nvsentinel/charts/preflight/values.schema.json: OK
./nvsentinel/charts/preflight/values.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/Chart.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/templates/_helpers.tpl: OK
./nvsentinel/charts/slurm-drain-monitor/templates/clusterrole.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/templates/clusterrolebinding.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/templates/configmap.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/templates/deployment.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/templates/serviceaccount.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/tests/pc_auth_strictness_test.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/tests/publish_wiring_test.yaml: OK
./nvsentinel/charts/slurm-drain-monitor/values.yaml: OK
./nvsentinel/charts/syslog-health-monitor/.helmignore: OK
./nvsentinel/charts/syslog-health-monitor/Chart.yaml: OK
./nvsentinel/charts/syslog-health-monitor/templates/_helpers.tpl: OK
./nvsentinel/charts/syslog-health-monitor/templates/configmap.yaml: OK
./nvsentinel/charts/syslog-health-monitor/templates/daemonset-kata.yaml: OK
./nvsentinel/charts/syslog-health-monitor/templates/daemonset-regular.yaml: OK
./nvsentinel/charts/syslog-health-monitor/tests/pc_auth_strictness_test.yaml: OK
./nvsentinel/charts/syslog-health-monitor/tests/publish_wiring_test.yaml: OK
./nvsentinel/charts/syslog-health-monitor/values.yaml: OK
./nvsentinel/files/node-condition-cleanup.sh: OK
./nvsentinel/policies/README.md: OK
./nvsentinel/policies/must-have-sbom.yaml: OK
./nvsentinel/policies/must-have-slsa.yaml: OK
./nvsentinel/templates/NOTES.txt: OK
./nvsentinel/templates/_external-mongo-init-eval.tpl: OK
./nvsentinel/templates/_helpers.tpl: OK
./nvsentinel/templates/_platform-connector-config.tpl: OK
./nvsentinel/templates/certmanager-postgresql.yaml: OK
./nvsentinel/templates/clusterrole.yaml: OK
./nvsentinel/templates/clusterrolebinding.yaml: OK
./nvsentinel/templates/configmap-datastore.yaml: OK
./nvsentinel/templates/configmap.yaml: OK
./nvsentinel/templates/daemonset.yaml: OK
./nvsentinel/templates/job-external-mongodb-setup.yaml: OK
./nvsentinel/templates/networkpolicy.yaml: OK
./nvsentinel/templates/node-condition-cleanup-hook.yaml: OK
./nvsentinel/templates/platform-connector-deployment-certificate.yaml: OK
./nvsentinel/templates/platform-connector-deployment-configmap.yaml: OK
./nvsentinel/templates/platform-connector-deployment-deployment.yaml: OK
./nvsentinel/templates/platform-connector-deployment-networkpolicy.yaml: OK
./nvsentinel/templates/platform-connector-deployment-pdb.yaml: OK
./nvsentinel/templates/platform-connector-deployment-service.yaml: OK
./nvsentinel/templates/platform-connector-deployment-serviceaccount.yaml: OK
./nvsentinel/templates/platform-connector-deployment-validation.yaml: OK
./nvsentinel/templates/podMonitor.yaml: OK
./nvsentinel/templates/serviceaccount.yaml: OK
./nvsentinel/tests/cluster_domain_test.yaml: OK
./nvsentinel/tests/csp_health_monitor_publish_wiring_test.yaml: OK
./nvsentinel/tests/datastore_credentials_test.yaml: OK
./nvsentinel/tests/deployed_crversion_test.yaml: OK
./nvsentinel/tests/external_mongodb_setup_job_test.yaml: OK
./nvsentinel/tests/health_events_analyzer_publish_wiring_test.yaml: OK
./nvsentinel/tests/pc_auth_cross_node_test.yaml: OK
./nvsentinel/tests/platform_connector_deployment_networkpolicy_test.yaml: OK
./nvsentinel/tests/platform_connector_deployment_server_test.yaml: OK
./nvsentinel/tests/platform_connector_deployment_test.yaml: OK
./nvsentinel/tests/platform_connector_retry_test.yaml: OK
./nvsentinel/tests/strict_render_test.yaml: OK
./nvsentinel/tests/values/publishers-on-deployment.yaml: OK
./nvsentinel/values-aicr-aws-uat.yaml: OK
./nvsentinel/values-aicr-gcp-uat.yaml: OK
./nvsentinel/values-aicr-uat.yaml: OK
./nvsentinel/values-atlas-gcp.yaml: OK
./nvsentinel/values-aws-docdb.yaml: OK
./nvsentinel/values-cosmosdb-test.yaml: OK
./nvsentinel/values-full.yaml: OK
./nvsentinel/values-npd-gpu-services.yaml: OK
./nvsentinel/values-npd-remediation.yaml: OK
./nvsentinel/values-nvcre.yaml: OK
./nvsentinel/values-postgresql.yaml: OK
./nvsentinel/values-preflight-kube.yaml: OK
./nvsentinel/values-remediation.yaml: OK
./nvsentinel/values-tilt-arm64.yaml: OK
./nvsentinel/values-tilt-deployment.yaml: OK
./nvsentinel/values-tilt-mongodb-percona.yaml: OK
./nvsentinel/values-tilt-mongodb-tls-disabled-percona.yaml: OK
./nvsentinel/values-tilt-mongodb-tls-disabled.yaml: OK
./nvsentinel/values-tilt-mongodb.yaml: OK
./nvsentinel/values-tilt-nvcre.yaml: OK
./nvsentinel/values-tilt-postgresql.yaml: OK
./nvsentinel/values-tilt-socket.yaml: OK
./nvsentinel/values-tilt.yaml: OK
./nvsentinel/values.yaml: OK
./values-lab.yaml: OK
WARN[0000] DEPRECATION: The `bin_dir` property of `[plugins."io.containerd.cri.v1.runtime".cni`] is deprecated since containerd v2.1 and will be removed in containerd v2.4. Use `bin_dirs` in the same section instead.
ghcr.io/nvidia/nvsentinel/platform conne saved
application/vnd.docker.distribution.manifest.v2+json sha256:c48b38e9fa3b19ea82d94faf705845b788cb23ed46f23b74059fff538495447f
Importing elapsed: 1.1 s total: 0.0 B (0.0 B/s)
WARN[0000] DEPRECATION: The `bin_dir` property of `[plugins."io.containerd.cri.v1.runtime".cni`] is deprecated since containerd v2.1 and will be removed in containerd v2.4. Use `bin_dirs` in the same section instead.
ghcr.io/nvidia/nvsentinel/syslog health saved
application/vnd.docker.distribution.manifest.v2+json sha256:e9dd5accd5f7f40d563a8f021999afc8363aafd5afe8486d7ed58e0e8a18dfa6
Importing elapsed: 1.9 s total: 0.0 B (0.0 B/s)
WARN[0000] DEPRECATION: The `bin_dir` property of `[plugins."io.containerd.cri.v1.runtime".cni`] is deprecated since containerd v2.1 and will be removed in containerd v2.4. Use `bin_dirs` in the same section instead.
REF TYPE DIGEST SIZE PLATFORMS LABELS
ghcr.io/kube-vip/kube-vip@sha256:fe8c7b6641ba7633a12b337821e9e83a6456e8edc3434f97577058edd4eecaf6 application/vnd.oci.image.index.v1+json sha256:fe8c7b6641ba7633a12b337821e9e83a6456e8edc3434f97577058edd4eecaf6 17.9 MiB linux/amd64,linux/arm/v7,linux/arm64,linux/ppc64le,linux/s390x io.cri-containerd.image=managed
ghcr.io/nvidia/nvsentinel/platform-connectors:lab-c15934773572 application/vnd.docker.distribution.manifest.v2+json sha256:c48b38e9fa3b19ea82d94faf705845b788cb23ed46f23b74059fff538495447f 62.5 MiB linux/amd64 io.cri-containerd.image=managed
ghcr.io/nvidia/nvsentinel/syslog-health-monitor:lab-c15934773572 application/vnd.docker.distribution.manifest.v2+json sha256:e9dd5accd5f7f40d563a8f021999afc8363aafd5afe8486d7ed58e0e8a18dfa6 118.9 MiB linux/amd64 io.cri-containerd.image=managed
localhost/nvke/ovn-kubernetes:git-8af51273d52dfa07454435d18d37230e582d797a application/vnd.oci.image.manifest.v1+json sha256:0d1244eeed5ec8639e4095b7c50d1fe23e6bf1700ecc550c29d799a4a26e6fc3 317.7 MiB linux/amd64 io.cri-containerd.image=managed
registry.k8s.io/coredns/coredns:v1.14.2 application/vnd.docker.distribution.manifest.v2+json sha256:bded7658b663044e940935c2abbfcf77dbf5bc902a4364e128c0305f29652dfd 22.2 MiB linux/amd64 io.cri-containerd.image=managed
registry.k8s.io/dns/k8s-dns-node-cache@sha256:bc6e64e2c85956af2fcc0aa720086410d41b4f31f378c9a92646fecc85cd4739 application/vnd.docker.distribution.manifest.list.v2+json sha256:bc6e64e2c85956af2fcc0aa720086410d41b4f31f378c9a92646fecc85cd4739 42.2 MiB linux/amd64,linux/arm/v7,linux/arm64,linux/ppc64le,linux/s390x io.cri-containerd.image=managed
registry.k8s.io/etcd:3.6.8-0 application/vnd.docker.distribution.manifest.v2+json sha256:d4b470fa1577c6ba2dfe16f1fbe87ae154783b64dcf2a750b9b0f5d3a2201ad2 21.8 MiB linux/amd64 io.cri-containerd.image=managed
registry.k8s.io/kube-apiserver:v1.36.3 application/vnd.docker.distribution.manifest.v2+json sha256:f475d2fd954dd884463d0a3258f7a74e1444dfeea685f77a2f5d48563c0cc870 28.4 MiB linux/amd64 io.cri-containerd.image=managed
registry.k8s.io/kube-controller-manager:v1.36.3 application/vnd.docker.distribution.manifest.v2+json sha256:78f8c779920a27b2071e33bf644e437ee42aef978f30b3fdc2de20308f713604 25.2 MiB linux/amd64 io.cri-containerd.image=managed
registry.k8s.io/kube-proxy:v1.36.3 application/vnd.docker.distribution.manifest.v2+json sha256:73bb34f89e82d790d848456775922f277bf06ad04517929139bfda01cd92bbdd 28.7 MiB linux/amd64 io.cri-containerd.image=managed
registry.k8s.io/kube-scheduler:v1.36.3 application/vnd.docker.distribution.manifest.v2+json sha256:14091b8f4828a64d1c373445c5852ef9452a35491bff5bafccac5f9be50538bc 19.5 MiB linux/amd64 io.cri-containerd.image=managed
registry.k8s.io/pause:3.10.2 application/vnd.docker.distribution.manifest.v2+json sha256:4e4602052e509e13a45d54cdd30bded6aba1750fead3b7e1b058071eda492a3c 310.5 KiB linux/amd64 io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned
sha256:09fc48c2942242115fc30f17864b5d474dcb6ece5c4ddcd7c82fd149eff29d3f application/vnd.docker.distribution.manifest.v2+json sha256:14091b8f4828a64d1c373445c5852ef9452a35491bff5bafccac5f9be50538bc 19.5 MiB linux/amd64 io.cri-containerd.image=managed
sha256:1a0b53262a8e29b178f3044c7793896fa18d946b44d23e609790ce1a367561f1 application/vnd.docker.distribution.manifest.v2+json sha256:e9dd5accd5f7f40d563a8f021999afc8363aafd5afe8486d7ed58e0e8a18dfa6 118.9 MiB linux/amd64 io.cri-containerd.image=managed
sha256:27f31ee27f0b7db984230de6252539045a8faed7594ff628900e132fbce9c3cd application/vnd.docker.distribution.manifest.v2+json sha256:c48b38e9fa3b19ea82d94faf705845b788cb23ed46f23b74059fff538495447f 62.5 MiB linux/amd64 io.cri-containerd.image=managed
sha256:2d72fa4089eb1c47638790047b1f5a5224261449c031a290a79f9f9c08a0340b application/vnd.docker.distribution.manifest.list.v2+json sha256:bc6e64e2c85956af2fcc0aa720086410d41b4f31f378c9a92646fecc85cd4739 42.2 MiB linux/amd64,linux/arm/v7,linux/arm64,linux/ppc64le,linux/s390x io.cri-containerd.image=managed
sha256:3400718b4dd57aee37c39a31af7e9ebb3d8abfd1c04b174d5ee68c6730234dda application/vnd.docker.distribution.manifest.v2+json sha256:f475d2fd954dd884463d0a3258f7a74e1444dfeea685f77a2f5d48563c0cc870 28.4 MiB linux/amd64 io.cri-containerd.image=managed
sha256:38667dd9be96c17227adde456e33ad701bdbdc5c6db0272a1d33a929bd8e62d6 application/vnd.docker.distribution.manifest.v2+json sha256:bded7658b663044e940935c2abbfcf77dbf5bc902a4364e128c0305f29652dfd 22.2 MiB linux/amd64 io.cri-containerd.image=managed
sha256:4a83b15d3ecfe0d916b2d0a7991bc2854a629b8097017c2ee1ff65b30ae4c07c application/vnd.docker.distribution.manifest.v2+json sha256:4e4602052e509e13a45d54cdd30bded6aba1750fead3b7e1b058071eda492a3c 310.5 KiB linux/amd64 io.cri-containerd.image=managed,io.cri-containerd.pinned=pinned
sha256:82c653b3047dbc89f2467501a8f754dff6346df68ae0e86768bbc39b628abd0d application/vnd.docker.distribution.manifest.v2+json sha256:78f8c779920a27b2071e33bf644e437ee42aef978f30b3fdc2de20308f713604 25.2 MiB linux/amd64 io.cri-containerd.image=managed
sha256:be76bac21961494f5c0509344cbf098e3e326af64ef9e4ca89cfef3d79496938 application/vnd.docker.distribution.manifest.v2+json sha256:73bb34f89e82d790d848456775922f277bf06ad04517929139bfda01cd92bbdd 28.7 MiB linux/amd64 io.cri-containerd.image=managed
sha256:e759996fde0ac03acc2fa50fdbfc2da3ceec2747fbc0a34982e21cbefe640f9b application/vnd.oci.image.index.v1+json sha256:fe8c7b6641ba7633a12b337821e9e83a6456e8edc3434f97577058edd4eecaf6 17.9 MiB linux/amd64,linux/arm/v7,linux/arm64,linux/ppc64le,linux/s390x io.cri-containerd.image=managed
sha256:ee85eb1f0edd27fd11e3bed20f21a4001ad63107eb2f8d489429648a799c0e86 application/vnd.docker.distribution.manifest.v2+json sha256:d4b470fa1577c6ba2dfe16f1fbe87ae154783b64dcf2a750b9b0f5d3a2201ad2 21.8 MiB linux/amd64 io.cri-containerd.image=managed
sha256:f7e4814d7ce68289e74760fc199c3363fbf615c73e561ccd383b78d811d61392 application/vnd.oci.image.manifest.v1+json sha256:0d1244eeed5ec8639e4095b7c50d1fe23e6bf1700ecc550c29d799a4a26e6fc3 317.7 MiB linux/amd64 io.cri-containerd.image=managed
root@asus3-vm1:~/nvke-demo#
root@asus3-vm1:~/nvke-demo#
root@asus3-vm1:~/nvke-demo#
root@asus3-vm1:~/nvke-demo# (
set -euo pipefail
for name in platform-connectors syslog-health-monitor; do
reference="ghcr.io/nvidia/nvsentinel/$name:lab-c15934773572"
crictl --runtime-endpoint unix:///run/containerd/containerd.sock \
--image-endpoint unix:///run/containerd/containerd.sock \
inspecti "$reference" >/dev/null
printf 'READY on %s: %s\n' "$(hostname)" "$reference"
done
)
READY on asus3-vm1: ghcr.io/nvidia/nvsentinel/platform-connectors:lab-c15934773572
READY on asus3-vm1: ghcr.io/nvidia/nvsentinel/syslog-health-monitor:lab-c15934773572
For a friendlier view of the same thing, crictl images lists what is on the node. The two NVSentinel images appear with the lab-c15934773572 tag at 65.5 MB and 125 MB, right next to the Kubernetes v1.36.3 control plane images, etcd 3.6.8, CoreDNS v1.14.2 and the localhost/nvke/ovn-kubernetes image.
root@asus3-vm1:~/nvke-demo# crictl images
IMAGE TAG IMAGE ID SIZE
ghcr.io/nvidia/nvsentinel/platform-connectors lab-c15934773572 27f31ee27f0b7 65.5MB
ghcr.io/nvidia/nvsentinel/syslog-health-monitor lab-c15934773572 1a0b53262a8e2 125MB
localhost/nvke/ovn-kubernetes git-8af51273d52dfa07454435d18d37230e582d797a f7e4814d7ce68 333MB
registry.k8s.io/kube-proxy v1.36.3 be76bac219614 30MB
registry.k8s.io/kube-scheduler v1.36.3 09fc48c294224 20.5MB
ghcr.io/kube-vip/kube-vip <none> e759996fde0ac 18.8MB
registry.k8s.io/dns/k8s-dns-node-cache <none> 2d72fa4089eb1 44.2MB
registry.k8s.io/coredns/coredns v1.14.2 38667dd9be96c 23.3MB
registry.k8s.io/etcd 3.6.8-0 ee85eb1f0edd2 22.9MB
registry.k8s.io/kube-apiserver v1.36.3 3400718b4dd57 29.8MB
registry.k8s.io/kube-controller-manager v1.36.3 82c653b3047db 26.4MB
registry.k8s.io/pause 3.10.2 4a83b15d3ecfe 318kB
Now the chart itself gets a final check on the cluster. After becoming root and pointing to the kubeconfig, I set ADDON to the unpacked bundle, run helm lint against the lab values and render the chart to a file in /root. The lint passes with zero failures and the render is a dry check that the chart is sound before anything is applied.
bschmaus@asus3-vm1:~$ sudo bash
root@asus3-vm1:/home/bschmaus# export KUBECONFIG=/root/nvke-demo/nvke-demo.kubeconfig
root@asus3-vm1:/home/bschmaus# export ADDON=/opt/nvke-addons/nvsentinel
root@asus3-vm1:/home/bschmaus# helm lint "$ADDON/nvsentinel" -f "$ADDON/values-lab.yaml"
==> Linting /opt/nvke-addons/nvsentinel/nvsentinel
[INFO] Chart.yaml: icon is recommended
1 chart(s) linted, 0 chart(s) failed
root@asus3-vm1:/home/bschmaus# helm template nvsentinel "$ADDON/nvsentinel" --namespace nvsentinel --include-crds -f "$ADDON/values-lab.yaml" > /root/nvsentinel-rendered.yaml
Then comes the install. helm upgrade --install deploys the chart from the local directory into a new nvsentinel namespace with the lab values, waiting up to 15 minutes. Helm reports that the release did not exist, installed it as revision 1 and finished with the message that NVSentinel was successfully deployed. Notice there is no repository and no registry involved, since the chart comes from the add-on bundle and the images from the node's own container store.
root@asus3-vm1:/home/bschmaus# helm upgrade --install nvsentinel "$ADDON/nvsentinel" --namespace nvsentinel --create-namespace -f "$ADDON/values-lab.yaml" --wait --timeout 15m
Release "nvsentinel" does not exist. Installing it now.
NAME: nvsentinel
LAST DEPLOYED: Thu Nov 5 21:50:18 2026
NAMESPACE: nvsentinel
STATUS: deployed
REVISION: 1
DESCRIPTION: Install complete
TEST SUITE: None
NOTES:
NVSentinel has been successfully deployed
We can verify this by looking at the events. The platform connector DaemonSet creates three pods and the scheduler assigns one to each of the three nodes. For every pod the image is reported as already present on machine, which confirms the pre-loaded images were found on all three nodes and nothing was pulled over the network. The containers are created and started right after.
root@asus3-vm1:/home/bschmaus# kubectl -n nvsentinel get events --sort-by=.metadata.creationTimestamp
LAST SEEN TYPE REASON OBJECT MESSAGE
58s Normal Scheduled pod/platform-connectors-7dv85 Successfully assigned nvsentinel/platform-connectors-7dv85 to nvke-demo-cp-90c9dcc2
58s Normal SuccessfulCreate daemonset/platform-connectors Created pod: platform-connectors-9llxl
58s Normal SuccessfulCreate daemonset/platform-connectors Created pod: platform-connectors-7dv85
58s Normal SuccessfulCreate daemonset/platform-connectors Created pod: platform-connectors-nd6nm
58s Normal Scheduled pod/platform-connectors-9llxl Successfully assigned nvsentinel/platform-connectors-9llxl to nvke-demo-cp-f46d7a88
58s Normal Scheduled pod/platform-connectors-nd6nm Successfully assigned nvsentinel/platform-connectors-nd6nm to nvke-demo-cp-4c8bbf0f
57s Normal Created pod/platform-connectors-9llxl Container created
57s Normal Started pod/platform-connectors-9llxl Container started
57s Normal Pulled pod/platform-connectors-9llxl Container image "ghcr.io/nvidia/nvsentinel/platform-connectors:lab-c15934773572" already present on machine and can be accessed by the pod
57s Normal Pulled pod/platform-connectors-nd6nm Container image "ghcr.io/nvidia/nvsentinel/platform-connectors:lab-c15934773572" already present on machine and can be accessed by the pod
57s Normal Created pod/platform-connectors-nd6nm Container created
57s Normal Started pod/platform-connectors-nd6nm Container started
57s Normal Started pod/platform-connectors-7dv85 Container started
57s Normal Created pod/platform-connectors-7dv85 Container created
57s Normal Pulled pod/platform-connectors-7dv85 Container image "ghcr.io/nvidia/nvsentinel/platform-connectors:lab-c15934773572" already present on machine and can be accessed by the pod
Last, the pods themselves. All three platform-connectors pods show 1/1 and Running with zero restarts after 69 seconds. At this point I feel pretty confident the whole chain works, from NVKR build outputs to an installable ISO, a three node NVKE cluster and a workload deployed without any outside access.
root@asus3-vm1:/home/bschmaus# kubectl get pods -n nvsentinel
NAME READY STATUS RESTARTS AGE
platform-connectors-7dv85 1/1 Running 0 69s
platform-connectors-9llxl 1/1 Running 0 69s
platform-connectors-nd6nm 1/1 Running 0 69s
Wrapping Up
Hopefully this walk through provided a useful look at how a versioned set of build artifacts can be carried all the way from a recipe run to a running cluster. The same pattern should apply to the rest of the NVSentinel components as they are enabled, and keeping the NVKR lifecycle records alongside the cluster means the next rebuild or upgrade can start from facts instead of memory.
