Scale Creates Problems
An AI Factory is not a data center that happens to run AI workloads but rather a purpose-built machine for turning power, silicon, and time into trained and served models. At the scale modern training and inference demand, that machine is built from tens of thousands of GPUs, the NVLink and network fabric stitching them together, the racks and power/cooling domains housing them, and the orchestration layers, whether Kubernetes or Slurm, scheduling work across all of it.
At that scale, hardware failure is not an edge case. It is a certainty, continuously, somewhere in the fleet. A single bad GPU, a marginal NVLink connection, a flaky network card, or a rack-level power event can silently degrade a multi-thousand-GPU training job for hours before anyone notices, or quietly return a "repaired" node to service that fails again a day later. The cost of that failure isn't measured in a single server outage; it's measured in wasted GPU-hours across an entire job, in engineers manually sifting through logs to find one bad node in a haystack of thousands, and in capacity that sits offline longer than it needs to because nobody could pinpoint what was actually wrong.
The components described here exist to answer one question, asked continuously and at every layer of the stack: is this hardware and this workload healthy, and if not, what do we do about it automatically, and fast?
Two Layers One Continuous Problem
It helps to think of the AI Factory in two layers, because failure and health look different depending on where we're standing.
The infrastructure layer is the physical and out-of-band reality: GPUs, NVLink and network fabric, racks, power domains, firmware, and the signals that come from the hardware itself independent of whatever is running on top of it. A GPU can be failing in ways that are invisible to the workload until it's too late: a rising XID error rate, or a marginal fabric link. The infrastructure layer is where those signals are collected before they become a training job's problem.
The platform layer is where work actually happens: Kubernetes clusters, Slurm clusters, and the training and inference jobs scheduled onto them. Health here means something more specific than "is the hardware fine." It's "can this cluster actually run the job we're about to hand it," and when a job does fail, "was that a hardware problem, a software problem, or something in between, and can we recover without starting from zero."
The value of the components below is that they don't treat these as separate problems solved by separate teams working from separate data. They form a continuous service loop that validates, monitors, detects, diagnoses, recovers, repairs, and re-validates across both layers, so a signal caught at the infrastructure layer can prevent a failure at the platform layer, and a failure surfaced at the platform layer can be traced back to its root cause in the infrastructure layer in seconds instead of hours.
Certify the Hardware Before Workloads Even Run
The most expensive failure is the one that happens after we've already committed hours or days of GPU time to it. NVIDIA Health Validation exists to answer the most basic question before that commitment is made: "is this node healthy right now?" It runs at the node layer and applies whether the question is being asked before dispatching a single job or before releasing a whole batch of hardware back into a shared pool, such as after a repair, after a tenant workload completes, or at initial bring-up. Nodes that come back critical get excluded before they ever touch a workload, which turns a failure that would otherwise show up mid-job into a non-event.
For workloads running at real scale, whether multi-day or multi-week training runs across large Kubernetes clusters, validating individual nodes isn't enough. The fabric connecting them matters just as much as the GPUs themselves. That's what the NVIDIA Cluster Readiness Engine(NVCRE) is for: it certifies that a Kubernetes cluster as a whole can actually handle real AI workloads, not just that each node individually reports healthy, before anyone commits days of training time to finding out the hard way.
Both of these plug directly into how new capacity comes online. When a new cluster arrives from hardware delivery, it goes through a validated bring-up sequence before it reaches production: Health Validation runs per node, CRE certifies the cluster as a whole, and any failures are handled automatically before that capacity is ever released for real work. The same sequence runs again every time capacity is being prepared for a workload run, just finished a workload run or the node came back online after a repair. Nothing goes back into rotation on trust but instead goes back in on evidence.
Monitoring Workload and Hardware Health
Once a workload is live, health and resilience doesn't stop; it just changes shape, and it has to happen at both layers simultaneously.
At the platform layer, NVSentinel watches Kubernetes clusters continuously to detect and isolate unhealthy GPU nodes before they can quietly degrade a running job or accumulate in the cluster unnoticed. It’s the mechanism that keeps a K8s cluster from slowly filling up with marginal nodes that nobody flagged. It does this by leveraging real-time monitoring to collect hardware and system telemetry directly from NVIDIA DCGM, syslogs, and cloud provider maintenance events, ensuring that silent GPU or network faults are caught the moment they happen. Once a failure is detected, the platform automatically protects ongoing workloads by cordoning the affected node to block new scheduling requests while safely draining active pods before corruption can spread to other jobs. Finally, NVSentinel initiates automated remediation, orchestrating targeted GPU resets or full machine reboots, and leverages an integrated NVCRE provider to trigger the NVIDIA Cluster Readiness Engine (NVCRE), automatically driving real distributed workloads, such as NCCL communication tests and DCGM diagnostics, to run through a comprehensive post-remediation certification check before safely returning the healthy node to service.
Tying all of this together across the fleet is Fleet Intelligence Agent, which answers "how do I track GPU fleet health, availability, integrity, and SLA compliance across my entire fleet, right now?" It maintains a living inventory, surfaces alerts as health status changes, and gives both infrastructure and platform teams a shared, real-time picture of fleet state through a dashboard and API, including recommended remediation actions, reporting, and built-in AI integrations. It's the continuous-operations backbone of what the fleet's health looks like at any point in time, including the moment something went wrong.
From Failure to Diagnosis in Seconds
Even with validation and continuous monitoring, failures happen. What separates a resilient AI Factory from a fragile one is how fast the cause gets identified and how much of the response is automatic.
NVIDIA Attribution exists for exactly this moment: "why did this Slurm or Kubernetes workload fail?" When a workload fails, Attribution attributes that failure to the specific responsible node and the specific signal involved within seconds, not hours of manual log correlation across a cluster with thousands of nodes. That verdict isn't just informational; it's structured evidence. It can be shared directly when escalating a hardware issue, and it feeds directly into automated node-exclusion decisions so the next attempt at the workload doesn't land back on the same bad hardware.
Recovery takes this one step further. NVIDIA Recovery answers "how do I automate recovery when my job fails?" When a training anomaly occurs, such as a hang or a crash, NVIDIA Recovery detects it and restarts the workload automatically from the last checkpoint, rather than requiring a human to notice, diagnose, and manually intervene. When Attribution is deployed alongside it, NVIDIA Recovery consults Attribution's node-exclusion logic before restarting, so the workload resumes only on hardware that's actually been verified healthy, and the verdict that drove that decision is surfaced back to the end user.
This is the difference between a training run that silently loses a day of progress because nobody caught a hang until morning, and one that self-heals within minutes because attribution and recovery are working together instead of as isolated tools.
Together is Better
At the massive scale of an AI Factory, treating infrastructure and scheduling platforms as isolated systems is no longer viable; instead, bringing these layers together into a unified loop is the only way to ensure operational resilience. Rather than forcing separate teams to operate in silos with fragmented data, specialized tools like NVIDIA Health Validation, NVCRE, NVSentinel, NVIDIA Attribution, and NVIDIA Recovery cooperate continuously to form an unbroken service loop. By seamlessly cross-correlating real-time physical hardware signals with application workload behaviors, this collaborative ecosystem can instantly trace a platform failure to its physical root cause, block or remediate faulty components, and automatically restore progress. True resilience proves that together is better, transforming a chaotic avalanche of isolated cluster logs into a collective, automated defense machine that keeps multi-node AI workloads running without missing a beat.
