When Every Packet Stops: Fixing e1000e Hangs

I sat down to do some work on one of my lab hosts and found I could no longer ssh to it. I went to the console and the system was up and responsive, but I could not ping anything: not the gateway, not another host on the same subnet, nothing. On top of that, the console was scrolling a steady stream of kernel messages that kept repeating every couple of seconds.

A host that is alive but completely cut off from the network turns into guesswork very quickly. Is it the switch port, the cable, the DHCP lease, or the host itself? In this case it turned out to be the network card driver, and the fix was a one line change. Let's walk through what I saw, what the messages mean, and how to make the fix survive a reboot.

The Error Messages

This is what was streaming across the console. The host is named asus2, the interface is eno1, and the messages repeat roughly every two seconds:

  Oct 22 15:38:24 asus2 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang:#012  TDH                  <da>#012  TDT                  <5>#012  next_to_use          <5>#012  next_to_clean        <d9>#012buffer_info[next_to_clean]:#012  time_stamp           <1d38993a8>#012  next_to_watch        <da>#012  jiffies              <1d3a2b4c0>#012  next_to_watch.status <0>#012MAC Status             <80083>#012PHY Status             <796d>#012PHY 1000BASE-T Status  <3800>#012PHY Extended Status    <3000>#012PCI Status             <10>
  Oct 22 15:38:26 asus2 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang:#012  TDH                  <da>#012  TDT                  <5>#012  next_to_use          <5>#012  next_to_clean        <d9>#012buffer_info[next_to_clean]:#012  time_stamp           <1d38993a8>#012  next_to_watch        <da>#012  jiffies              <1d3a2bcc0>#012  next_to_watch.status <0>#012MAC Status             <80083>#012PHY Status             <796d>#012PHY 1000BASE-T Status  <3800>#012PHY Extended Status    <3000>#012PCI Status             <10>
  Oct 22 15:38:28 asus2 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang:#012  TDH                  <da>#012  TDT                  <5>#012  next_to_use          <5>#012  next_to_clean        <d9>#012buffer_info[next_to_clean]:#012  time_stamp           <1d38993a8>#012  next_to_watch        <da>#012  jiffies              <1d3a2c480>#012  next_to_watch.status <0>#012MAC Status             <80083>#012PHY Status             <796d>#012PHY 1000BASE-T Status  <3800>#012PHY Extended Status    <3000>#012PCI Status             <10>
  Oct 22 15:38:30 asus2 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang:#012  TDH                  <da>#012  TDT                  <5>#012  next_to_use          <5>#012  next_to_clean        <d9>#012buffer_info[next_to_clean]:#012  time_stamp           <1d38993a8>#012  next_to_watch        <da>#012  jiffies              <1d3a2cc40>#012  next_to_watch.status <0>#012MAC Status             <80083>#012PHY Status             <796d>#012PHY 1000BASE-T Status  <3800>#012PHY Extended Status    <3000>#012PCI Status             <10>
  Oct 22 15:38:32 asus2 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang:#012  TDH                  <da>#012  TDT                  <5>#012  next_to_use          <5>#012  next_to_clean        <d9>#012buffer_info[next_to_clean]:#012  time_stamp           <1d38993a8>#012  next_to_watch        <da>#012  jiffies              <1d3a2d401>#012  next_to_watch.status <0>#012MAC Status             <80083>#012PHY Status             <796d>#012PHY 1000BASE-T Status  <3800>#012PHY Extended Status    <3000>#012PCI Status             <10>
  Oct 22 15:38:34 asus2 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang:#012  TDH                  <da>#012  TDT                  <5>#012  next_to_use          <5>#012  next_to_clean        <d9>#012buffer_info[next_to_clean]:#012  time_stamp           <1d38993a8>#012  next_to_watch        <da>#012  jiffies              <1d3a2dc00>#012  next_to_watch.status <0>#012MAC Status             <80083>#012PHY Status             <796d>#012PHY 1000BASE-T Status  <3800>#012PHY Extended Status    <3000>#012PCI Status             <10>
  Oct 22 15:38:36 asus2 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang:#012  TDH                  <da>#012  TDT                  <5>#012  next_to_use          <5>#012  next_to_clean        <d9>#012buffer_info[next_to_clean]:#012  time_stamp           <1d38993a8>#012  next_to_watch        <da>#012  jiffies              <1d3a2e3c0>#012  next_to_watch.status <0>#012MAC Status             <80083>#012PHY Status             <796d>#012PHY 1000BASE-T Status  <3800>#012PHY Extended Status    <3000>#012PCI Status             <10>
  Oct 22 15:38:38 asus2 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang:#012  TDH                  <da>#012  TDT                  <5>#012  next_to_use          <5>#012  next_to_clean        <d9>#012buffer_info[next_to_clean]:#012  time_stamp           <1d38993a8>#012  next_to_watch        <da>#012  jiffies              <1d3a2eb80>#012  next_to_watch.status <0>#012MAC Status             <80083>#012PHY Status             <796d>#012PHY 1000BASE-T Status  <3800>#012PHY Extended Status    <3000>#012PCI Status             <10>
  Oct 22 15:38:40 asus2 kernel: e1000e 0000:00:1f.6 eno1: Detected Hardware Unit Hang:#012  TDH                  <da>#012  TDT                  <5>#012  next_to_use          <5>#012  next_to_clean        <d9>#012buffer_info[next_to_clean]:#012  time_stamp           <1d38993a8>#012  next_to_watch        <da>#012  jiffies              <1d3a2f340>#012  next_to_watch.status <0>#012MAC Status             <80083>#012PHY Status             <796d>#012PHY 1000BASE-T Status  <3800>#012PHY Extended Status    <3000>#012PCI Status             <10>

The Error Messages Meaning

The first thing to pull out of those lines is the device. e1000e 0000:00:1f.6 eno1 is the Intel network controller built into the motherboard, handled by the e1000e driver, and eno1 is the interface name the system gave it. The messages come from the kernel, so this is the driver talking, not an application.

"Detected Hardware Unit Hang" is the driver telling us it queued packets for transmit and the network card never finished sending them. The transmit descriptor ring has a head and a tail. The tail (TDT, here <5>) is where the driver is adding new work and the head (TDH, here <da>) is where the hardware has gotten to. Healthy hardware keeps chasing the tail. Here the head never moves: next_to_clean is <d9>, next_to_watch is <da>, and the time_stamp stays at <1d38993a8> in every message while jiffies keeps counting up. In other words, the same descriptor has been sitting there for a long time.

The other detail that matters is that the link itself looks healthy. The MAC Status of <80083> and the PHY Status of <796d> along with PHY 1000BASE-T Status tell us the link is up at gigabit speed. Only the transmit path is wedged. That fits my symptoms: the host was up and the console was alive, but nothing I tried to send ever left the machine, which is why ssh to the host failed and ping got no replies. The driver repeats the message every two seconds until it manages to reset the adapter, and on this host that was not enough to recover.

Note: I cannot prove a root cause from this log alone. What I can say is that this pattern is a well documented e1000e problem, and the next section covers what usually causes it.

Common Triggers

There is no single cause, but these are the ones that come up most often with the e1000e driver on Intel onboard controllers:

• TCP segmentation offload settings (TSO, GSO and GRO). Bugs in how the driver handles offloaded segmentation are one of the most commonly reported triggers.
• Power management on the card or the PCIe link, such as Energy Efficient Ethernet (EEE) or Active State Power Management (ASPM). EEE lets the card and the switch port drop into a low power idle state, and on some controllers the transmit path does not wake back up cleanly.
• Less often, an old driver, kernel or BIOS and Intel Management Engine firmware, or a failing card or cable.

It is also worth thinking about what else depends on the port. On a host that runs virtual machines, a hung port stalls the networking of everything behind it, not just the host.  This particular setup does host multiple virtual machines across a bridge port.

Resolution

Before changing anything it helps to see what is currently enabled. We can check the driver details and the current offload and EEE settings with:

ethtool -i eno1
ethtool -k eno1 | grep -E 'tcp-segmentation|generic-segmentation|generic-receive'
ethtool --show-eee eno1

The usual first suspect is the offload settings, which can be turned off on the running system with:

sudo ethtool -K eno1 tso off gso off gro off

In my case I did not need that. Turning off Energy Efficient Ethernet is what made my errors go away:

sudo ethtool --set-eee eno1 eee off

The moment I ran it, the stream of messages stopped. That identifies the culprit: with EEE on, the card and the switch port negotiate low power idle and this Intel controller sometimes fails to wake its transmit path, which is the stall the log was describing. Since EEE was the trigger, there was no reason to change the TSO, GSO or GRO settings, so I left them alone.

We can verify the change took effect by asking for the EEE state again and looking at the kernel log:

ethtool --show-eee eno1
sudo journalctl -k -b | grep -i 'e1000e'

The EEE status should now read disabled, and no new "Detected Hardware Unit Hang" lines should show up. At this point the console should be quiet again, and the host should be able to reach the network.

Persistent The Fix

There is one catch. ethtool --set-eee only changes the running state of the card, so the setting is lost on reboot and the hangs would come right back. Since I did not want to find myself at the console again, let's make the fix apply automatically. There are two ways to do it.

Option 1: A systemd link file

The first method is a systemd .link file. It is declarative, and the setting is applied when udev configures the interface. The EnergyEfficientEthernet= option needs systemd 248 or newer. Ubuntu 24.04 ships systemd 255, and we can check what a host has with systemctl --version.

sudo tee /etc/systemd/network/10-eno1-eee.link <<'EOF'
[Match]
OriginalName=eno1

[Link]
EnergyEfficientEthernet=false
EOF
sudo update-initramfs -u

The update-initramfs -u step makes sure the link file is also present in the initramfs so it is picked up early in the boot. Then reboot the host and check the result:

ethtool --show-eee eno1

If EEE reports disabled after the reboot, the link file did its job. If it shows enabled again, move on to the second option.

Option 2: A oneshot systemd service

While the link file is the cleaner choice in my opinion, a small service that runs the same ethtool command at boot works everywhere and is easy to reason about. It waits for the interface to exist and then turns EEE off:

sudo tee /etc/systemd/system/eno1-eee-off.service <<'EOF'
[Unit]
Description=Disable EEE on eno1
After=network-pre.target sys-subsystem-net-devices-eno1.device
Wants=sys-subsystem-net-devices-eno1.device

[Service]
Type=oneshot
ExecStart=/usr/sbin/ethtool --set-eee eno1 eee off

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now eno1-eee-off.service

If the service reports that it cannot find the binary, confirm where ethtool lives with command -v ethtool and adjust the path in ExecStart. Either way, the check afterward is the same.

Validating After a Reboot

With either option in place, reboot the host and confirm the setting was applied without any manual help. We can verify this by checking the EEE state and then watching the kernel log for any stray hangs:

ethtool --show-eee eno1
sudo journalctl -k -b | grep -i 'e1000e'

If everything looks good the EEE status will read disabled and there will be no "Detected Hardware Unit Hang" lines in the log for the current boot. At this point I would feel pretty confident the node can be rebooted at any time and come back up with a working network.

Keep in mind EEE only saves a few watts of idle power, so leaving it off is essentially free on a lab host. If the hangs ever return with EEE disabled, the next things I would try are the TSO, GSO and GRO offload settings from earlier, the kernel parameter pcie_aspm=off, and a kernel or BIOS and Intel Management Engine firmware update. As a last resort a different network card, such as a USB or add-in adapter, takes the onboard controller out of the picture entirely.

Hopefully this write-up saves someone a trip to the console the next time an Intel e1000e interface hangs and the host drops off the network.

Health and Resilience in the AI Factory

Scale Creates Problems

An AI Factory is not a data center that happens to run AI workloads but rather a purpose-built machine for turning power, silicon, and time into trained and served models. At the scale modern training and inference demand, that machine is built from tens of thousands of GPUs, the NVLink and network fabric stitching them together, the racks and power/cooling domains housing them, and the orchestration layers, whether Kubernetes or Slurm, scheduling work across all of it.

At that scale, hardware failure is not an edge case. It is a certainty, continuously, somewhere in the fleet. A single bad GPU, a marginal NVLink connection, a flaky network card, or a rack-level power event can silently degrade a multi-thousand-GPU training job for hours before anyone notices, or quietly return a "repaired" node to service that fails again a day later. The cost of that failure isn't measured in a single server outage; it's measured in wasted GPU-hours across an entire job, in engineers manually sifting through logs to find one bad node in a haystack of thousands, and in capacity that sits offline longer than it needs to because nobody could pinpoint what was actually wrong.

The components described here exist to answer one question, asked continuously and at every layer of the stack: is this hardware and this workload healthy, and if not, what do we do about it automatically, and fast?

Two Layers One Continuous Problem

It helps to think of the AI Factory in two layers, because failure and health look different depending on where we're standing.

The infrastructure layer is the physical and out-of-band reality: GPUs, NVLink and network fabric, racks, power domains, firmware, and the signals that come from the hardware itself independent of whatever is running on top of it. A GPU can be failing in ways that are invisible to the workload until it's too late: a rising XID error rate, or a marginal fabric link. The infrastructure layer is where those signals are collected before they become a training job's problem.

The platform layer is where work actually happens: Kubernetes clusters, Slurm clusters, and the training and inference jobs scheduled onto them. Health here means something more specific than "is the hardware fine." It's "can this cluster actually run the job we're about to hand it," and when a job does fail, "was that a hardware problem, a software problem, or something in between, and can we recover without starting from zero."

The value of the components below is that they don't treat these as separate problems solved by separate teams working from separate data. They form a continuous service loop that validates, monitors, detects, diagnoses, recovers, repairs, and re-validates across both layers, so a signal caught at the infrastructure layer can prevent a failure at the platform layer, and a failure surfaced at the platform layer can be traced back to its root cause in the infrastructure layer in seconds instead of hours.

Certify the Hardware Before Workloads Even Run

The most expensive failure is the one that happens after we've already committed hours or days of GPU time to it. NVIDIA Health Validation exists to answer the most basic question before that commitment is made: "is this node healthy right now?" It runs at the node layer and applies whether the question is being asked before dispatching a single job or before releasing a whole batch of hardware back into a shared pool, such as after a repair, after a tenant workload completes, or at initial bring-up. Nodes that come back critical get excluded before they ever touch a workload, which turns a failure that would otherwise show up mid-job into a non-event.

For workloads running at real scale, whether multi-day or multi-week training runs across large Kubernetes clusters, validating individual nodes isn't enough. The fabric connecting them matters just as much as the GPUs themselves. That's what the NVIDIA Cluster Readiness Engine(NVCRE) is for: it certifies that a Kubernetes cluster as a whole can actually handle real AI workloads, not just that each node individually reports healthy, before anyone commits days of training time to finding out the hard way.

Both of these plug directly into how new capacity comes online. When a new cluster arrives from hardware delivery, it goes through a validated bring-up sequence before it reaches production: Health Validation runs per node, CRE certifies the cluster as a whole, and any failures are handled automatically before that capacity is ever released for real work. The same sequence runs again every time capacity is being prepared for a workload run, just finished a workload run or the node came back online after a repair. Nothing goes back into rotation on trust but instead goes back in on evidence.

Monitoring Workload and Hardware Health

Once a workload is live, health and resilience doesn't stop; it just changes shape, and it has to happen at both layers simultaneously.

At the platform layer, NVSentinel watches Kubernetes clusters continuously to detect and isolate unhealthy GPU nodes before they can quietly degrade a running job or accumulate in the cluster unnoticed. It’s the mechanism that keeps a K8s cluster from slowly filling up with marginal nodes that nobody flagged. It does this by leveraging real-time monitoring to collect hardware and system telemetry directly from NVIDIA DCGM, syslogs, and cloud provider maintenance events, ensuring that silent GPU or network faults are caught the moment they happen. Once a failure is detected, the platform automatically protects ongoing workloads by cordoning the affected node to block new scheduling requests while safely draining active pods before corruption can spread to other jobs. Finally, NVSentinel initiates automated remediation, orchestrating targeted GPU resets or full machine reboots, and leverages an integrated NVCRE provider to trigger the NVIDIA Cluster Readiness Engine (NVCRE), automatically driving real distributed workloads, such as NCCL communication tests and DCGM diagnostics, to run through a comprehensive post-remediation certification check before safely returning the healthy node to service.

Tying all of this together across the fleet is Fleet Intelligence Agent, which answers "how do I track GPU fleet health, availability, integrity, and SLA compliance across my entire fleet, right now?" It maintains a living inventory, surfaces alerts as health status changes, and gives both infrastructure and platform teams a shared, real-time picture of fleet state through a dashboard and API, including recommended remediation actions, reporting, and built-in AI integrations. It's the continuous-operations backbone of what the fleet's health looks like at any point in time, including the moment something went wrong.

From Failure to Diagnosis in Seconds

Even with validation and continuous monitoring, failures happen. What separates a resilient AI Factory from a fragile one is how fast the cause gets identified and how much of the response is automatic.

NVIDIA Attribution exists for exactly this moment: "why did this Slurm or Kubernetes workload fail?" When a workload fails, Attribution attributes that failure to the specific responsible node and the specific signal involved within seconds, not hours of manual log correlation across a cluster with thousands of nodes. That verdict isn't just informational; it's structured evidence. It can be shared directly when escalating a hardware issue, and it feeds directly into automated node-exclusion decisions so the next attempt at the workload doesn't land back on the same bad hardware.

Recovery takes this one step further. NVIDIA Recovery answers "how do I automate recovery when my job fails?" When a training anomaly occurs, such as a hang or a crash, NVIDIA Recovery detects it and restarts the workload automatically from the last checkpoint, rather than requiring a human to notice, diagnose, and manually intervene. When Attribution is deployed alongside it, NVIDIA Recovery consults Attribution's node-exclusion logic before restarting, so the workload resumes only on hardware that's actually been verified healthy, and the verdict that drove that decision is surfaced back to the end user.

This is the difference between a training run that silently loses a day of progress because nobody caught a hang until morning, and one that self-heals within minutes because attribution and recovery are working together instead of as isolated tools.

Together is Better

At the massive scale of an AI Factory, treating infrastructure and scheduling platforms as isolated systems is no longer viable; instead, bringing these layers together into a unified loop is the only way to ensure operational resilience. Rather than forcing separate teams to operate in silos with fragmented data, specialized tools like NVIDIA Health Validation, NVCRE, NVSentinel, NVIDIA Attribution, and NVIDIA Recovery cooperate continuously to form an unbroken service loop. By seamlessly cross-correlating real-time physical hardware signals with application workload behaviors, this collaborative ecosystem can instantly trace a platform failure to its physical root cause, block or remediate faulty components, and automatically restore progress. True resilience proves that together is better, transforming a chaotic avalanche of isolated cluster logs into a collective, automated defense machine that keeps multi-node AI workloads running without missing a beat.

Walking Through NVIDIA Attribution

When a GPU job dies in the middle of the night, the first question is almost never "what happened?" It is "whose problem is this?" Is the node bad and does it need to be pulled out of rotation, or did someone submit a job that asked for more memory than the GPU has? NVIDIA Attribution is a service built to answer that question. It starts with a described workload, hand it the evidence collected (log lines from the nodes the job ran on, for example), and it matches that evidence against a ruleset to produce a verdict: the root symptom, the outcome and the failure domain it belongs to.

In this post I am going to walk through a small end to end demo of Attribution running on a single virtual machine. We will build the Attribution Blueprint container image, push it to a registry, deploy it into a MicroK8s cluster, and then call the API by hand with a simulated failed PyTorch job. The failure here will be a CUDA out of memory error, which makes for a nice example because the right answer is that the user owns the problem and the hardware is fine. I should be upfront that the workload here is made up and so is the single log line. It is a way to see the moving pieces, not a real incident. With all that said let's get started together on our journey.

Build and Push the Blueprint Image

The first step is building the container image from the Attribution repository. The virtual machine is an arm64 system so I am passing --platform linux/arm64, and I am tagging the image for the registry that runs in my lab environment on port 5000. This is a multi-stage build. One stage uses a Node 22 image to compile the web dashboard frontend with pnpm and Vite, and the final stage is Python 3.14 on Debian trixie where uv installs the blueprint, the service app and the attribution library. The build also installs gcc just long enough to compile a few dependencies and then purges it again to keep the image small.

$ docker build --platform linux/arm64  -f blueprint/docker/Dockerfile  -t registry.schmaustech.com:5000/attribution-blueprint-web:latest .
[+] Building 1.4s (32/32) FINISHED                                                                                                                                                                                                docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                                                        0.0s
 => => transferring dockerfile: 3.07kB                                                                                                                                                                                                      0.0s
 => resolve image config for docker-image://docker.io/docker/dockerfile:1.7                                                                                                                                                                 0.4s
 => CACHED docker-image://docker.io/docker/dockerfile:1.7@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e                                                                                                           0.0s
 => [internal] load metadata for ghcr.io/astral-sh/uv:0.10.12                                                                                                                                                                               0.4s
 => [internal] load metadata for docker.io/library/node:22-bookworm-slim                                                                                                                                                                    0.6s
 => [internal] load metadata for docker.io/library/python:3.14.6-slim-trixie                                                                                                                                                                0.7s
 => [internal] load build context                                                                                                                                                                                                           0.0s
 => => transferring context: 60.10kB                                                                                                                                                                                                        0.0s
 => [web-build 1/4] FROM docker.io/library/node:22-bookworm-slim@sha256:c3de60bf2f9dd0ac6370e6117950ff62d6e339527e7472301c9c78a017978392                                                                                                    0.0s
 => FROM ghcr.io/astral-sh/uv:0.10.12@sha256:72ab0aeb448090480ccabb99fb5f52b0dc3c71923bffb5e2e26517a1c27b7fec                                                                                                                               0.0s
 => [stage-1  1/19] FROM docker.io/library/python:3.14.6-slim-trixie@sha256:7bec7ddcddeff7975d6ba9b4be7dd6f6b2f55e7491539145e2978f7f97ce9144                                                                                                0.0s
 => CACHED [stage-1  2/19] COPY --from=ghcr.io/astral-sh/uv:0.10.12 /uv /uvx /bin/                                                                                                                                                          0.0s
 => CACHED [stage-1  3/19] WORKDIR /opt/blueprint                                                                                                                                                                                           0.0s
 => CACHED [stage-1  4/19] COPY blueprint/pyproject.toml blueprint/uv.lock ./                                                                                                                                                               0.0s
 => CACHED [stage-1  5/19] COPY blueprint/src/ ./src/                                                                                                                                                                                       0.0s
 => CACHED [web-build 2/4] WORKDIR /frontend                                                                                                                                                                                                0.0s
 => CACHED [web-build 3/4] COPY blueprint/src/attribution_blueprint/webui/frontend/ ./                                                                                                                                                      0.0s
 => CACHED [web-build 4/4] RUN corepack enable && pnpm install --frozen-lockfile && pnpm exec tsc -b && pnpm exec vite build --outDir dist                                                                                                  0.0s
 => CACHED [stage-1  6/19] COPY --from=web-build /frontend/dist ./src/attribution_blueprint/webui/static/                                                                                                                                   0.0s
 => CACHED [stage-1  7/19] COPY service/service-app/nvidia_attribution_client/ /opt/service/service-app/nvidia_attribution_client/                                                                                                          0.0s
 => CACHED [stage-1  8/19] RUN uv sync --extra web --no-dev --no-editable --frozen                                                                                                                                                          0.0s
 => CACHED [stage-1  9/19] WORKDIR /opt/attribution-service                                                                                                                                                                                 0.0s
 => CACHED [stage-1 10/19] COPY service/attribution-lib/ /opt/attribution-lib/                                                                                                                                                              0.0s
 => CACHED [stage-1 11/19] COPY service/service-app/pyproject.toml service/service-app/uv.lock ./                                                                                                                                           0.0s
 => CACHED [stage-1 12/19] COPY service/service-app/_build_version.py ./                                                                                                                                                                    0.0s
 => CACHED [stage-1 13/19] COPY service/service-app/README.md service/service-app/LICENSE ./                                                                                                                                                0.0s
 => CACHED [stage-1 14/19] COPY service/service-app/app/ ./app/                                                                                                                                                                             0.0s
 => CACHED [stage-1 15/19] COPY service/service-app/nvidia_attribution_client/ ./nvidia_attribution_client/                                                                                                                                 0.0s
 => CACHED [stage-1 16/19] RUN apt-get update     && apt-get install -y --no-install-recommends gcc libc6-dev     && uv sync --no-dev --no-editable --frozen     && apt-get purge -y --auto-remove gcc libc6-dev     && rm -rf /var/lib/ap  0.0s
 => CACHED [stage-1 17/19] COPY blueprint/docker/start.sh /usr/local/bin/attribution-blueprint-start                                                                                                                                        0.0s
 => CACHED [stage-1 18/19] RUN chmod +x /usr/local/bin/attribution-blueprint-start                                                                                                                                                          0.0s
 => CACHED [stage-1 19/19] WORKDIR /opt/blueprint                                                                                                                                                                                           0.0s
 => exporting to image                                                                                                                                                                                                                      0.0s
 => => exporting layers                                                                                                                                                                                                                     0.0s
 => => writing image sha256:7417910ee0d5bc1d35e8030e63996939c804eff310e63c9209131f4cc443df96                                                                                                                                                0.0s
 => => naming to registry.schmaustech.com:5000/attribution-blueprint-web:latest

The above output we can observe almost every step says CACHED and the whole build finished in a little over a second. That is only because I had built this image before, so on a first run expect it to take considerably longer while the frontend and Python dependencies are pulled and built. At this point we have an image sitting in the local Docker image store, so let's push it to the registry so the cluster can pull it.

$ docker push registry.schmaustech.com:5000/attribution-blueprint-web:latest
The push refers to repository [registry.schmaustech.com:5000/attribution-blueprint-web]
5f70bf18a086: Pushed 
b78e1eeee5c9: Pushed 
e5c73dd8dfa2: Pushed 
bd7bce39331b: Pushed 
66a7a0b9edd3: Pushed 
6a52fb04e724: Pushed 
87fa1d674280: Pushed 
199eb863db71: Pushed 
9c5d441c4e95: Pushed 
fe962dda67f4: Pushed 
56f16490e6f3: Pushed 
5b91b81ea413: Pushed 
740cf88824b2: Pushed 
2a56874aabc8: Pushed 
a4a5b6e3a44e: Pushed 
d142ef99965e: Pushed 
c0c0660a42bc: Pushed 
740399ee44b6: Pushed 
5f35d866a418: Pushed 
2e68013aee3d: Pushed 
86d005034ff8: Pushed 
c01c35a040a2: Pushed 
local: digest: sha256:7c387a2c2750c7e780f9ce6d397c97ddc6193da287daf0827e69358753e84119 size: 4923

Twenty-two layers pushed and a digest at the bottom. If everything looks good we can move onto deploying it.

Deploy Attribution to Kubernetes

For the deployment I always prefer yaml manifests, so the following is a single file that holds everything Attribution needs. Let's look at it and then go through what each object does.

$ cat attribution-deployment.yaml 
apiVersion: v1
kind: Namespace
metadata:
  name: attribution
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: attribution-data
  namespace: attribution
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path-persistent
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: attribution
  namespace: attribution
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: attribution
  template:
    metadata:
      labels:
        app: attribution
    spec:
      containers:
        - name: attribution
          image: registry.schmaustech.com:5000/attribution-blueprint-web:latest
          imagePullPolicy: Always
          env:
            - name: ATTRIBUTION_DB
              value: /data/attributions.db
            - name: ATTRIBUTION_SERVICE_PORT
              value: "8000"
            - name: ATTRIBUTION_WEB_PORT
              value: "8080"
          ports:
            - name: api
              containerPort: 8000
            - name: web
              containerPort: 8080
          startupProbe:
            httpGet:
              path: /
              port: web
            periodSeconds: 5
            failureThreshold: 36
          readinessProbe:
            httpGet:
              path: /
              port: web
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /v2/healthz
              port: api
            periodSeconds: 20
            failureThreshold: 3
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              memory: 1Gi
          volumeMounts:
            - name: data
              mountPath: /data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: attribution-data
---
apiVersion: v1
kind: Service
metadata:
  name: attribution
  namespace: attribution
spec:
  type: NodePort
  selector:
    app: attribution
  ports:
    - name: web
      port: 8080
      targetPort: web
      nodePort: 30080
    - name: api
      port: 8000
      targetPort: api

The file starts with an attribution namespace and a one gigabyte PersistentVolumeClaim named attribution-data. That claim is what holds the SQLite database the web dashboard reads from, and I am using the local-path-persistent storage class since this is a single node lab. The Deployment runs one replica and uses the Recreate strategy rather than a rolling update. Since the volume is ReadWriteOnce and SQLite does not like two writers, we want the old pod gone before the new one starts.

The container exposes two ports: 8000 for the Attribution API and 8080 for the web dashboard, and the environment variables point the database at /data/attributions.db on the mounted volume. The probes are worth a quick look. The startup and readiness probes hit the web UI on /, while the liveness probe hits /v2/healthz on the API port. The startup probe gives the pod up to three minutes (5 second period times 36 failures) to come up, which is plenty of time for the first start. Finally the Service is a NodePort, with the web UI pinned to 30080 and the API port left for Kubernetes to assign.

With the manifest ready we can create everything efficiently in one shot.

$ kubectl create -f attribution-deployment.yaml 
namespace/attribution created
persistentvolumeclaim/attribution-data created
deployment.apps/attribution created
service/attribution created

Now let's see if the pod comes up. Below we are asking for the pods, the claim and the service all at once so we can confirm each piece.

$ kubectl -n attribution get pods,pvc,service -o wide
NAME                               READY   STATUS    RESTARTS   AGE   IP             NODE         NOMINATED NODE   READINESS GATES
pod/attribution-75f57d76d9-rw592   1/1     Running   0          45s   10.1.174.232   gpu-node     <none>           <none>

NAME                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS            VOLUMEATTRIBUTESCLASS   AGE   VOLUMEMODE
persistentvolumeclaim/attribution-data   Bound    pvc-9e497dc7-2b0f-4cef-bb8e-e0d074241f75   1Gi        RWO            local-path-persistent   <unset>                 45s   Filesystem

NAME                  TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE   SELECTOR
service/attribution   NodePort   10.152.183.26   <none>        8080:30080/TCP,8000:30496/TCP   45s   app=attribution

The pod is Running and 1/1 ready, the claim is Bound to a one gigabyte volume, and the service shows the web UI on 8080 mapped to node port 30080 and the API on 8000 mapped to node port 30496. We will use both of those node ports in a moment. A plain pod listing a bit later confirms it has stayed up with no restarts.

$ kubectl get pods -n attribution
NAME                           READY   STATUS    RESTARTS   AGE
attribution-75f57d76d9-rw592   1/1     Running   0          74s

Validate the Service

Before sending any real requests I want to confirm both ports answer. We can verify this by calling the health endpoint on the API node port, using the IP address of the virtual machine.

$ curl --max-time 10 -fsS http://192.168.0.111:30496/v2/healthz
null

A bare null looks odd, but it is what the health endpoint returns on success. What matters is that curl was run with -f, so it would have failed loudly on any HTTP error. Next up is the web dashboard, where I only care about the status code.

$ curl --max-time 10 -fsS -o /dev/null -w 'Web UI HTTP %{http_code}\n' http://192.168.0.111:30080/
Web UI HTTP 200

An HTTP 200 from the dashboard. At this point I feel pretty confident the deployment is healthy. To save some typing for the rest of the demo I am exporting the API address into a variable and calling the health endpoint once more through it.

$ export SVC="http://192.168.0.111:30496"
$ curl --fail-with-body -sS "${SVC}/v2/healthz"

Create an Attributor and Submit an Observation

Now for the interesting part. Everything in the Attribution API hangs off an attributor, which is how I think about it: a record of one workload you want an answer about. To create one we POST to /v2/attributor with a description of the workload and some metadata about where it ran.

The workload here is a made up PyTorch job named curl-example, submitted through Kueue, that ran for thirty minutes on GPUs 0 and 1 of a node called node-a and ended in a FAILED status. The metadata identifies the cluster, the agent that reported it, a tenant and the ruleset to use, which is default. I am piping the response through jq to pull out just the attributor_id and store it in a variable called ATT so we can reuse it.

$ export ATT="$(
   curl --fail-with-body -sS \
     -X POST \
     -H 'Content-Type: application/json' \
     "${SVC}/v2/attributor" \
     --data-binary @- <<'JSON' | jq -er '.attributor_id'
 {
   "workload": {
     "id": "curl-example-001",
     "type": "kueue",
     "job_start_time": "2026-10-07T10:00:00Z",
     "job_end_time": "2026-10-07T10:30:00Z",
     "interest_start_time": "2026-10-07T10:00:00Z",
     "interest_end_time": "2026-10-07T10:30:00Z",
     "timezone": "UTC",
     "status": "FAILED",
     "nodes": {
       "node-a": [0, 1]
     },
     "name": "curl-example",
     "username": "example-user",
     "framework": "pytorch"
   },
   "metadata": {
     "cluster": "local-microk8s",
     "agent": "curl",
     "tenant": "example-tenant",
     "ruleset": "default"
   }
 }
JSON
 )"

$ 

1094296642135612619

That large number at the bottom is the attributor ID. The attributor knows about the workload but it has not seen any evidence yet, so let's give it some. Evidence is submitted as an observation against the attributor. The observation has a context saying it came from an application log (applog), and a body that carries the log line along with the resource it came from. In this case that is a single line, [rank 0] RuntimeError: CUDA out of memory, from node-a. The body is itself a JSON string, which is why you see all of the escaped quotes.

$ curl --fail-with-body -sS \
    -X POST \
    -H 'Content-Type: application/json' \
    "${SVC}/v2/attributor/${ATT}/observation" \
    --data-binary @- <<'JSON'
  {
    "context": {
      "time_interval": {
        "start": null,
        "end": null
      },
      "source": "applog"
    },
    "body": "[{\"body\":\"[rank 0] RuntimeError: CUDA out of memory\",\"resources\":[{\"kind\":\"node\",\"hostname\":\"node-a\"}]}]"
  }
JSON

{"observation_id":-5460837407236178041,"model_type":"ObservationResponseV2"

The API hands back an observation ID, which tells us the line was accepted and recorded against our attributor.

Read the Verdict

With the evidence in place we can ask Attribution what it thinks happened.

$ curl --fail-with-body -sS "${SVC}/v2/attributor/${ATT}/attribution" | jq .
{
  "verdict": {
    "root_symptom": {
      "symptom": "cudaError out of memory",
      "source": {
        "observation_id": -5460837407236178041,
        "source": "applog",
        "pos": 1,
        "line": "[rank 0] RuntimeError: CUDA out of memory",
        "rule": "cudaError out of memory",
        "time": null
      },
      "catalog_id": "cudaerror-out-of-memory"
    },
    "attribution_outcome": "Fail User",
    "detail": "Not Applicable",
    "terminal_states": [
      "Fail"
    ],
    "failure_domains": [
      "User"
    ],
    "cancellation_actors": [],
    "uncertainty_type": null
  },
  "symptoms": {
    "cudaError out of memory": [
      {
        "type": "NodeSource",
        "node": "node-a",
        "status": "bad",
        "source": {
          "observation_id": -5460837407236178041,
          "source": "applog",
          "pos": 1,
          "line": "[rank 0] RuntimeError: CUDA out of memory",
          "rule": "cudaError out of memory",
          "time": null
        }
      }
    ]
  },
  "observation_context": [
    -5460837407236178041
  ],
  "collectors": {
    "nccl_version": "NA",
    "cuda_version": "NA",
    "watchdog_error_time": null,
    "watchdog_timeout_seconds": null,
    "runtime_args": "{}",
    "bad_pairs": "",
    "nvrx_straggler": {},
    "megatron_straggler": {},
    "xid_pci_mappings": "{}",
    "recovery_actions": "",
    "investigatory_actions": ""
  },
  "model_type": "AttributionResponseV2"
}

There is a lot in that response, so let's go through the parts that matter. The verdict is the answer. The root_symptom shows that the log line matched the rule cudaError out of memory, and it points back at the observation, the log source and the exact line that triggered it. The attribution_outcome is Fail User and the failure_domains list contains only User. In plain terms, Attribution is telling us this job failed because of something in the job itself and not because of the node it ran on. There is no reason to cordon node-a or open a hardware ticket for this one.

The symptoms section ties the symptom to the specific node and marks it bad, and the collectors section holds extra context that Attribution can gather for a job, such as NCCL and CUDA versions, watchdog timeouts and straggler data. Those are mostly NA or empty here since we only gave it one log line and there was no real job to collect from.

Save the Result to the Dashboard

This next part tripped me up a little at first, so it is worth calling out. The API calls we just made return the attribution result, but they do not populate the web dashboard's SQLite database. The Blueprint CLI normally handles that persistence. Since we are driving the API by hand with curl, I am going to do the same thing the CLI would do, in a few small steps.

First I save the workload definition to a file in /tmp. It is the same workload we posted earlier, just without the metadata block.

 $ cat > /tmp/attribution-workload.json <<'JSON'
  {
    "workload": {
      "id": "curl-example-001",
      "type": "kueue",
      "job_start_time": "2026-10-07T10:00:00Z",
      "job_end_time": "2026-10-07T10:30:00Z",
      "interest_start_time": "2026-10-07T10:00:00Z",
      "interest_end_time": "2026-10-07T10:30:00Z",
      "timezone": "UTC",
      "status": "FAILED",
      "nodes": {
        "node-a": [0, 1]
      },
      "name": "curl-example",
      "username": "example-user",
      "framework": "pytorch"
    }
  }
JSON

Then I fetch the attribution result for our attributor and write it to a file.

$ export SVC="http://192.168.0.111:30496"
$ curl --fail-with-body -sS "${SVC}/v2/attributor/${ATT}/attribution" -o attribution-result.json

With both pieces on disk, jq merges them into a single dashboard record that holds the workload and its result side by side.

$ jq -n --slurpfile request /tmp/attribution-workload.json --slurpfile result attribution-result.json '{workload: $request[0].workload, result: $result[0]}' > attribution-dashboard-record.json

The last step is writing that record into the database. I do this by running a short Python snippet inside the running Attribution pod with kubectl exec, so it uses the same code and the same /data/attributions.db file on the persistent volume that the dashboard reads. The snippet loads the record from standard input, checks that it really contains a verdict, builds the workload context the writer expects (ID, nodes, start and end times, workload type, state and so on), and uses the blueprint's SQLiteOutput writer to save it.

$  kubectl -n attribution exec -i deployment/attribution -- \
    /opt/blueprint/.venv/bin/python -c '
  import json
  import sys
  from datetime import datetime
  from attribution_blueprint.connectors.outputs.sqlite import SQLiteOutput
  from attribution_blueprint.result import enrich_attribution_result

  record = json.load(sys.stdin)
  w = record["workload"]
  result = record["result"]

  if not isinstance(result.get("verdict"), dict):
      raise ValueError("Input does not contain an attribution verdict")

  context = {
      "id": w["id"],
      "nodes": list(w["nodes"]),
      "start_time": datetime.fromisoformat(w["job_start_time"]),
      "end_time": datetime.fromisoformat(w["job_end_time"]),
      "timezone": w.get("timezone", "UTC"),
      "workload_type": w["type"],
      "state": w["status"],
      "name": w.get("name", ""),
      "user_name": w.get("username", ""),
      "workload_framework": w.get("framework", ""),
  }

  writer = SQLiteOutput("/data/attributions.db")
  try:
      writer.write(context, enrich_attribution_result(result, []))
  finally:
      writer.close()

  print("Saved dashboard record:", w["id"])
  ' < attribution-dashboard-record.json
Saved dashboard record: curl-example-001

The pod confirms with Saved dashboard record: curl-example-001. At this point the record lives in the database the dashboard is reading from, so the failed job should now be visible in the web UI on node port 30080.

Clean Up

Since this was only a demo, the last step is deleting the attributor we created. A zero exit code tells us the delete went through.

$ curl --fail-with-body -sS -X DELETE "${SVC}/v2/attributor/${ATT}"
$ echo $?
0

Summary

In this walkthrough we built the NVIDIA Attribution Blueprint image for arm64, pushed it to a local registry, and deployed it to MicroK8s with a namespace, a persistent volume, a single replica Deployment and a NodePort service. After validating the health endpoint and the web dashboard, we created an attributor for a simulated failed PyTorch job, submitted a CUDA out of memory log line as an observation, and asked for the verdict. Attribution traced the log line to a matching rule and concluded the failure belonged to the user and not the infrastructure. We finished by saving the result into the dashboard's SQLite database through the pod and cleaning up the attributor.

The part I like most is how small the contract is. A workload description plus a bit of evidence goes in, and a structured verdict with the rule, the line and the failure domain comes out. That is exactly what we want when you are trying to decide whether a failed job needs a user to fix their code or an operator to fix a node.

The Attribution project lives on GitHub if you want to try this yourself or dig into the rulesets: https://github.com/dsx-ai-factory/attribution

Hopefully this provides a clear example of how to deploy Attribution and walk a failed workload through to a verdict.