When a GPU job dies in the middle of the night, the first question is almost never "what happened?" It is "whose problem is this?" Is the node bad and does it need to be pulled out of rotation, or did someone submit a job that asked for more memory than the GPU has? NVIDIA Attribution is a service built to answer that question. It starts with a described workload, hand it the evidence collected (log lines from the nodes the job ran on, for example), and it matches that evidence against a ruleset to produce a verdict: the root symptom, the outcome and the failure domain it belongs to.
In this post I am going to walk through a small end to end demo of Attribution running on a single virtual machine. We will build the Attribution Blueprint container image, push it to a registry, deploy it into a MicroK8s cluster, and then call the API by hand with a simulated failed PyTorch job. The failure here will be a CUDA out of memory error, which makes for a nice example because the right answer is that the user owns the problem and the hardware is fine. I should be upfront that the workload here is made up and so is the single log line. It is a way to see the moving pieces, not a real incident. With all that said let's get started together on our journey.
Build and Push the Blueprint Image
The first step is building the container image from the Attribution repository. The virtual machine is an arm64 system so I am passing --platform linux/arm64, and I am tagging the image for the registry that runs in my lab environment on port 5000. This is a multi-stage build. One stage uses a Node 22 image to compile the web dashboard frontend with pnpm and Vite, and the final stage is Python 3.14 on Debian trixie where uv installs the blueprint, the service app and the attribution library. The build also installs gcc just long enough to compile a few dependencies and then purges it again to keep the image small.
$ docker build --platform linux/arm64 -f blueprint/docker/Dockerfile -t registry.schmaustech.com:5000/attribution-blueprint-web:latest .
[+] Building 1.4s (32/32) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 3.07kB 0.0s
=> resolve image config for docker-image://docker.io/docker/dockerfile:1.7 0.4s
=> CACHED docker-image://docker.io/docker/dockerfile:1.7@sha256:a57df69d0ea827fb7266491f2813635de6f17269be881f696fbfdf2d83dda33e 0.0s
=> [internal] load metadata for ghcr.io/astral-sh/uv:0.10.12 0.4s
=> [internal] load metadata for docker.io/library/node:22-bookworm-slim 0.6s
=> [internal] load metadata for docker.io/library/python:3.14.6-slim-trixie 0.7s
=> [internal] load build context 0.0s
=> => transferring context: 60.10kB 0.0s
=> [web-build 1/4] FROM docker.io/library/node:22-bookworm-slim@sha256:c3de60bf2f9dd0ac6370e6117950ff62d6e339527e7472301c9c78a017978392 0.0s
=> FROM ghcr.io/astral-sh/uv:0.10.12@sha256:72ab0aeb448090480ccabb99fb5f52b0dc3c71923bffb5e2e26517a1c27b7fec 0.0s
=> [stage-1 1/19] FROM docker.io/library/python:3.14.6-slim-trixie@sha256:7bec7ddcddeff7975d6ba9b4be7dd6f6b2f55e7491539145e2978f7f97ce9144 0.0s
=> CACHED [stage-1 2/19] COPY --from=ghcr.io/astral-sh/uv:0.10.12 /uv /uvx /bin/ 0.0s
=> CACHED [stage-1 3/19] WORKDIR /opt/blueprint 0.0s
=> CACHED [stage-1 4/19] COPY blueprint/pyproject.toml blueprint/uv.lock ./ 0.0s
=> CACHED [stage-1 5/19] COPY blueprint/src/ ./src/ 0.0s
=> CACHED [web-build 2/4] WORKDIR /frontend 0.0s
=> CACHED [web-build 3/4] COPY blueprint/src/attribution_blueprint/webui/frontend/ ./ 0.0s
=> CACHED [web-build 4/4] RUN corepack enable && pnpm install --frozen-lockfile && pnpm exec tsc -b && pnpm exec vite build --outDir dist 0.0s
=> CACHED [stage-1 6/19] COPY --from=web-build /frontend/dist ./src/attribution_blueprint/webui/static/ 0.0s
=> CACHED [stage-1 7/19] COPY service/service-app/nvidia_attribution_client/ /opt/service/service-app/nvidia_attribution_client/ 0.0s
=> CACHED [stage-1 8/19] RUN uv sync --extra web --no-dev --no-editable --frozen 0.0s
=> CACHED [stage-1 9/19] WORKDIR /opt/attribution-service 0.0s
=> CACHED [stage-1 10/19] COPY service/attribution-lib/ /opt/attribution-lib/ 0.0s
=> CACHED [stage-1 11/19] COPY service/service-app/pyproject.toml service/service-app/uv.lock ./ 0.0s
=> CACHED [stage-1 12/19] COPY service/service-app/_build_version.py ./ 0.0s
=> CACHED [stage-1 13/19] COPY service/service-app/README.md service/service-app/LICENSE ./ 0.0s
=> CACHED [stage-1 14/19] COPY service/service-app/app/ ./app/ 0.0s
=> CACHED [stage-1 15/19] COPY service/service-app/nvidia_attribution_client/ ./nvidia_attribution_client/ 0.0s
=> CACHED [stage-1 16/19] RUN apt-get update && apt-get install -y --no-install-recommends gcc libc6-dev && uv sync --no-dev --no-editable --frozen && apt-get purge -y --auto-remove gcc libc6-dev && rm -rf /var/lib/ap 0.0s
=> CACHED [stage-1 17/19] COPY blueprint/docker/start.sh /usr/local/bin/attribution-blueprint-start 0.0s
=> CACHED [stage-1 18/19] RUN chmod +x /usr/local/bin/attribution-blueprint-start 0.0s
=> CACHED [stage-1 19/19] WORKDIR /opt/blueprint 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:7417910ee0d5bc1d35e8030e63996939c804eff310e63c9209131f4cc443df96 0.0s
=> => naming to registry.schmaustech.com:5000/attribution-blueprint-web:latest
The above output we can observe almost every step says CACHED and the whole build finished in a little over a second. That is only because I had built this image before, so on a first run expect it to take considerably longer while the frontend and Python dependencies are pulled and built. At this point we have an image sitting in the local Docker image store, so let's push it to the registry so the cluster can pull it.
$ docker push registry.schmaustech.com:5000/attribution-blueprint-web:latest
The push refers to repository [registry.schmaustech.com:5000/attribution-blueprint-web]
5f70bf18a086: Pushed
b78e1eeee5c9: Pushed
e5c73dd8dfa2: Pushed
bd7bce39331b: Pushed
66a7a0b9edd3: Pushed
6a52fb04e724: Pushed
87fa1d674280: Pushed
199eb863db71: Pushed
9c5d441c4e95: Pushed
fe962dda67f4: Pushed
56f16490e6f3: Pushed
5b91b81ea413: Pushed
740cf88824b2: Pushed
2a56874aabc8: Pushed
a4a5b6e3a44e: Pushed
d142ef99965e: Pushed
c0c0660a42bc: Pushed
740399ee44b6: Pushed
5f35d866a418: Pushed
2e68013aee3d: Pushed
86d005034ff8: Pushed
c01c35a040a2: Pushed
local: digest: sha256:7c387a2c2750c7e780f9ce6d397c97ddc6193da287daf0827e69358753e84119 size: 4923
Twenty-two layers pushed and a digest at the bottom. If everything looks good we can move onto deploying it.
Deploy Attribution to Kubernetes
For the deployment I always prefer yaml manifests, so the following is a single file that holds everything Attribution needs. Let's look at it and then go through what each object does.
$ cat attribution-deployment.yaml
apiVersion: v1
kind: Namespace
metadata:
name: attribution
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: attribution-data
namespace: attribution
spec:
accessModes:
- ReadWriteOnce
storageClassName: local-path-persistent
resources:
requests:
storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: attribution
namespace: attribution
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: attribution
template:
metadata:
labels:
app: attribution
spec:
containers:
- name: attribution
image: registry.schmaustech.com:5000/attribution-blueprint-web:latest
imagePullPolicy: Always
env:
- name: ATTRIBUTION_DB
value: /data/attributions.db
- name: ATTRIBUTION_SERVICE_PORT
value: "8000"
- name: ATTRIBUTION_WEB_PORT
value: "8080"
ports:
- name: api
containerPort: 8000
- name: web
containerPort: 8080
startupProbe:
httpGet:
path: /
port: web
periodSeconds: 5
failureThreshold: 36
readinessProbe:
httpGet:
path: /
port: web
periodSeconds: 10
livenessProbe:
httpGet:
path: /v2/healthz
port: api
periodSeconds: 20
failureThreshold: 3
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
memory: 1Gi
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: attribution-data
---
apiVersion: v1
kind: Service
metadata:
name: attribution
namespace: attribution
spec:
type: NodePort
selector:
app: attribution
ports:
- name: web
port: 8080
targetPort: web
nodePort: 30080
- name: api
port: 8000
targetPort: api
The file starts with an attribution namespace and a one gigabyte PersistentVolumeClaim named attribution-data. That claim is what holds the SQLite database the web dashboard reads from, and I am using the local-path-persistent storage class since this is a single node lab. The Deployment runs one replica and uses the Recreate strategy rather than a rolling update. Since the volume is ReadWriteOnce and SQLite does not like two writers, we want the old pod gone before the new one starts.
The container exposes two ports: 8000 for the Attribution API and 8080 for the web dashboard, and the environment variables point the database at /data/attributions.db on the mounted volume. The probes are worth a quick look. The startup and readiness probes hit the web UI on /, while the liveness probe hits /v2/healthz on the API port. The startup probe gives the pod up to three minutes (5 second period times 36 failures) to come up, which is plenty of time for the first start. Finally the Service is a NodePort, with the web UI pinned to 30080 and the API port left for Kubernetes to assign.
With the manifest ready we can create everything efficiently in one shot.
$ kubectl create -f attribution-deployment.yaml
namespace/attribution created
persistentvolumeclaim/attribution-data created
deployment.apps/attribution created
service/attribution created
Now let's see if the pod comes up. Below we are asking for the pods, the claim and the service all at once so we can confirm each piece.
$ kubectl -n attribution get pods,pvc,service -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/attribution-75f57d76d9-rw592 1/1 Running 0 45s 10.1.174.232 gpu-node <none> <none>
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE VOLUMEMODE
persistentvolumeclaim/attribution-data Bound pvc-9e497dc7-2b0f-4cef-bb8e-e0d074241f75 1Gi RWO local-path-persistent <unset> 45s Filesystem
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/attribution NodePort 10.152.183.26 <none> 8080:30080/TCP,8000:30496/TCP 45s app=attribution
The pod is Running and 1/1 ready, the claim is Bound to a one gigabyte volume, and the service shows the web UI on 8080 mapped to node port 30080 and the API on 8000 mapped to node port 30496. We will use both of those node ports in a moment. A plain pod listing a bit later confirms it has stayed up with no restarts.
$ kubectl get pods -n attribution
NAME READY STATUS RESTARTS AGE
attribution-75f57d76d9-rw592 1/1 Running 0 74s
Validate the Service
Before sending any real requests I want to confirm both ports answer. We can verify this by calling the health endpoint on the API node port, using the IP address of the virtual machine.
$ curl --max-time 10 -fsS http://192.168.0.111:30496/v2/healthz
null
A bare null looks odd, but it is what the health endpoint returns on success. What matters is that curl was run with -f, so it would have failed loudly on any HTTP error. Next up is the web dashboard, where I only care about the status code.
$ curl --max-time 10 -fsS -o /dev/null -w 'Web UI HTTP %{http_code}\n' http://192.168.0.111:30080/
Web UI HTTP 200
An HTTP 200 from the dashboard. At this point I feel pretty confident the deployment is healthy. To save some typing for the rest of the demo I am exporting the API address into a variable and calling the health endpoint once more through it.
$ export SVC="http://192.168.0.111:30496"
$ curl --fail-with-body -sS "${SVC}/v2/healthz"
Create an Attributor and Submit an Observation
Now for the interesting part. Everything in the Attribution API hangs off an attributor, which is how I think about it: a record of one workload you want an answer about. To create one we POST to /v2/attributor with a description of the workload and some metadata about where it ran.
The workload here is a made up PyTorch job named curl-example, submitted through Kueue, that ran for thirty minutes on GPUs 0 and 1 of a node called node-a and ended in a FAILED status. The metadata identifies the cluster, the agent that reported it, a tenant and the ruleset to use, which is default. I am piping the response through jq to pull out just the attributor_id and store it in a variable called ATT so we can reuse it.
$ export ATT="$(
curl --fail-with-body -sS \
-X POST \
-H 'Content-Type: application/json' \
"${SVC}/v2/attributor" \
--data-binary @- <<'JSON' | jq -er '.attributor_id'
{
"workload": {
"id": "curl-example-001",
"type": "kueue",
"job_start_time": "2026-10-07T10:00:00Z",
"job_end_time": "2026-10-07T10:30:00Z",
"interest_start_time": "2026-10-07T10:00:00Z",
"interest_end_time": "2026-10-07T10:30:00Z",
"timezone": "UTC",
"status": "FAILED",
"nodes": {
"node-a": [0, 1]
},
"name": "curl-example",
"username": "example-user",
"framework": "pytorch"
},
"metadata": {
"cluster": "local-microk8s",
"agent": "curl",
"tenant": "example-tenant",
"ruleset": "default"
}
}
JSON
)"
$
1094296642135612619
That large number at the bottom is the attributor ID. The attributor knows about the workload but it has not seen any evidence yet, so let's give it some. Evidence is submitted as an observation against the attributor. The observation has a context saying it came from an application log (applog), and a body that carries the log line along with the resource it came from. In this case that is a single line, [rank 0] RuntimeError: CUDA out of memory, from node-a. The body is itself a JSON string, which is why you see all of the escaped quotes.
$ curl --fail-with-body -sS \
-X POST \
-H 'Content-Type: application/json' \
"${SVC}/v2/attributor/${ATT}/observation" \
--data-binary @- <<'JSON'
{
"context": {
"time_interval": {
"start": null,
"end": null
},
"source": "applog"
},
"body": "[{\"body\":\"[rank 0] RuntimeError: CUDA out of memory\",\"resources\":[{\"kind\":\"node\",\"hostname\":\"node-a\"}]}]"
}
JSON
{"observation_id":-5460837407236178041,"model_type":"ObservationResponseV2"
The API hands back an observation ID, which tells us the line was accepted and recorded against our attributor.
Read the Verdict
With the evidence in place we can ask Attribution what it thinks happened.
$ curl --fail-with-body -sS "${SVC}/v2/attributor/${ATT}/attribution" | jq .
{
"verdict": {
"root_symptom": {
"symptom": "cudaError out of memory",
"source": {
"observation_id": -5460837407236178041,
"source": "applog",
"pos": 1,
"line": "[rank 0] RuntimeError: CUDA out of memory",
"rule": "cudaError out of memory",
"time": null
},
"catalog_id": "cudaerror-out-of-memory"
},
"attribution_outcome": "Fail User",
"detail": "Not Applicable",
"terminal_states": [
"Fail"
],
"failure_domains": [
"User"
],
"cancellation_actors": [],
"uncertainty_type": null
},
"symptoms": {
"cudaError out of memory": [
{
"type": "NodeSource",
"node": "node-a",
"status": "bad",
"source": {
"observation_id": -5460837407236178041,
"source": "applog",
"pos": 1,
"line": "[rank 0] RuntimeError: CUDA out of memory",
"rule": "cudaError out of memory",
"time": null
}
}
]
},
"observation_context": [
-5460837407236178041
],
"collectors": {
"nccl_version": "NA",
"cuda_version": "NA",
"watchdog_error_time": null,
"watchdog_timeout_seconds": null,
"runtime_args": "{}",
"bad_pairs": "",
"nvrx_straggler": {},
"megatron_straggler": {},
"xid_pci_mappings": "{}",
"recovery_actions": "",
"investigatory_actions": ""
},
"model_type": "AttributionResponseV2"
}
There is a lot in that response, so let's go through the parts that matter. The verdict is the answer. The root_symptom shows that the log line matched the rule cudaError out of memory, and it points back at the observation, the log source and the exact line that triggered it. The attribution_outcome is Fail User and the failure_domains list contains only User. In plain terms, Attribution is telling us this job failed because of something in the job itself and not because of the node it ran on. There is no reason to cordon node-a or open a hardware ticket for this one.
The symptoms section ties the symptom to the specific node and marks it bad, and the collectors section holds extra context that Attribution can gather for a job, such as NCCL and CUDA versions, watchdog timeouts and straggler data. Those are mostly NA or empty here since we only gave it one log line and there was no real job to collect from.
Save the Result to the Dashboard
This next part tripped me up a little at first, so it is worth calling out. The API calls we just made return the attribution result, but they do not populate the web dashboard's SQLite database. The Blueprint CLI normally handles that persistence. Since we are driving the API by hand with curl, I am going to do the same thing the CLI would do, in a few small steps.
First I save the workload definition to a file in /tmp. It is the same workload we posted earlier, just without the metadata block.
$ cat > /tmp/attribution-workload.json <<'JSON'
{
"workload": {
"id": "curl-example-001",
"type": "kueue",
"job_start_time": "2026-10-07T10:00:00Z",
"job_end_time": "2026-10-07T10:30:00Z",
"interest_start_time": "2026-10-07T10:00:00Z",
"interest_end_time": "2026-10-07T10:30:00Z",
"timezone": "UTC",
"status": "FAILED",
"nodes": {
"node-a": [0, 1]
},
"name": "curl-example",
"username": "example-user",
"framework": "pytorch"
}
}
JSON
Then I fetch the attribution result for our attributor and write it to a file.
$ export SVC="http://192.168.0.111:30496"
$ curl --fail-with-body -sS "${SVC}/v2/attributor/${ATT}/attribution" -o attribution-result.json
With both pieces on disk, jq merges them into a single dashboard record that holds the workload and its result side by side.
$ jq -n --slurpfile request /tmp/attribution-workload.json --slurpfile result attribution-result.json '{workload: $request[0].workload, result: $result[0]}' > attribution-dashboard-record.json
The last step is writing that record into the database. I do this by running a short Python snippet inside the running Attribution pod with kubectl exec, so it uses the same code and the same /data/attributions.db file on the persistent volume that the dashboard reads. The snippet loads the record from standard input, checks that it really contains a verdict, builds the workload context the writer expects (ID, nodes, start and end times, workload type, state and so on), and uses the blueprint's SQLiteOutput writer to save it.
$ kubectl -n attribution exec -i deployment/attribution -- \
/opt/blueprint/.venv/bin/python -c '
import json
import sys
from datetime import datetime
from attribution_blueprint.connectors.outputs.sqlite import SQLiteOutput
from attribution_blueprint.result import enrich_attribution_result
record = json.load(sys.stdin)
w = record["workload"]
result = record["result"]
if not isinstance(result.get("verdict"), dict):
raise ValueError("Input does not contain an attribution verdict")
context = {
"id": w["id"],
"nodes": list(w["nodes"]),
"start_time": datetime.fromisoformat(w["job_start_time"]),
"end_time": datetime.fromisoformat(w["job_end_time"]),
"timezone": w.get("timezone", "UTC"),
"workload_type": w["type"],
"state": w["status"],
"name": w.get("name", ""),
"user_name": w.get("username", ""),
"workload_framework": w.get("framework", ""),
}
writer = SQLiteOutput("/data/attributions.db")
try:
writer.write(context, enrich_attribution_result(result, []))
finally:
writer.close()
print("Saved dashboard record:", w["id"])
' < attribution-dashboard-record.json
Saved dashboard record: curl-example-001
The pod confirms with Saved dashboard record: curl-example-001. At this point the record lives in the database the dashboard is reading from, so the failed job should now be visible in the web UI on node port 30080.
Clean Up
Since this was only a demo, the last step is deleting the attributor we created. A zero exit code tells us the delete went through.
$ curl --fail-with-body -sS -X DELETE "${SVC}/v2/attributor/${ATT}"
$ echo $?
0
Summary
In this walkthrough we built the NVIDIA Attribution Blueprint image for arm64, pushed it to a local registry, and deployed it to MicroK8s with a namespace, a persistent volume, a single replica Deployment and a NodePort service. After validating the health endpoint and the web dashboard, we created an attributor for a simulated failed PyTorch job, submitted a CUDA out of memory log line as an observation, and asked for the verdict. Attribution traced the log line to a matching rule and concluded the failure belonged to the user and not the infrastructure. We finished by saving the result into the dashboard's SQLite database through the pod and cleaning up the attributor.
The part I like most is how small the contract is. A workload description plus a bit of evidence goes in, and a structured verdict with the rule, the line and the failure domain comes out. That is exactly what we want when you are trying to decide whether a failed job needs a user to fix their code or an operator to fix a node.
The Attribution project lives on GitHub if you want to try this yourself or dig into the rulesets: https://github.com/dsx-ai-factory/attribution
Hopefully this provides a clear example of how to deploy Attribution and walk a failed workload through to a verdict.
