Friday, November 20, 2026

Who Owns GPU Scheduling and Allocation?

Once the pieces of an AI Factory start to come together, one question shows up in almost every customer conversation: who owns GPU scheduling and allocation? Is it the provider that owns the hardware, the tenant that rents a slice of it, or the workload platform sitting on top of the cluster? The answer people hope for is a single name, and the honest answer is that "scheduling" is really three or four decisions made at different layers by different parties. A customer who hears "the provider" and a provider who hears "the tenant" can both walk away thinking the other side has it covered.

I went looking through the dsx-ai-factory repositories to see where each of those decisions actually lives. The picture that came out of it is less a chain of command and more a set of hand-offs. The goal of this post is to lay those hand-offs out by owner, so that a provider, a tenant or a platform team can see which decisions are theirs and which belong to someone else. Most of what follows comes from the NICo tenant management guide, the NCP Software Reference Guide and the Run:ai and Topograph repositories, and I will call out where something is a design intent and not an implemented feature.

Challenge: Two Words, Four Decisions

When someone says "GPU scheduling" they could mean any of the following, and each one has a different owner. The first is capacity: how many machines of which type a tenant is allowed to consume at all. The second is placement: which rack or NVLink domain those machines come from. The third is the cluster layer: how many nodes a cluster gets and how a pod ends up with a specific GPU. The fourth is job scheduling: which team's training job runs next, who gets preempted and who gets a fair share.

Mixing these up is where the confusion starts. A provider might believe it has handed scheduling to the tenant because the tenant runs Kubernetes, while the tenant assumes capacity limits are enforced somewhere below it. Neither is wrong, because both are talking about different layers. Let's walk through them by owner.

Provider: Who Gets What Hardware

The provider, or operator, owns everything before a tenant ever touches a node. In NVIDIA Infra Controller (NICo) that starts with a tenant account and an allocation. The tenant management guide is direct about this: without at least one allocation, a tenant cannot create instances, VPCs, VPC Prefixes or subnets. Allocations are created by someone holding the Provider Admin role, and each one ties a provider, a tenant and a site to exactly one allocation constraint.

For compute, that constraint is an instance type and a count. The docs show the command line form, and the example is simple enough to read at a glance.

nico-admin-cli compute-allocation create --tenant-organization-id bjs34511233a --instance-type-id DGX-H100-640GB --count 8

The REST flavor uses nicocli and a small JSON body, where the constraint type is Reserved and the constraint value is the machine count.

{ "name": "acme-gpu-pool", "tenantId": "<tenant-uuid>", "siteId": "<site-uuid>", "allocationConstraints": [ { "resourceType": "InstanceType", "resourceTypeId": "<instance-type-uuid>", "constraintType": "Reserved", "constraintValue": 8 } ] }

The system validates that enough machines of that instance type are available before it accepts the allocation. That is the provider's capacity grant, and it is the strongest ownership line in the whole stack.

There are two things worth knowing here. First, Reserved is the only constraint type implemented. The API validator also accepts OnDemand and Preemptible, but the documentation says those are not implemented end-to-end, so there is no oversubscription or preemption at the provider layer. Second, placement also sits with the provider. The Software Reference Guide puts placement inside the compute service's machine lifecycle manager, and the NVIDIA cloud requirements ask for NVLink-aware allocation on NVL72 systems and for reserving a whole topology block as a single unit. The provider decides where the machines come from, and the tenant generally does not.

Tenant: How to Carve Up the Share

Once an allocation exists, the tenant owns what happens inside it. In the NVIDIA Kubernetes Engine (NVKE) model the tenant, or an operator acting on the tenant's behalf, names an instance type and a node count, and NVKE validates the request and asks NICo for the machines. The tenant never touches the infrastructure layer directly. The tenant can scale a node pool up or down, change labels and taints, and decide when to move to a new node image, all inside the capacity the provider granted.

Inside the cluster, the GPU-to-pod decision belongs to the Kubernetes control plane. The Software Reference Guide calls for Kubernetes 1.34 or later so that Dynamic Resource Allocation (DRA) is available, and the cloud requirements state that DRA must be enabled. The drivers and operators that make GPUs allocatable, such as the GPU Operator and the DRA driver, are delivered through an AI Cluster Runtime (AICR) recipe. Those recipes ship golden configurations, and the tenant can override them. So even at this layer, ownership depends on whether the tenant changed the defaults.

There are limits on how much control NVKE itself gives. NVKE has no quota object today, and it does no authorization of its own, so any caller the API admits can call every endpoint. Capacity limits come from the NICo allocation and from whatever service layer the operator builds above the NVKE API. That puts a quiet responsibility on the provider: the tenant's freedom to scale is only as bounded as the shim in front of the API.

Workload Platform: Who Runs Which Job

The third owner is the workload platform, and this is the layer most people picture when they hear the word scheduling. Run:ai is the clearest example. It runs as a scheduling and orchestration layer on top of Kubernetes, ships the KAI Scheduler, and adds fair-share, preemption, bin-packing, fractional GPUs and quota hierarchies across departments and projects. The Run:ai documentation is explicit that it does not replace Kubernetes, so it complements the control plane scheduler and does not take its place.

Run:ai is not the only option. The Software Reference Guide describes Slurm as the choice for single-tenant HPC and large-scale training, and NVIDIA Cloud Functions (NVCF) as serverless inference with dedicated multitenancy, where function traffic is routed to GPU clusters that register through the NVIDIA Cluster Agent. Kueue and Grove show up as additional scheduling building blocks in the AICR component set. The guide's own advice is that providers should select based on their tenant requirements, which is a polite way of saying there is no default.

Topology: The Input Schedulers Depend On

Topograph deserves more than a passing mention because it ties the physical fabric to the scheduling decisions above it. It does not schedule anything. Its job is to discover how a cluster is wired and hand that to whichever scheduler is in use. That matters because where a job lands changes how fast it runs. A distributed training job that spans nodes on opposite sides of the fabric pays for it in communication latency, and disaggregated inference, which splits prefill and decode into separate pieces that exchange a KV cache, is sensitive to placement in a similar way. A scheduler can only make a good choice if something has told it what the fabric looks like.

Topograph works in two halves. A provider queries a topology source, such as a cloud provider API, NVIDIA NetQ, InfiniBand discovery, Nautobot or DRA labels, and returns one canonical graph in which tier 0 is the switch closest to the node. An engine then translates that graph into whatever a given scheduler expects. For Slurm that is a topology.conf in tree or block form, and for Slinky it is a ConfigMap. For Kubernetes it is a set of node labels, and the docs show what a node ends up with.

accelerator.topograph.run/domain: nvl1 accelerator.topograph.run/sub-domain: nvl1.rack01 fabric.topograph.run/tier-0: s1 fabric.topograph.run/tier-1: s2 fabric.topograph.run/tier-2: s3

That example is a node in NVLink domain nvl1 and sub-domain nvl1.rack01, connected to switch s1, which connects to s2 and then s3. The label names are configurable through the Helm chart, except for the sub-domain key.

Labels do nothing until a scheduler reads them. The Topograph docs name the KAI Scheduler and Kueue with Topology Aware Scheduling as the consumers. The tas-lab demo in the repository shows two schedulers reading the same labels: kube-scheduler, with a gang whose pods share a constraint on tier-0 so the whole job lands in one rack, and Kueue, with topology levels of tier-1, tier-0 and hostname. I have only read the demo and not run it, and it uses simulated nodes on a kind cluster, so it shows the mechanism and not behavior on a real fabric.

There is also a related Kubernetes feature worth keeping straight. The kubelet Topology Manager aligns CPU, GPU and NIC allocations to the same NUMA domain inside a single node, while Topograph covers the space between nodes. The docs say both can be active at the same time, one handling placement across the cluster and the other handling alignment inside each node.

For ownership, this splits cleanly. The provider knows the fabric, so the provider installs Topograph and feeds it a topology source. The scheduler owner, which may be the provider in a managed platform or the tenant when it runs its own scheduler, decides what to do with the labels, such as which tier a gang has to fit inside. One gap is worth stating plainly: the NVKE requirements list the topology source for workload placement as an open decision, and I did not find anything in NVKE that consumes Topograph today.

Putting the Owners Side by Side

The table below is the short version of everything above.

Decision Typical owner Where it lives
How many machines of which type a tenant may use Provider NICo compute allocation, Reserved constraint
Which rack or NVLink domain machines come from Provider NICo machine lifecycle and placement
Cluster and node pool size within the grant Tenant, or the operator on its behalf NVKE, calling NICo
Which pod gets which GPU in a cluster Kubernetes control plane kube-scheduler with DRA, configured by an AICR recipe
Queues, fair share, gang scheduling and preemption Workload platform Run:ai with KAI Scheduler, Kueue, Slurm or NVCF
Giving schedulers the physical layout Provider installs, schedulers consume Topograph

One more wrinkle is that the word tenant is used at two levels. In NICo, a tenant is the provider's customer organization. In Run:ai, quotas go to departments and projects that live inside that tenant. When a customer asks who owns quota, the useful follow-up is which level they mean, because the provider sets one and the tenant often sets the other.

Dependent on Business Model

This is where the "who owns it" question stops having a single answer. The same components produce different ownership depending on what the provider is selling, and I think three models cover most of the conversations.

In the bare metal model, the provider sells machines and stops at the allocation. Everything above that line, including the Kubernetes install, DRA, the scheduler and any queuing system, belongs to the tenant. This gives the tenant the most control and the provider the smallest operational surface.

In the managed Kubernetes model, the provider runs the control plane and decides how the cluster is configured. NVKE fits here, with the provider owning the control plane, the scheduler configuration and the capacity guardrails, while the tenant owns the workloads and any overrides it makes to the runtime defaults. Ownership of scheduling is genuinely shared, so the shared responsibility table needs to be written down.

In the managed AI platform model, the provider operates Run:ai, NVCF or Slurm and tenants submit jobs. Here the provider owns job-level scheduling, fair share and quota policy as part of the service, and the tenant mostly owns the workload definitions. This is the most to operate and the easiest for a tenant to consume.

None of these however can be considered the absolute right one. A provider with a small team and customers who want raw performance may do well with bare metal. A provider competing on convenience may need the managed AI platform. A provider somewhere in the middle may offer managed Kubernetes and let tenants bring their own scheduler on top. The questions that decide it are mostly about the business: how much operational load the provider wants to carry, how much control tenants expect, whether tenants share a cluster or get dedicated ones, and what the provider is willing to promise in a service level.

Checking the Assumptions

Before settling on a model, it is worth checking a few assumptions against what the components actually do today. We can verify this by asking a short list of questions of any design. Does every layer have a named owner, including quota and placement? Is the reserved-only allocation behavior acceptable, or does the business need oversubscription that the provider layer does not offer? If tenants share a cluster, who sets department and project quotas, and in which system? And who is accountable when a faulty node gets pulled from service and capacity quietly shrinks?

At this point I feel pretty confident about the main takeaway. The components divide the work cleanly enough, but the ownership of each hand-off is a decision for people and not for software.

Hopefully this gave a clearer view of how allocation and scheduling responsibility is spread across the DSX stack and why the answer depends on the business model. The NICo tenant management guide, the NCP Software Reference Guide and the Run:ai and Topograph repositories are the best places to go deeper, and each component's documentation is the right source for its current capabilities and supported use.