In the first post I covered what Private AI Services 3.0 changes on VCF 9.1.1. This one is the part of the talk that happens under the hood. A model endpoint carries two things: a model, and a VM class. Everything below is about what is inside that VM class.
As before: I work at Broadcom, this is my own reading of public material, and the lab numbers are from consumer hardware.
TL;DR
- There are three kinds of GPU servers: normal PCIe cards, pairs joined by an NVLink bridge, and HGX systems with eight GPUs behind NVSwitch. Know which one you have before you read any option slide.
- Three ways to give a VM a GPU on one host: own the host, own the card, own a slice. More virtualisation buys more vSphere features and more driver and licence coupling.
- One rule covers all of them: whatever you pass through pins the VM to the host.
- Inside the box, NVLink is about ten times faster than any network leaving it. Fill one host before you spread a model across several.
- For several hosts there are five options, but only two decisions. The letter is the GPU, the number is the network.
- Most teams will not start with an HGX system. The everyday case is PCIe cards, GPU passed through, network virtualised, no vGPU licence.
Three kinds of GPU servers
Before any option makes sense, look at the hardware:
- PCIe servers. Normal add-in cards. Each GPU talks to the others over PCIe and the CPU.
- NVLink pairs. Two cards joined by a bridge. The pair exchanges data without touching PCIe.
- HGX systems. Eight GPU modules on one baseboard, fully connected through NVSwitch. NVIDIA’s DGX is its own server around the same board.
The options below behave differently on each of these. Several of them do not exist on HGX at all.
One host: own the host, own the card, own a slice
Option A: own the host. Full passthrough. One VM takes everything: all GPUs, the NVSwitches, the network cards if you need them. Kubernetes inside sees bare metal and runs the full NVIDIA stack in the guest. There is no driver dependency on the host. The price: no vMotion, no DRS, no HA. One cluster owns the whole host.
Option B: own the card. Still passthrough, but vSphere understands it. Two Kubernetes clusters can share one host, each with its own GPUs. A VM class asks for, say, two H100, and DRS finds a host that has them. DRS chooses the seat once. After that, nobody moves. Two limits: one GPU cannot be shared between VMs, and HGX systems are not supported in this mode.
Option C: own a slice. vGPU. The host driver cuts each GPU into profiles, and your VM class catalogue becomes a menu of GPU sizes. On HGX, device groups hand NVLink-connected GPUs to a VM as one unit: one, two, four or eight. You get vMotion and clean host maintenance, as long as you do not need unified memory. The price is coupling. Host driver and guest driver must stay on the same branch, for every tenant, and you need the NVIDIA AI Enterprise licence.
The rule that runs through all three: whatever you pass through pins the VM to the host.
One note on sizing a slice, from my own card. A model that lost twelve percent of its memory to another process dropped from 99 to 56 tokens per second, with no error. Doubling the context window from 64k to 128k dropped it from 98 to 27, because the KV cache grew from four to eight gigabytes and no longer fit. Size the profile for the model plus its context, not for the model file.
Inside the box: NVLink and NVSwitch
One host is not always enough. Before connecting hosts, sixty seconds on how GPUs talk inside one.
NVLink is a private lane between GPUs. Data moves from one GPU to the next without touching PCIe or the CPU. An H100 has eighteen of these links, together about 900 gigabytes per second. That is roughly nine 400-gigabit network ports. Per GPU. Mind the units: NVLink is quoted in gigabytes, networks in gigabits.
NVSwitch is the junction. Without it, each GPU divides its links among its neighbours. With it, every GPU reaches every other at full speed. Eight cards with 80 gigabytes each then behave almost like one device with 640 gigabytes. That is how a model too large for one card still runs fast.
Three things to remember:
- Someone manages that fabric. With vGPU, the ESX host does it. With passthrough, your VM does it.
- If you partition a GPU with MIG, NVLink is off. Time-sliced profiles keep it.
- The inside of the box is about ten times faster than any network leaving it. Fill one host before you spread a model across several.
Several hosts: the letter is the GPU, the number is the network
Five options, two decisions. Do you pass the GPU through or use vGPU? That is the letter, A or B. Do you pass the network card through or virtualise it? That is the number, 1 or 2.
| GPU passed through (A) | vGPU (B) | |
|---|---|---|
| Physical NIC (1) | A-1: bare metal with a vSphere badge. GPUs of two hosts exchange data directly over RDMA. No vMotion. If the host fails, the VM is gone. For one tenant doing distributed training or very large inference. | B-1: simple guest, one vGPU profile, host handles NVLink below the VM. NVIDIA AI Enterprise licence, matching drivers on host and guest. For when your NICs or your VCF version cannot do the virtualised variant yet. |
| Virtual NIC (2) | A-2: same GPUs, NIC virtualised inside a VPC. You keep RDMA and the workload sits in a normal tenant network. For providers who hand whole GPU hosts to tenants and re-wire the network when a host changes hands. | B-2: both virtualised. DRS places the workload. vMotion stays, as long as you do not use unified memory or NVLink between GPUs. If you ask me where this is going: here. |
And the option that does not fit the grid, which is really the everyday case. Option C: no HGX, no NVSwitch, normal servers with PCIe cards. Several GPU VMs per host. The GPU is passed through, the network card is virtualised. No vGPU licence. Dense, cost-efficient model serving. Most teams start here, and it is a good place to start.
The details that bite when you build it
- RDMA needs a lossless network. Either InfiniBand, which is lossless by design, or RoCE on Ethernet with priority flow control and ECN configured. That is a job for your network team, not a vSphere setting.
- GPU and NIC on the same NUMA node. Direct NIC-to-GPU transfers only work when both devices sit under the same PCIe root complex. HGX boards pair them for this reason. The VM has to reproduce that pairing.
- At most 128 vCPUs for GPUDirect VMs. Beyond that, a VM needs a virtual IOMMU, and that must be off here.
- Do not “disable ACS”. ACS stays enabled in ESX. Peer-to-peer is allowed per VM class, in the VM’s advanced settings.
- “No MIG” in option C is about the option, not the card. It hands out whole cards. The RTX PRO 6000 Blackwell Server Edition does support MIG.
What I would decide first
- Which of the three server kinds do you have, or are you buying? That removes half the options.
- Do you need vMotion for GPU VMs? If yes, you are in vGPU territory, with the licence and the driver coupling that comes with it.
- Does one model fit one host? If yes, stop at NVLink. Multi-host RDMA is a network project before it is a GPU project.
- Pass-through or vGPU, physical or virtual NIC. Pick the letter and the number knowing what each one pins.
Watch the original
The full GPU block, with every option diagram, is in the Las Vegas recording CLOB1084LV: Design & Architect: VMware Cloud Foundation Private AI Services. William Lam’s vmware-explore-2026-session-urls lists the other sessions.
Sources
- VMware Cloud Foundation 9.1 Design Library, Private AI section, Broadcom TechDocs
- Broadcom white paper on GPUDirect RDMA on vSphere, September 2025
- Broadcom KB 313255 on virtual IOMMU and VMs with more than 128 vCPUs
- NVIDIA AI Enterprise 8 documentation, supported vGPU types