GPU cluster fabric: GPU servers on InfiniBand or 400GbE
Training clusters connect GPU servers through a separate high-speed fabric so GPUs in different servers can exchange data directly (RDMA). Large clusters give each GPU its own 200Gb or 400Gb InfiniBand or RoCE Ethernet port, cabled in rails so GPU 1 of every server lands on the same switch. Storage and management run on their own Ethernet networks next to the GPU fabric.
Cabling: OSFP / QSFP112 optics or AOC.
What to check
- InfiniBand adapters and switches must match the generation: HDR is 200Gb, NDR is 400Gb. VPI adapters such as ConnectX-6 and ConnectX-7 can run either InfiniBand or Ethernet.
- A GPU server draws several kilowatts, so rack power and cooling usually limit how many fit in a rack before space does.
- Keep the storage network separate from the GPU fabric; checkpoint writes can saturate it.
Hardware for this setup
Every model and part in the reference that fits this path, by role and brand. Open one to see what it connects to.
Servers (12)
Dell
Dell PowerEdge XE8545 4U rack
Dell PowerEdge XE8640 4U rack
Dell PowerEdge XE9640 2U rack
Dell PowerEdge XE9680 6U rack
HPE
HPE ProLiant DL385 Gen11 2U rack
HPE ProLiant DL580 Gen10 4U rack
HPE ProLiant ML350 Gen10 4U tower
HPE ProLiant ML350 Gen11 4U tower
Lenovo
Lenovo ThinkSystem SR670 V2 3U rack
Lenovo ThinkSystem SR860 V2 4U rack
Lenovo ThinkSystem ST650 V2 tower
Lenovo ThinkSystem ST650 V3 tower
GPUs (19)
AMD
AMD Instinct MI210 PCIe 4.0 dual-slot full-height full-length
AMD Instinct MI250X OAM (OCP Accelerator Module)
AMD Instinct MI300X OAM (OCP Accelerator Module)
NVIDIA
NVIDIA Data Center GPU A10 PCIe single-slot full-height full-length
NVIDIA Data Center GPU A100 PCIe 80GB PCIe dual-slot full-height full-length
NVIDIA Data Center GPU A100 SXM4 80GB SXM4 module
NVIDIA Data Center GPU A30 PCIe dual-slot full-height full-length
NVIDIA Data Center GPU A40 PCIe 4.0 dual-slot full-height full-length
NVIDIA Data Center GPU A800 PCIe 80GB PCIe dual-slot full-height full-length
NVIDIA Data Center GPU H100 PCIe 80GB PCIe 5.0 dual-slot full-height full-length
NVIDIA Data Center GPU H100 SXM5 80GB SXM5 module
NVIDIA Data Center GPU H20 SXM5 module or PCIe
NVIDIA Data Center GPU H800 SXM5 80GB SXM5 module
NVIDIA Data Center GPU L20 PCIe dual-slot full-height full-length
NVIDIA Data Center GPU L4 PCIe 4.0 low-profile half-height half-length
NVIDIA Data Center GPU L40S PCIe 4.0 dual-slot full-height full-length
NVIDIA Data Center GPU T4 PCIe low-profile half-height half-length
NVIDIA Data Center GPU Tesla V100 SXM2 32GB SXM2 module
NVIDIA GeForce RTX 5090D PCIe 5.0 x16 triple-slot
Server adapters (6)
Dell
540-BCVE Dell NVIDIA Mellanox ConnectX-6 VPI Single Port 200Gb/s HDR InfiniBand / 200GbE PCIe Adapter
540-BDDW Dell NVIDIA Mellanox ConnectX-6 VPI Dual Port 200Gb/s HDR InfiniBand / 200GbE PCIe Adapter
540-BDRV Dell NVIDIA Mellanox ConnectX-7 Dual Port 400Gb/s NDR InfiniBand / 400GbE PCIe Gen5 Adapter
NVIDIA
MCX556A-ECAT NVIDIA ConnectX-5 VPI Dual-Port 100Gb/s EDR InfiniBand and 100GbE QSFP28 PCIe 3.0 x16 Adapter
MCX75310AAS-NEAT NVIDIA ConnectX-7 Single-Port 400Gb/s NDR InfiniBand and 400GbE OSFP PCIe 5.0 x16 Adapter
MCX755106AS-HEAT NVIDIA ConnectX-7 Dual-Port 200Gb/s NDR200 InfiniBand and 200GbE QSFP112 PCIe 5.0 x16 Adapter
Plan and size it
- Draw this setup in the topology planner mrplanb.com
- Tokens per kWh calculator for GPU racks rack-density.com
- Add up rack power and space rack-density.com
- Power, cooling and racks for dense GPU servers mrplanb.com
Other ways to connect
- Fibre Channel SAN
- iSCSI and NVMe/TCP over Ethernet
- Direct-attached SAS
- File storage (NFS and SMB)
- Hyperconverged cluster
FAQ
InfiniBand or Ethernet for GPU clusters?
InfiniBand is the established choice for large training clusters. RoCE Ethernet with tuned switches comes close and suits sites that want one network technology. Inference servers usually need only Ethernet.
How many network ports does a GPU server need?
For training, typically one 200Gb or 400Gb port per GPU plus two Ethernet ports for storage and two for management. Inference servers often use two to four Ethernet ports.
Buying or selling GPU cluster hardware?
New, refurbished or used. Tell us the quantity and condition and we connect you with a vetted reseller in your region, usually within one working day.