Home › Works together

iSCSI and NVMe/TCP: block storage over an Ethernet switch

iSCSI and NVMe/TCP carry block storage over ordinary Ethernet, so the same type of switch serves both the data network and the storage network. Servers use two NIC ports at 10, 25 or 100GbE, one to each switch, and the array's Ethernet host ports on both controllers connect the same way. Most sites keep storage on its own VLAN or its own pair of switches, with jumbo frames end to end.

Server10/25/100GbE NICEthernet switchstorage VLAN, two switchesStorage arrayiSCSI or NVMe/TCP ports

Cabling: DAC or optics with fibre.

What to check

Keep the speed the same at all three points: a 25GbE server, a 25GbE switch port and a 25GbE array port. Mixed speeds work but the slowest link sets the pace.
SFP28 ports take 10GbE SFP+ and 25GbE SFP28 optics or DAC cables; QSFP28 ports are 100GbE and can usually be split into four 25GbE links with a breakout cable.
NVMe/TCP needs both the array and the host operating system to support it; iSCSI works almost everywhere.

Hardware for this setup

Every model and part in the reference that fits this path, by role and brand. Open one to see what it connects to.

Servers (117)

Dell

HPE

HPE ProLiant DL325 Gen10 Plus v2 1U rack
HPE ProLiant DL325 Gen11 1U rack
HPE ProLiant DL360 Gen10 1U rack
HPE ProLiant DL360 Gen10 Plus 1U rack
HPE ProLiant DL360 Gen11 1U rack
HPE ProLiant DL380 Gen10 2U rack
HPE ProLiant DL380 Gen10 Plus 2U rack
HPE ProLiant DL380 Gen11 2U rack
HPE ProLiant DL385 Gen10 Plus v2 2U rack
HPE ProLiant DL385 Gen11 2U rack
HPE ProLiant DL560 Gen10 2U rack
HPE ProLiant DL560 Gen11 2U rack
HPE ProLiant DL580 Gen10 4U rack
HPE ProLiant ML MicroServer Gen10 Plus v2 Ultra micro tower
HPE ProLiant ML110 Gen11 4.5U tower
HPE ProLiant ML30 Gen11 4U tower
HPE ProLiant ML350 Gen10 4U tower
HPE ProLiant ML350 Gen11 4U tower

Lenovo

Server adapters (50)

Broadcom

BCM957414A4142CC Broadcom NetXtreme-E BCM57414 Dual-Port 25GbE SFP28 PCIe 3.0 Ethernet Network Adapter
BCM957416A4160C Broadcom NetXtreme-E BCM57416 Dual-Port 10GBASE-T PCIe 3.0 Ethernet Network Adapter
BCM957504A5041CC Broadcom NetXtreme-E BCM57504 Quad-Port 100GbE QSFP56 PCIe 4.0 Ethernet Network Adapter

Dell

540-BBGN Dell Broadcom 57412 Dual Port 10GbE SFP+ PCIe Adapter
540-BBGO Dell Broadcom 57416 Dual Port 10GBASE-T RJ45 PCIe Adapter
540-BBGP Dell Broadcom 57416 Dual Port 10GBASE-T RJ45 OCP 3.0 Adapter
540-BBGY Dell Broadcom 57414 Dual Port 10/25GbE SFP28 PCIe Low Profile Network Adapter
540-BBHB Dell Broadcom 57414 Dual Port 10/25GbE SFP28 OCP 3.0 Mezzanine Network Adapter
540-BBHG Dell Intel X520/I350 2x10GbE SFP+ + 2x1GbE BASE-T Network Daughter Card (rNDC)
540-BBHH Dell Intel X540/I350 2x10GBASE-T + 2x1GbE BASE-T Network Daughter Card (rNDC)
540-BBIU Dell Broadcom 57416/5720 2x10GBASE-T + 2x1GbE Network Daughter Card (rNDC)
540-BBIV Dell Broadcom 57412/5720 2x10GbE SFP+ + 2x1GbE Network Daughter Card (rNDC)
540-BBJL Dell Mellanox ConnectX-4 Lx Dual Port 25GbE SFP28 Network Daughter Card (rNDC)
540-BBOZ Dell Intel X550 Dual Port 10GBASE-T RJ45 PCIe Adapter
540-BBPA Dell Intel X550 Dual Port 10GBASE-T RJ45 OCP 3.0 Adapter
540-BBRL Dell Intel X710 Quad Port 10GbE SFP+ Full-Height PCIe Adapter
540-BBRM Dell Intel X710 Dual Port 10GbE SFP+ OCP 3.0 Adapter
540-BCND Dell Mellanox ConnectX-6 Lx Dual Port 25GbE SFP28 PCIe Adapter
540-BCOS Dell Mellanox ConnectX-5 Dual Port 25GbE SFP28 PCIe Adapter
540-BCOT Dell Mellanox ConnectX-5 Dual Port 25GbE SFP28 OCP 3.0 Adapter
540-BCOW Dell Intel E810-XXV Dual Port 10/25GbE SFP28 OCP 3.0 Network Adapter
540-BCVE Dell NVIDIA Mellanox ConnectX-6 VPI Single Port 200Gb/s HDR InfiniBand / 200GbE PCIe Adapter
540-BDBX Dell NVIDIA Mellanox ConnectX-6 Dx Dual Port 100GbE QSFP56 PCIe Adapter
540-BDCE Dell Intel E810-XXV Dual Port 25GbE SFP28 PCIe Gen4 Adapter
540-BDCF Dell Intel E810-CQDA2 Dual Port 100GbE QSFP28 PCIe Gen4 Adapter
540-BDCP Dell Broadcom 57508 Dual Port 100GbE QSFP56 PCIe Gen4 Adapter
540-BDCQ Dell Broadcom 57508 Dual Port 100GbE QSFP56 OCP 3.0 Adapter
540-BDDF Dell Mellanox ConnectX-6 Dx Dual Port 100GbE QSFP56 OCP 3.0 Adapter
540-BDDO Dell NVIDIA Mellanox ConnectX-6 Dx Dual Port 100GbE QSFP56 PCIe Adapter
540-BDDW Dell NVIDIA Mellanox ConnectX-6 VPI Dual Port 200Gb/s HDR InfiniBand / 200GbE PCIe Adapter
540-BDRV Dell NVIDIA Mellanox ConnectX-7 Dual Port 400Gb/s NDR InfiniBand / 400GbE PCIe Gen5 Adapter

HPE

P26256-B21 HPE Broadcom BCM57412 Ethernet 10Gb 2-port SFP+ OCP3 Adapter

Intel

E10G42BTDA Intel Ethernet Server Adapter X520-DA2 Dual-Port 10GbE SFP+ PCIe 2.0 x8 NIC
E810CQDA2 Intel Ethernet Network Adapter E810-CQDA2 Dual-Port 100GbE QSFP28 PCIe 4.0 x16 NIC
E810XXVDA2 Intel Ethernet Network Adapter E810-XXVDA2 Dual-Port 25GbE SFP28 PCIe 4.0 x8 NIC
X550T2 Intel Ethernet Converged Network Adapter X550-T2 Dual-Port 10GBASE-T Copper PCIe 3.0 x4 NIC
X710DA2 Intel Ethernet Converged Network Adapter X710-DA2 Dual-Port 10GbE SFP+ PCIe 3.0 x8 NIC
X710DA4FH Intel Ethernet Converged Network Adapter X710-DA4 Quad-Port 10GbE SFP+ Full-Height PCIe 3.0 x8 NIC

Lenovo

4XC7A08238 ThinkSystem Broadcom 57414 10/25GbE SFP28 2-Port PCIe Ethernet Adapter
4XC7A08248 Lenovo ThinkSystem Mellanox ConnectX-6 Dx 100GbE QSFP56 2-Port PCIe Ethernet Adapter
4XC7A08295 ThinkSystem Intel E810-DA2 10/25GbE SFP28 2-Port PCIe Ethernet Adapter
7XC7A05523 Lenovo ThinkSystem Intel X710-DA2 PCIe 10Gb 2-Port SFP+ Ethernet Adapter
7XC7A05525 Lenovo ThinkSystem Broadcom 57416 10GBASE-T 2-Port PCIe Ethernet Adapter

NVIDIA

MBF2H516C-CESCOT NVIDIA BlueField-2 DPU Dual-Port 25GbE SFP28 16GB Arm Crypto PCIe 4.0 x8 Data Processing Unit
MCX512A-ACAT NVIDIA ConnectX-5 EN Dual-Port 25GbE SFP28 PCIe 3.0 x8 Low-Profile Ethernet Network Adapter
MCX556A-ECAT NVIDIA ConnectX-5 VPI Dual-Port 100Gb/s EDR InfiniBand and 100GbE QSFP28 PCIe 3.0 x16 Adapter
MCX623105AN-VDAT NVIDIA ConnectX-6 Dx EN Dual-Port 25GbE SFP56 PCIe 4.0 x16 Ethernet Network Adapter
MCX623106AN-CDAT NVIDIA ConnectX-6 Dx EN Dual-Port 100GbE QSFP56 PCIe 4.0 x16 Ethernet Network Adapter
MCX75310AAS-NEAT NVIDIA ConnectX-7 Single-Port 400Gb/s NDR InfiniBand and 400GbE OSFP PCIe 5.0 x16 Adapter
MCX755106AS-HEAT NVIDIA ConnectX-7 Dual-Port 200Gb/s NDR200 InfiniBand and 200GbE QSFP112 PCIe 5.0 x16 Adapter

Switches (6)

Dell

Storage (32)

Dell

HPE

Lenovo

Optics (5)

Dell

407-BBOU Dell 10GbE SFP+ SR Optical Transceiver (850nm, 300m MMF)
407-BBOY Dell 100GbE QSFP28 SR4 Optical Transceiver (850nm, 100m MPO-12)
407-BCBF Dell 25GbE SFP28 SR Optical Transceiver (850nm, 100m MMF)

Lenovo

46C3447 Lenovo ThinkSystem 10Gb SFP+ SR Optical Transceiver
7G17A03537 Lenovo ThinkSystem 25Gb SFP28 SR Optical Transceiver

Plan and size it

Draw this setup in the topology planner mrplanb.com
Add up rack power and space rack-density.com
iSCSI explained mrplanb.com
Virtualization storage Ledger: scored storage platforms mrplanb.com
Network monitoring tools compared mrplanb.com
Dell E-Lab Navigator (Dell interoperability) dell.com
HPE SPOCK (HPE storage compatibility) hpe.com
Lenovo ServerProven (Lenovo compatibility) lenovo.com

Other ways to connect

FAQ

Is iSCSI slower than Fibre Channel?

On a dedicated 25GbE or faster network with jumbo frames, iSCSI is close to 16Gb or 32Gb FC for most workloads. NVMe/TCP lowers latency further. FC still wins on isolation and predictable latency under heavy load.

Can storage traffic share the switch with normal traffic?

Yes, on its own VLAN with enough uplink capacity. Busy sites use a separate pair of switches for storage so a broadcast storm or a full uplink on the data side cannot slow the disks.

Do I need special NICs for iSCSI?

No. Any server NIC works with the software iSCSI initiator in Linux, Windows and VMware ESXi. NICs with offload or RDMA help at 100GbE and above.

Buying or selling SAN, NAS and network hardware?

New, refurbished or used. Tell us the quantity and condition and we connect you with a vetted reseller in your region, usually within one working day.

Buy SAN, NAS and network hardware Sell yours