InfiniBand EDR vs HDR vs NDR: Speeds, Connectors and Cables

· Updated · 5 min read · Itzhak Finkelstein, T.E.S IT-SOLUTIONS

Three cable connectors side by side: two QSFP-size plugs and one larger OSFP plug with a finned top

EDR InfiniBand runs 100Gb/s per port on QSFP28. HDR runs 200Gb/s on QSFP56. NDR runs 400Gb/s on OSFP (or QSFP112 on some adapters). All three use 4-lane ports; the lane rate and the connector change, and the connector decides which cable fits. This guide is for engineers and buyers who mix generations, extend an older fabric, or order cables for a switch they already run.

InfiniBand generations at a glance

InfiniBand speed by generationFDR 56 Gb/s (4 × 14.0625, QSFP+), EDR 100 Gb/s (4 × 25, QSFP28), HDR 200 Gb/s (4 × 50, QSFP56), NDR 400 Gb/s (4 × 100, OSFP / QSFP112), XDR 800 Gb/s (4 × 200, OSFP).56 Gb/sFDR4 × 14.0625QSFP+100 Gb/sEDR4 × 25QSFP28200 Gb/sHDR4 × 50QSFP56400 Gb/sNDR4 × 100OSFP / QSFP112800 Gb/sXDR4 × 200OSFPInfiniBand speed by generationFDR 56 Gb/s (4 × 14.0625, QSFP+), EDR 100 Gb/s (4 × 25, QSFP28), HDR 200 Gb/s (4 × 50, QSFP56), NDR 400 Gb/s (4 × 100, OSFP / QSFP112), XDR 800 Gb/s (4 × 200, OSFP).FDR56 Gb/s4 × 14.0625 · QSFP+EDR100 Gb/s4 × 25 · QSFP28HDR200 Gb/s4 × 50 · QSFP56NDR400 Gb/s4 × 100 · OSFP / QSFP112XDR800 Gb/s4 × 200 · OSFP
Port speed per generation: lanes × lane rate, and the connector.

A standard InfiniBand port is 4X: four lanes. HDR and NDR also have 2-lane variants (HDR100 and NDR200) that double the port count of a switch. The table lists the speeds from NVIDIA switch and adapter manuals, with one example switch, adapter and cable family per generation.

GenerationPort speedLanes x lane rateConnectorExample switchExample adapterExample cables
FDR56Gb/s4 x 14.0625Gb/sQSFP (QSFP+)SX6036, 36 portsConnectX-3MC2207130 DAC, MC2207310 AOC
EDR100Gb/s4 x 25Gb/sQSFP28SB7800, 36 portsConnectX-5MCP1600-E DAC, MFA1A00-E AOC
HDR200Gb/s4 x 50Gb/sQSFP56QM8700, 40 portsConnectX-6 (MCX653105A-HDAT)MCP1650-H DAC, MFS1S00-H AOC
HDR100100Gb/s2 x 50Gb/sQSFP56QM8700, up to 80 ports splitConnectX-6 (MCX653106A-ECAT)MFS1S50-H AOC splitter, MCP7H50 DAC splitter
NDR400Gb/s4 x 100Gb/sOSFP; QSFP112 on some adaptersQM9700, 64 ports on 32 OSFP cagesConnectX-7 (MCX75310AAS-NEAT)MCP4Y10 DAC, MCP7Y00 splitter, MMA4Z00 optics
NDR200200Gb/s2 x 100Gb/sOSFP or QSFP112QM9700, up to 128 ports splitConnectX-7 (MCX755106AS-HEAT)MCP7Y50 1:4 splitter
XDR800Gb/s4 x 200Gb/sOSFPQ3200 (72 ports) / Q3400 (144 ports)ConnectX-8Not covered here

Port counts: the SX6036 has 36 FDR ports with up to 4.032Tb/s aggregate throughput. The SB7800 has 36 EDR ports and 7.2Tb/s. The QM8700 has 40 HDR ports and 16Tb/s. The QM9700 has 64 NDR ports and 51.2Tb/s bidirectional. For a detailed switch decision, see our comparison MQM8700-HS2F vs MQM9700.

What the connector tells you

  • QSFP+ and QSFP28 are the FDR and EDR form factors. ConnectX-3 cards use QSFP+ ports; ConnectX-5 cards use QSFP28.
  • QSFP56 is the HDR form factor. NVIDIA's LinkX guide states that Quantum HDR "QSFP56 cages accept 100G EDR cables and transceivers." The QM8700 manual lists each QSFP56 port for 40/56/100/200Gb/s.
  • QSFP112 appears on some ConnectX-7 cards. Per the same LinkX guide, QSFP112 cages accept 200G QSFP56 and 100G QSFP28.
  • OSFP is the NDR switch form factor. On the QM9700, each OSFP cage carries 2 logical NDR ports. OSFP is a different, larger module, so QSFP cables do not plug into it. To reach QSFP56 devices you need a cable with an OSFP end.

Backward compatibility: what speed you actually get

NVIDIA switch manuals state that "InfiniBand speed is auto-adjusted by the InfiniBand protocol." The QM8700 supports QDR, FDR, EDR and HDR. The QM9700 supports QDR, FDR, EDR, HDR and NDR. In practice, plan on the slowest part of the link: switch port, cable or adapter.

  • HDR port with an EDR cable: the QSFP56 cage accepts the QSFP28 cable. The cable is rated for 100Gb/s, so the link runs at EDR, not HDR.
  • HDR switch with EDR adapters: a ConnectX-5 (EDR, 100Gb/s) on a QM8700 port links at EDR.
  • NDR switch to HDR devices: the MCP7Y60 splitter connects a Quantum-2 NDR switch to Quantum HDR switches, ConnectX-6 adapters or BlueField-2 DPUs at 2 x 200Gb/s. NVIDIA states it can downshift from 4 x 50G PAM4 to 4 x 25G NRZ for EDR.
  • NDR switch to NDR adapters: the MCP7Y00 links a Quantum-2 twin-port OSFP cage to two OSFP-based 400Gb/s ConnectX-7 cards.

Split cables: HDR100 and NDR200

How HDR and NDR ports splitAn HDR 200 port with 4 lanes splits into 2 HDR100 ports with 2 lanes each. An NDR 400 port with 4 lanes splits into 2 NDR200 ports with 2 lanes each.HDR 2004 lanesHDR1002 lanesHDR1002 lanesHDR 200 → 2 × HDR100NDR 4004 lanesNDR2002 lanesNDR2002 lanesNDR 400 → 2 × NDR200How HDR and NDR ports splitAn HDR 200 port with 4 lanes splits into 2 HDR100 ports with 2 lanes each. An NDR 400 port with 4 lanes splits into 2 NDR200 ports with 2 lanes each.HDR 2004 lanesHDR1002 lanesHDR1002 lanesHDR 200 → 2 × HDR100NDR 4004 lanesNDR2002 lanesNDR2002 lanesNDR 400 → 2 × NDR200
A split cable divides the 4 lanes of one port between 2 ports.
Switch ports with and without split cablesQM8700: 40 HDR ports at 200Gb/s, or up to 80 HDR100 ports with split cables. QM9700: 64 NDR ports at 400Gb/s, or up to 128 ports of 200Gb/s with split cables.Full 4-lane portsPorts with split cablesQM87004080QM970064128HDR100 and NDR200 ports use 2 lanes each, at half the port speed.Switch ports with and without split cablesQM8700: 40 HDR ports at 200Gb/s, or up to 80 HDR100 ports with split cables. QM9700: 64 NDR ports at 400Gb/s, or up to 128 ports of 200Gb/s with split cables.Full 4-lane portsPorts with split cablesQM87004080QM970064128HDR100 and NDR200 ports use 2 lanes each, at half theport speed.
Splitting each port into two 2-lane ports doubles the port count of the switch.

HDR to 2 x HDR100. The QM8700 manual states that "by utilizing two pairs of two lanes per port," the switch "can support up to 80 ports of 100G," together with ConnectX-6 adapters. The MFS1S50-H0xxV active optical splitter carries one 200Gb/s QSFP56 end to two 100Gb/s QSFP56 ends, at lengths up to 30m. The MCP7H50 is the copper version for short in-rack runs.

A 1-to-2 splitter copper cable laid out in a Y shape on a grey ESD mat

NDR to 2 x NDR200. The QM9700 supports up to 128 ports of 200Gb/s through port splitting. NVIDIA's MCP7Y50 splits one twin-port OSFP into 4 x 200G, each on 2 lanes of 100G PAM4. The MCP7Y00 instead splits it into 2 x 400G for single-port OSFP ConnectX-7 cards. Plan which switch ports are split before you order cables.

Common mistakes

  • Plugging HDR QSFP56 cables into an NDR switch. They do not fit the OSFP cage. Use an OSFP to 2 x QSFP56 cable such as the MCP7Y60 (DAC) or MFA7U10 (AOC).
  • Treating HDR100 as EDR. NVIDIA states that HDR100 (2 x 50G PAM4) "is 100G and not compatible with 100GbE based on 4x25G-NRZ." The same lane mismatch applies to EDR, so an HDR100 splitter end does not link to an EDR ConnectX-5 port.
  • Expecting 200Gb/s from an EDR cable. An MCP1600-E or MFA1A00-E cable in a QM8700 port gives 100Gb/s.
  • Using the MCP7Y00 with QSFP112 adapters. NVIDIA limits it to ConnectX-7 OSFP cards. BlueField-3 and QSFP112 ConnectX-7 cards need splitters with QSFP112 ends (MCP7Y10, MCP7Y40).
  • Ordering the Ethernet version. Several LinkX families exist in InfiniBand and Ethernet versions. We list MCP1650-H parts as HDR and MCP1650-V parts as 200GbE. Order the part number rated for InfiniBand.

Sources: NVIDIA QM87xx HDR switch manual: introduction; QM87xx interfaces; NVIDIA QM97xx NDR switch manual: introduction; QM97xx interfaces; NVIDIA SB7xx0 EDR switch manual; Mellanox SX60xx switch manual (PDF); Mellanox 56Gb/s QSFP+ DAC brief (PDF); NVIDIA LinkX guide for 50G-PAM4 and 25G-NRZ cables (PDF); MCP7Y60 splitter specifications; MCP7Y00 splitter specifications; ConnectX-6 VPI manual; ConnectX-7 port link type; NVIDIA XDR switch manual.

Parts we stock

Parts in this post

We rebuild and test every switch and cable before it ships. For cables that means connector and latch inspection, label, part number and length checks, jacket condition, and an electrical test where a test setup exists for that cable type. See how we build, all switches and all cables and interconnect.

FAQ

What speed does an HDR port run with an EDR cable?

EDR, 100Gb/s. NVIDIA states that Quantum HDR QSFP56 cages accept 100G EDR cables and transceivers, and that InfiniBand speed is auto-adjusted by the protocol. An EDR cable such as the MCP1600-E is rated for 100Gb/s, so a QM8700 port with that cable links at EDR, not 200Gb/s HDR.

How many HDR100 ports does a QM8700 support?

Up to 80. The QM8700 has 40 HDR 200Gb/s QSFP56 ports, and each port can be split into two 2-lane HDR100 ports of 100Gb/s, used together with ConnectX-6 adapters. Splitter cables such as the MFS1S50-H0xxV (AOC, up to 30m) make the connection.

Can an NDR QM9700 switch connect to HDR ConnectX-6 adapters?

Yes. The QM9700 supports QDR, FDR, EDR, HDR and NDR. The MCP7Y60 DAC splitter connects one Quantum-2 OSFP cage to two QSFP56 devices, such as ConnectX-6 adapters, at 2 x 200Gb/s HDR, with reach up to 2m.

Is HDR100 the same as EDR 100Gb/s?

No. HDR100 uses 2 lanes of 50G PAM4, while EDR uses 4 lanes of 25Gb/s. NVIDIA states that 100G based on 2 x 50G PAM4 is not compatible with 100G based on 4 x 25G NRZ. An HDR100 splitter end does not link to an EDR ConnectX-5 port.

What is the difference between NDR and NDR200?

NDR is 400Gb/s on 4 lanes of 100Gb/s. NDR200 is 200Gb/s on 2 lanes. A QM9700 offers 64 NDR ports or up to 128 ports of 200Gb/s with port splitting. The ConnectX-7 MCX755106AS-HEAT (dual-port QSFP112) runs up to NDR200 on InfiniBand and ships set to 200GbE Ethernet by default.

Mixing generations or ordering cables for an existing fabric? Send us the part number or the requirement and we will match switch, cable and adapter.

All posts · Shop by product line