NVMe NVMe NVMe Host Host Host PCIe Switch NVMe NVMe NVMe Host Host Host PCIe Switch NVMe NVMe NVMe Host Host Host PCIe Switch NVMe SSD NVMe SSD NVMe SSD NVMe SSD NVMe SSD NVMe SSD NVMe SSD NVMe SSD NVMe SSD NVMe SSD • 4 4 4 •4 44 4/ 4 4/ I C OG PG T DOG Tቤተ መጻሕፍቲ ባይዱ SRM A Disk Disk Disk Disk Disk Disk Disk Disk / / / / A ToR B / / B
02
SAS 4.0 Expander 100 24Gb/s PHYs PCIe 4.0 Switch 100 20Gb/s Lanes PCIe 4.0 Switch 100 20Gb/s Lanes PCIe Switch PCIe Switch PCIe Switch NVMe NVMe NVMe Host Host Host NVMe NVMe NVMe Host Host Host NVMe NVMe NVMe Host Host Host Mesh HostDevice O - -- F D RB F -- - - - - RB F - -F AMIH RB F - -- -EC -- -- -N ECJ - -- N ECJ -- -- - EC PG B -- -- -- - --- - -- . 5 5 0 .D . H QPY N 5 0 .D O 88 .H 9GB 2E LU B 0 4 TMR 8 8 BIA 2 . H QPY 1 0 AE A 2 1 0 AE A 2 0 AE A 2 1 1 0 C: +4C # B 3 3 00 00 N 00 04 2 4BB CA4 B AH D : A 4C I M OL 4A C C: A C A 12 02 + + 2+ 1 2 2 2 • 3 • 3 -3 3 # -3 3 # - -3 3 # -3 3 # # -1 VF0 VF1 NVMe SSD NVMe NVMe NVMe NVMe UPI Mesh Unified address space CPU 1 x86 x16 x16 PAX Fabric 2 Raid x16 Card 1 CPU 2 x86 x16 x16 3 CPU 3 x86 x16 x16 CPU 4 x86 x16 x16 4 5 Switch Hops CPU to GPU: 2 to 4 ROC to GPU; 3 CPU to ROC: 2 IB x16 Fanout Switch GPU GPU IB x16 Fanout Switch GPU GPU IB x16 Fanout Switch GPU GPU IB