d-Matrix drinks the Nvidia Kool-Aid with NVLink Fusion and MGX rack designs

AI infrastructure startup d-Matrix on Thursday joined the growing list of chipmakers licensing Nvidia’s NVLink Fusion interconnect tech and rack-scale reference designs to make its high-performance inference platform more accessible to customers.

Under the deal, d-Matrix will integrate support for NVLink Fusion, a high-speed chip-to-chip interconnect that Nvidia began licensing last year, into future chip designs, including its upcoming Raptor accelerators.

As we’ve previously reported, by embracing the tech, d-Matrix sidesteps many of the challenges associated with scaling its chip architecture across large compute clusters.

By using NVLink over alternative interconnects and designing its compute blades around the GPU giant’s MGX reference designs, its customers can deploy its chips using the same racks and NVSwitch fabrics as Nvidia.

By the end of next year, d-Matrix expects to offer systems with up to 144 Raptor accelerators connected by a single all-to-all NVLink fabric.

While there’s a lot we don’t know about Raptor just yet, at the Hot Chips conference last month the company revealed each Raptor “card” would feature 32 GB of ultra-fast 3D-stacked DRAM on board capable of delivering 100 TB/s of memory bandwidth — roughly 4.5 times the memory bandwidth of Nvidia’s Rubin GPU.

By the looks of things, the XPUs that will power d-Matrix's NVL144 racks are about half the size of the Raptor cards shown off at Hot Chips and include around 16 GB of 3D-DRAM and about 50 TB/s of memory bandwidth — still quite respectable by any measure.

With 144 of these per rack, d-Matrix is looking at about 2.3 TB of memory capacity — enough for models exceeding four trillion parameters in size at 4-bit precision — and about 7.2 petabytes a second of peak aggregate memory bandwidth.

This is achieved by bonding compute logic atop a stack of DRAM.

The result is an in-memory compute platform that offers modest capacity while maintaining memory bandwidth closer to that of SRAM than is achievable using HBM.

Memory bandwidth, as you may recall, is the biggest bottleneck for AI inference.

The faster your memory, the faster the system can spew out tokens.

This is exactly why Nvidia dropped $20 billion last year to license Groq’s IP and hire away its engineering talent.

The chip’s SRAM-heavy dataflow architecture was capable of hitting 150 TB/s per chip, but the tradeoff is that SRAM isn’t very space-efficient and the chipmaker could only pack 500 MB of it onto a single die. d-Matrix's Raptor promises to deliver a decent fraction of that bandwidth with 64x higher capacity, which means the company can get away with using far fewer chips per model.