What Nvidia's first Groq 3 LPU benchmarks do and don't tell us about its $20B gamble

Nvidia’s $20 billion bet on Groq’s LPU tech sure looks like it was a good one.

On Monday, the GPU giant offered the first glimpse of just how big a speedup its Groq 3-based LPX racks will provide.

In an independent benchmark conducted by Artificial Analysis, Nvidia’s LPX rack systems managed to churn out 3,400 tokens a second (tok/s) with a 100,000-token input sequence in Google’s Gemma 4 31B model.

According to Nvidia, this makes it 4x faster than the nearest alternative platform, which going off Artificial Analysis’ leaderboard would be a direct dig at Cerebras, which managed a still impressive 882 tok/s under the same conditions.

Acquihired by Nvidia in late December, Groq has LPUs that feature an SRAM-heavy dataflow architecture designed specifically for high-performance inference serving.

Unlike traditional datacenter GPUs, which rely on high-speed DRAM memory tech like GDDR7 and HBM4, Groq’s chips rely entirely on a large pool of on-die SRAM orders of magnitude faster than even the best HBM stacks (around 2.75 TB/s each) available today.

And when it comes to inference, memory bandwidth is the bottleneck to beat.

The third generation of the chips launched as part of Nvidia’s broader Vera Rubin platform earlier this year boasted 150 TB/s of memory bandwidth.

The catch, of course, is that SRAM chews up a ton of die area, which means you can’t pack all that much into a reticle-sized part.

Where Nvidia’s top-specced Rubin GPU boasts 288 GB of onboard memory, each Groq 3 LPU has just 500 MB — 576x less.

That’s far too little memory to run Gemma 4 31B on a single LPU, so Nvidia’s architecture uses Ethernet to distribute models across multiple accelerators.

Each LPX rack can be equipped with up to 256 LPUs for 128 GB of high bandwidth SRAM.

For large models, multiple LPX racks can be ganged together.

So why would anyone need to run Gemma 4 31B at 3,400 tokens a second? You probably don’t, but the AI code assistant or agent you're using it with might.