Next-gen AI networks may hinge on the telephone switchboard's return

If you thought 72 GPUs per rack was dense, next-generation designs from Nvidia and others will cram hundreds or even thousands of accelerators into a single massive system.

But for any of that to happen, they're going to need a lot of optics and technology that's reminiscent of the old-fashioned telephone switchboard.

This reality has fueled a flurry of investment in everything from photonics startups to established optical equipment and fiber manufacturing.

In March, Nvidia invested $6 billion ($2 billion apiece) in Coherent, Lumentum, and Marvell to advance their optics tech.

Some of these investments went to support optical circuit switching (OCS) technology.

On Wednesday, Nvidia joined Maverick Silicon and Light Street Capital to spend another $125 million to support the development of iPronics' second-generation OCS tech.

Why switch packets when you can switch light? Optical circuit switches are switches only in the literal sense.

Unlike a Broadcom Tomahawk or Marvell Teralynx ASIC, an OCS appliance can't switch packets.

The opto-electrical appliances are the modern equivalent of a telephone switchboard.

But rather than human operators manually patching together two telephone lines, a high-speed actuator built using microscopic mirrors, piezoelectric actuators, LCDs, or other technologies reconfigures the network in the literal blink of an eye.

OCS isn't particularly common in modern GPU deployments, but they have been used in AI clusters for quite a while.

Specifically, Google has used OCSes in its TPU clusters for years now.

Traditionally, Google's TPU pods have employed 2D and 3D torus topologies where accelerators communicate in a great big mesh, rather than relying on packet switched fabrics to connect them all together.

The trade-off with mesh networks is potentially higher chip-to-chip latency and rigidity.