Meta's new MTIA 400 chip has a split personality: Training AI and serving ads

If you’re going to build a custom AI accelerator, inference is a good place to start.

The clusters are smaller, the chips can be simpler, and at the end of the day they only need to do one thing: serve tokens.

OpenAI’s spicy new Jalapeño inference chips are just the latest example.

Social media magnate Meta is bucking this trend.

Its first proper generative AI accelerator, the MTIA 400 — short for Meta Training and Inference Accelerator — is aimed squarely at LLM training.

The Facebook parent is no stranger to custom silicon.

But, much like Amazon and Google, its first AI accelerators weren’t built to run AI chatbots or train generative AI models.

They were built to serve ads.

The MTIA 400, teased earlier this year and detailed at the annual Hot Chips semiconductor development conference this week, will perform some inference duties.

Just not of the GenAI variety.

Instead, it’ll be saddled with running the ad recommender systems that actually pay Meta’s bills.

The combination of LLM training and the deep learning recommender model (DLRM) inference used for serving ads is unusual, as the two have wildly different performance demands.

LLM training is enormously compute-intensive, often requiring tens or even hundreds of thousands of accelerators to train models in a reasonable amount of time.

DLRM inference, on the other hand, is a predominantly memory-bound job, which means most of the FLOPS that make the chip good at training are going to be left sitting idle.