Nvidia's B200 lands at GTC: Blackwell is built for inference

At GTC 2024, Nvidia's developer conference in San Jose, the company unveiled its next GPU, the B200, built on the new Blackwell architecture. Nvidia's own numbers: 4x the training performance of the Hopper H100, up to 30x the inference performance, and up to 25x the energy efficiency. Those are vendor claims and not yet independently verified, but the specifications underneath them are concrete.

Two dies acting as one chip

The B200 packs about 1,600 square millimeters of processor across two 800 mm² silicon dies, linked inside the same package by a 10 terabytes-per-second connection so they behave like a single 208-billion-transistor chip. The dies are built on TSMC's N4P process, which Nvidia says gives a 6% performance edge over the N4 process used for Hopper GPUs like the H100.

Memory is where the spec jump is easy to see: 192 GB of HBM3e high-bandwidth memory, up from 141 GB on the H200, with bandwidth doubled to 8 TB/s from 4.8 TB/s. High-bandwidth memory is increasingly the constraint on both latency and power for large AI models, so this is the kind of number infrastructure teams track closely. The architecture itself is named after David Harold Blackwell, the first Black inductee into the U.S. National Academy of Sciences.

Fewer bits per number

Chip process did some of the work, but Nvidia attributes most of Blackwell's gains to doing math with smaller numbers. At IEEE Hot Chips last year, Nvidia chief scientist Bill Dally said the bulk of the company's AI performance gains had come from using fewer bits to represent numbers in AI calculations.

Hopper introduced what Nvidia calls the transformer engine: hardware that inspects each neural network layer and decides whether it can run in lower precision, down to 8-bit floating point. Blackwell goes further. Its units handle 4-bit floating-point matrix math, and they can be applied to parts of a layer rather than whole layers, which Nvidia VP of HPC Ian Buck says is a level of granularity his team did not have before. Smaller numbers are faster and cheaper to compute, use less memory and bandwidth, and take less silicon.

From one card to a SuperPOD

The scale story rests on NVLink, now on its fifth generation: 1.8 TB/s bidirectional bandwidth between GPUs, double Hopper's, with high-speed communication among up to 576 GPUs. The building blocks are Grace Blackwell superchips: a Grace CPU and two Blackwell GPUs in one module connected by NVLink. A DGX GB200 contains 36 of those, and eight DGX GB200s form a 576-GPU DGX SuperPOD that Nvidia says reaches 11.5 exaflops at 4-bit precision, with Quantum InfiniBand scaling out to tens of thousands of GPUs.

Blackwell also adds a dedicated reliability engine that Nvidia says uses AI-based diagnostics to forecast hardware problems, aimed at keeping massive training clusters running for the weeks a large language model training run can take. Nvidia says SuperPODs and other Blackwell systems will be available later this year, and it also announced GR00T, a foundation model for humanoid robots.

Why it matters

For a team running models in the cloud or considering on-prem GPUs, the headline number that actually matters is cost per token, and Nvidia has not published pricing yet. If the 4-bit inference results hold up in production workloads, that number could drop substantially, but "up to 30x" is a ceiling measured under specific conditions, and some are already treating it as the marketing number to hold Nvidia to. The hardware arrives later in 2024, so the practical move is to wait for real deployments and third-party benchmarks before committing a budget.