OpenAI posts first Jalapeño benchmarks: ahead of Blackwell on tokens per user

OpenAI published the first benchmark results for Jalapeño, its in-house inference chip, at the Hot Chips conference on Tuesday. Tested on SemiAnalysis' InferenceX benchmark, the system recorded more tokens per user and more throughput per kilowatt than the currently available state-of-the-art inference processors, according to the results the company presented. The comparison baseline is an Nvidia Blackwell system.

Richard Ho, OpenAI's head of hardware, called the results "a very, very significant performance advance over state of the art" in a press call. He described Jalapeño as serving more AI work per unit of power while returning responses faster: efficient for serving a large number of customers, and capable of low latency as well.

How the chip is built

Hot Chips is the annual conference where major chip designers walk through their architectures in detail, the point at which a design stops being an announcement and starts being a spec.

Jalapeño was first announced last October and developed in close collaboration with Broadcom. OpenAI says its own models assisted in the development process, and the company plans to make Jalapeño a multigenerational platform, with products, models, chips, and memory developed in concert.

The benchmark focus follows from that full-stack approach. OpenAI targeted the phases of inference that tend to become bottlenecks, the prefill phase and communication between components. In a blog post presenting the results, the company explained that the chip is designed to minimize data movement and communication delays, so that model state, including the KV cache used while generating a response, can be explicitly placed and kept local, while the system activates the right combination of compute, memory, and networking for each inference phase.

For a team running inference at scale, that design maps to two numbers that matter: throughput per kilowatt, which drives datacenter power costs, and tokens per user, which drives perceived response time.

When it ships

Ho estimates deployment in very small volumes at the end of 2026, with more significant deployment coming in 2027. By then, the Blackwell comparison will have aged. These numbers are a snapshot of today's inference hardware, not a forecast of the gap a year from now.

What the results show is concrete: the architecture targets the KV cache and data movement, the parts where general-purpose accelerators spend energy moving state around. What is unverified: whether the measured gap holds against the next generation of server hardware. The benchmarks are OpenAI's own, run on a third-party benchmark suite, and details like the models tested or the context lengths were not in the presentation.

For a team deciding on compute for inference workloads today, Jalapeño does not change the decision, since it is not available in any quantity yet. What it does change is the longer-term picture. OpenAI now has its own silicon track for the workloads it serves, and the cost per token of self-hosted inference has a second design pushing on it.