OpenAI reports significant performance gains from its custom Jalapeño inference chip, utilizing AI-assisted design to optimize hardware for agentic model workloads.
OpenAI has released performance data for its custom Jalapeño inference silicon, revealing substantial gains in throughput and latency reduction across multiple large language model architectures. Developed in collaboration with Broadcom, the hardware reached production in nine months by integrating AI-assisted design cycles throughout the development process.
The company tested the silicon against the InferenceX benchmark suite using GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T models. Results indicate that Jalapeño achieves 1.5 to 1.9 times higher work-per-watt efficiency compared to existing accelerators while simultaneously reducing end-to-end latency by 1.7 to 3.6 times.
Hardware engineers prioritized minimizing data movement between memory and compute cores to address the specific performance bottlenecks inherent in multi-step agentic workflows. By keeping model state and the KV cache local, the architecture reduces the synchronization delays that typically compound during sequential inference tasks.
The chip is rated at 700 watts, though internal testing showed peak power draw remained at 550 watts during the evaluation period. Performance metrics on highly interactive workloads demonstrated speedups ranging from 2.1 to 4.1 times over standard industry benchmarks, providing a distinct advantage over general-purpose GPU clusters.
The architecture utilizes a specialized memory hierarchy designed to mitigate the bandwidth constraints common in transformer-based models. By optimizing the placement of KV cache data, the system minimizes the frequency of off-chip memory access, which is often the primary source of latency in large-scale inference.
Furthermore, the chip’s internal networking fabric allows for more efficient data distribution across the system, ensuring that compute units are not stalled while waiting for memory fetches. This design choice specifically targets the memory-bound nature of token generation, where the system must retrieve large amounts of weights and KV cache state for every single token produced.
OpenAI utilized its internal models to accelerate the design, measurement, and verification phases of the silicon lifecycle. This methodology allowed the team to map, place, and schedule workloads across the system with greater precision than traditional manual optimization, effectively shortening the time-to-tapeout.
AI-generated implementations of specific GPT-OSS attention and mixture-of-experts blocks demonstrated a 1.5 to 1.8 times performance increase over human-written counterparts. This capability suggests a shift toward hardware designs that are explicitly structured for machine-driven optimization rather than solely human-coded kernels.
The integration of Codex with GPT-Astra enabled the hardware team to achieve high-performance targets for three additional open-weight models within a two-month timeframe. This rapid adaptation highlights the potential for automated kernel generation to keep pace with the evolving requirements of new model families.
The significance of this development lies in the decoupling of throughput improvements from the latency penalties typically associated with batching. By maintaining high responsiveness during interactive tasks, the architecture addresses a critical limitation in current inference infrastructure that relies on general-purpose GPUs, which often struggle with the sequential dependencies of agentic models.
This shift toward custom silicon allows OpenAI to exert greater control over the hardware-software co-design process as its model roadmap progresses. The ability to program the chip using local tensors and explicit communication protocols provides a foundation for more efficient scaling of future model iterations, moving away from the rigid constraints of standard accelerator programming models.
The reliance on AI to rewrite and optimize low-level code represents a transition in how hardware accelerators are deployed for large-scale inference, where the software layer is now dynamically generated to match the silicon’s physical layout. Future iterations will focus on further reducing the manual effort required to adapt new model architectures to the underlying silicon, aiming to achieve near-optimal utilization of compute resources for each specific model family.
OpenAI intends to deploy Jalapeño within its own production infrastructure by the end of the year. Development teams are already engaged in the design of the next two generations of the chip to ensure the hardware maintains parity with the increasing parameter counts and architectural complexity of upcoming generative models.
