OpenAI details GPT-Live architecture for continuous realtime voice AI

OpenAI has published a detailed engineering account of GPT-Live, its third-generation voice system, explaining how the company moved its voice AI away from a turn-based model and toward continuous full-duplex conversation. The August 3 post frames the change as a systems problem as much as a model problem: the voice model must listen and speak at the same time while deeper reasoning, tool use, transcript handling, and application logic continue around it.

The core shift is that GPT-Live removes the separate turn detector from the audio path. Earlier voice assistants often waited for a small model to decide whether the user had finished speaking before sending a request to a larger model. OpenAI says GPT-Live instead streams audio into the voice model and streams speech back out, while calls to frontier models such as GPT-5.5 can run asynchronously when a conversation needs more reasoning or tool use.

Why it matters

That design matters because voice assistants are judged on timing as much as accuracy. A response that arrives late, interrupts too early, or freezes while a tool runs can make an otherwise capable AI system feel awkward. OpenAI says it rebuilt parts of the media frontend and inference path in Go, separated realtime media from application logic, and used WebRTC as the transport base so packet loss, reconnects, and clock drift can be handled without stalling the conversation.

The post also describes stateful inference work that lets long voice sessions continue while model instances are replaced or conversation context is compacted. Instead of pausing a call while a new model instance is prepared, OpenAI says the system can warm a replacement beside the active one, prefill it with session context, and switch over once it is ready. The same approach is used when accumulated context needs to be reduced for longer conversations.

• GPT-Live is built around a dedicated realtime media path for audio.
• Deeper reasoning and tool calls run on a separate asynchronous path.
• OpenAI says its WARP protocol work reduces WebRTC media and data startup from six network round trips to one.
• The architecture already powers ChatGPT Voice features and is planned to underpin a future GPT-Live API.

OpenAI says production testing exposed bottlenecks beyond GPU throughput, including regional routing, CPU-side stream handling, memory pressure, reconnect behavior, and observability gaps. The result is a platform-style architecture intended to support more voice-driven agent features without making the live audio loop wait for every backend task to finish.

Sources

• openai.com

Cover photo by Egor Komarov on Pexels, used under the Pexels License.