OpenAI just changed how Voice AI is built

GPT-Live-1 makes full duplex voice AI the new baseline, and it is going to be hard to beat with today’s turn taking architecture.

The piece in question can be found here. What follows is an analysis of how this new model and architecture is going to affect the Voice AI industry, placing everyone at an impasse as to where they should be headed versus the trajectory they’ve been taking up until today.

We’re moving away from lowering latency across the pipeline, polishing and optimizing our turn detection algorithms - and towards continuous, streamed conversations. It is the difference between playing chess and playing Quake.

Justin Uberti, Open AI and Built with WebRTC

OpenAI exposed technical details a few months back, sharing with everyone how they reimplemented their WebRTC infrastructure for Voice AI. It was an interesting post and a revealing tell on their decision to invest further in WebRTC.

I immediately reached out to Justin Uberti to get a session with him on LinkedIn and he agreed. For those who don’t know, Justin is one of the OGs of WebRTC at Google. The one who is most associated with the development of libWebRTC. He now works at OpenAI, making their voice (and video?) become a reality. He told me to wait a month until he has something more to discuss.

After a couple of months of back and forth with a long wait and a few cancelled Built with WebRTC sessions, the news finally broke and my session with Justin got scheduled and delivered.

It was an eyeopener. For me, this placed the whole industry on notice, changing the whole architecture.

Here's the full session with Justin, 44 minutes: Built with WebRTC, with Justin Uberti of OpenAI.

Watch it when you have the time. What follows is what I took away from it, and where I think it leaves everyone else.

To understand why, let's first see how Voice AI stacks are being taught and built today (myself included here).

The Voice AI stack everyone has been building

The industry as a whole has been looking at Voice AI from the eye of a reasonable engineer: take a big and challenging task and split it into smaller problems. Now go fix each problem separately.

What does this mean?

You start from the LLM. That’s the part we’re all used to chatting with already via text using prompts.

Now that we want to talk to it, all that is left is to turn the words we say into a text prompt - that’s the STT part. And then turn the reply of the LLM from text to speech - TTS.

The next set of problems surface, in the form of faster LLMs for such interactions, knowing when someone hits the virtual enter while speaking - where we introduce VAD and turn detection components. Then dealing with the fact that we need to lower latencies further and turning this whole pipeline into something that processes smaller and smaller chunks of “time” data. We started with the whole segment - and we switched to token generation, or chunks of seconds and milliseconds of audio data.

It works. Up to a point.

This ongoing work of optimizing, tweaking and finetuning has a similar problem to the way we downsize the size of our CPUs and GPUs - we get closer and closer to the size of an atom in our miniaturized fabrication of the transistors we use.

With Voice AI, we try to do something similar by reducing the chunks processed in order to reduce the time between the user stops speaking until the time to first audio (TTFA).

The thing is we’re already getting close to the size of an atom with our optimizations here. And it seems like getting any better requires a paradigm shift - and a change in architecture.

OpenAI deleted the turn detector, it did not improve it

While everyone has been focusing on optimizing the turn detector, OpenAI simply removed it from the equation.

You can trace this back to Justin’s time at Ultravox. Justin co-founded it in 2022, then left it at the end of 2024 to OpenAI. Ultravox was focused then on an open source speech-to-speech model. So Justin had the two pieces of the puzzle that are unique to him in the industry:

1. Deep knowledge and understanding of WebRTC. He practically built it in Google, baking it into Chrome
2. Understanding of speech models - and where and how they replace STT-LLM-TTS - not by using these models and algorithms - but by training them and creating them himself

Coming to OpenAI had a main reason for Justin - making Voice AI a reality. Something he likely couldn’t get done with the resources Ultravox had at their disposal.

The mechanism used here is rather simple in its approach: a full-duplex model that listens and speaks at the same time. The problem is how do you keep it fluid and smart enough to have thinking capabilities - and the answer is - you don’t. You separate that into two pieces:

1. The talking part, which has conversation smarts
2. The thinking part, a frontier LLM model in the traditional sense as far as I could understand

This solves the problem that Justin said was like “asking a child to make decisions for an adult” - turn detection can’t really figure out when people expect a response good enough - “if it even works 95% of the time, it's just not going to be good enough”. So they made “turn detection” part of the speaking model - GPT-Live-1 - that runs natively on the audio.

Two tracks: one holds the conversation, one does the thinking

Back to that two-models approach.

We have the one that handles speaking. Fluidly. It can’t think and speak well - too many variables. Maybe in the future, but not with today’s technology. So it simply delegates to a frontier LLM model for the thinking.

This opens up two very important capabilities:

1. It decouples speaking from thinking. A frontier model can continue to be trained sans voice and still be used for Voice AI. It gives you the best of both worlds. Theoretically, they could place Anthropic’s Fable or Mythos LLM right there - or just their own gpt5.6 (or newer?) models
2. Mid session, they added the ability to replace the session with another. Things you could do with this capability?
   1. Run compaction, to be able to hold context for longer conversations
   2. Switch to another model if needed
   3. That’s 3 models running at the same time - GPT-Live-1, Frontier, the replacement Frontier being compacted or prepping to be handed over to - not an easy feat and an elegant solution to many challenging problems today

This change in architecture gives huge flexibility to OpenAI that others don’t have. It places them on the bleeding edge of Voice AI.

The protocol layer moved too, and it moved in public

Something that OpenAI wrapped into this announcement was the implementation of WARP - the reduction of call setup time in WebRTC by reducing the number of necessary network round trips to get to media from six down to two (and in the case of OpenAI - down to one).

Behind this approach, there’s the idea and understanding that shorter connection time means better user experience. The way this was framed and reasoned by Justin was that the first voice utterance we start with in our interaction with an AI agent gets missed when the setup time is long, and that means missing the context at the beginning of a session where it is the most important - since there’s no history to rely on yet.

An alternative take is simply that Google was always focused on this. Years ago, at Kranky Geek, they gave a talk about speeding up signaling for WebRTC. Justin, as a Googler, has that in his DNA, which leads to this being an optimization problem he was itching to scratch and fix.

That turned into a view of where WebRTC can be improved. And then into actions - something we rarely see in WebRTC’s ecosystem 👇

Shortening call setup time in WebRTC

How do you speed up WebRTC call setup time?

The way you do that anywhere else in complex protocols - you start parallelizing and piggybacking things after making an inventory of capabilities and contracts that need to be met by each layer.

Connecting a WebRTC session is an arduous task. This is partly the reason some are so enchanted by MOQ (go figure). You start off with signaling, out of scope of WebRTC, passing your SDP blobs in offer/answer. From these, ICE protocol kicks in, relying on STUN and TURN. And then, you end up using DTLS to run encryption for your SRTP and data channel traffic. Each layer has its job cut out for it. And each is used also outside of the context of WebRTC.

In a way, WebRTC is the cookbook recipe, and each of the other protocols is an ingredient that is used. These ingredients can be used elsewhere as well, but their mix together is important.

And in WebRTC, the ingredients overlap. You add less sugar to a cake recipe when it is a chocolate cake that has milk chocolate in its ingredients - because you know there’s sugar in that milk chocolate already. This one is probably not the best analogy… I’d put more sugar anyways 😉

In the same way, there are redundant tasks running across these protocols that run one after the other. DoS attacks are already taken care of prior to starting a data channel using SCTP - that’s what the other layers are doing already, so that piece for example, can be “removed”.

The above is the concept behind WARP - a new contribution to the WebRTC standard for accelerating setup time by piggybacking on previous protocol layers information that can be used by the next layer to reduce the number of round trips needed to negotiate its connection.

The end result? 2 round trips instead of 6 to connect a typical WebRTC session.

OpenAI, being OpenAI, took it further and squeezed that into a single round trip with what they call Instant Connect. Not by keeping sessions warm and waiting: Justin threw that idea out as "a million idle peer connections" burning resources for sessions that never happen. What they do instead is pre-negotiate the session parameters with zero server state and zero server processing, so nothing gets reserved until the user actually presses the button. Is it worth it? Maybe.

The rest of WARP? This is being introduced to libWebRTC, and by extension to Chrome (and possibly other browsers?). If this is something you’d like to benefit from, you’ll need it implemented in the media servers you are using as well.

Something to be said here… I used to preach companies to work with WebRTC and not fight it - accept that this is what’s in the browser. OpenAI took a different approach here and it worked well for them. I was wrong here. You can improve WebRTC, but it requires collaboration and a concerted effort that isn’t just code.

Bringing WebRTC to 2026 and Voice AI

Human perception isn’t machine perception. The way we speak, think and hear is different from how machines do that. When that boundary is crossed and a human talks to a machine, there needs to be an accommodation made for the machine as well - that is what is going to lead to better quality of the conversation. It is what OpenAI is now doing.

This opens up a bigger opportunity. The one of making changes in how WebRTC works to make it work better for more use cases.

Oftentimes, that is something that has been Google’s role in the industry. Other vendors complain that things are missing and that Google isn’t doing its job (who decided that?) while Google handpicking the features that make sense to Google and implementing them.

This was never sustainable.

In comes Justin, picks Philipp Hancke for the ride, and off they go figuring out how something can be designed, specified, standardized and implemented end to end. All while making sure Google ends up integrating the work into Chrome itself.

While the code piece was a challenge - it isn’t anymore. A competent developer can get that done today using AI faster than ever - he just needs to know enough about WebRTC to do it properly. The hard piece was always getting that code approved by Google to be incorporated in libWebRTC+Chrome AND in following through the arduous standardization processes - both people problem and processes more than engineering work.

There’s more in the way Voice AI behaves that lends itself to changes in WebRTC that can increase the overall quality. Justin alluded to having his own list of things to work on next here.

Should the rest of the industry sit on the sidelines and wait for OpenAI and Justin to get these things done and launched by OpenAI before they start their own effort on these things, always coming as a distant second on innovation?

Pion as our second most important WebRTC stack implementation

One thing that has surfaced more frequently these past two years - Pion is becoming the second most important WebRTC implementation out there.

When the initiative started years ago I was skeptical. How can a small initiative, not backed by any large corp or even startup ever gain popularity and succeed when the original implementation of libWebRTC is so dominant, rich and… ships inside Chrome (you can’t really beat that in getting close to the browser behavior).

Sean DuBois did the impossible here - he almost single handedly got Pion to where it is today - a piece of infrastructure that companies such as OpenAI and LiveKit rely on. His optimism and can-do attitude, coupled with openness and endless enthusiasm to what developers are doing with Pion and WebRTC in the industry is second to none.

Pion has become for many reasons the second WebRTC stack implementation the industry needed.

I think that for 1:LLM scenarios, it will likely become the go-to stack to use moving forward.

Full duplex voice AI makes WebRTC more necessary, not less

The switch OpenAI made from turn-taking to continuous streaming folds into another reality.

WebRTC is the way to go for the foreseeable future with Voice AI.

WebSocket will stay the solution for server-to-server but user-to-server is going to go with WebRTC.

WebTransport, MOQ, ROQ or other experiments won’t cut it. Yes. They make for nice demos and parlor tricks, but they require so much more wiring that they make no real sense.

Everything WebRTC brings out of the box here is something you need to self develop with alternative solutions.

The reduction of call setup time in WebRTC brings it on par with some of the complaints the MOQ proponents have.

Upcoming optimizations to it, based on what large scale production solutions (ChatGPT Live anyone?) will put it at the forefront of how things are done and solidify it is the best practices approach even further.

Echo cancellation, packet concealment, double talk - all these are hard problems to solve. And WebRTC has been solving them for years and is now being directed and reshaped towards Voice AI problems with its stack. It is going to be hard to beat that.

The ops reality that is not in anyone's architecture diagram

This switch to full streaming instead of turn taking means more resources. Justin stated how this means more compute capacity is needed to hold and run this - including the two model lanes (GPT-Live-1 alongside a frontier LLM).

He also alluded to the fact that running the media piece using Python has its limitations - it just doesn’t do real time so well - you can’t get an accuracy of 10 milliseconds that you need for running media around, which leads to higher jitter as an example. It gets hidden if you’re not really measuring across your implementation.

What Justin shared was that when they switched from Python to Go (Pion), they saw their performance improve across the board - their p50 was now their new p95. Quality has greatly improved in ways that are hard to achieve otherwise.

This is likely also a path that OpenAI will take, in replacing Python (an AI darling language) with Go or Rust across the media processing pipeline, where performance and latency matters a lot more than availability of tooling and ecosystem.

Voice AI and definitely Video AI vendors who want to get to realtime need to take note here.

What this actually means if you are building Voice AI

Assuming you are developing Voice AI solutions today. What does that mean to you?

If you are building on a framework

Time to look with fresh eyes in your architecture.

Question the route you’ve taken. See if it is still relevant, or should you walk a different path.

If you want to stay a leader in the space, my suggestion is to start preparing to go towards standardization and improvements of libWebRTC itself. This can be the missing piece in your long term puzzle.

If you sell a Voice AI platform or CPaaS

Turn detectors and latencies are nice.

But maybe the battle and conversation is going to switch towards fluidity and true conversational experience. Then there’s the part of being able to hold long conversations.

Again, this approach is going to lend itself to a different architecture than the one you currently have.

If you are buying Voice AI

You are at an impasse.

OpenAI has shown the way.

The vendors you are likely using are still not there, and haven’t even committed to making that change yet.

Time for some tough conversations and decisions here.

A bit more on this

A few days after the session with Justin, I sat down with Arin Sime to talk this through out loud:

Same subject, conversation form. Some of what came up there didn't make it into this article.

What’s next?

Change. Which is the only constant in our lives it seems.

If you want to join me for the ride, be sure to subscribe to this blog.

If you are serious - I’d suggest the WebRTC Insights service. That’s where me and Philipp Hancke make sure you keep up to date with the goings of WebRTC - technical and market. A month from now, we are also hosting WebRTC Insights Live for our customers, touching on some key topics of this industry. Obviously, we did mention the pieces that led to WARP very early on.

We will be answering the question of What’s next in various aspects and areas chosen by our clients.