H.264 is set to replace VP8 for WebRTC services.
You can thank Fippo for making me write this one.
Microsoft ended last week with an announcement of sorts on their Edge dev blog, indicating that H.264/AVC support for ORTC is now available in Edge.
- Yes. It is ORTC and not WebRTC
- Yes. It is only behind a runtime flag
- Yes. It is only on Edge. No IE
But then again, it is the only way today (or at least tomorrow) to get a video call running cross browser between Firefox, Chrome and Edge. VP8 or VP9 gets you as far as Chrome and Firefox.
Which got me to this one over here. Edge support for H.264 in ORTC isn’t much. It isn’t even interesting in the bigger scheme of things (Edge has literally no market share compared to the other browsers, so why bother with it?). And still it marks a turning point – one in which we can all ask ourselves what video codec should we be leaning towards if we started developing a product that uses WebRTC today?
Last year, the answer would have been “VP8”.
A few months ago, it was, “it depends”.
Today, it will lean towards “H.264, unless you must use VP8”.
So… which of these video codecs should you use in your application? Here’s a free mini video course to help you decide.
Here are 4 reasons why this is happening:
#1 – Browser interop baseline
If you want your service to get the most coverage on as many browsers as possible and you need video, then H.264 is the way to go. In a few months, H.264 will get official support by all of these vendors and that will be the end of it. Furthermore, you can expect Apple to use H.264 first and contemplate VP8 – same as Microsoft is doing now with Edge.
#2 – Mobile
Mobile devices like H.264 more than they like VP8. Video codecs take up a lot of resources. To overcome this, mobile handsets use hardware acceleration for video codecs. They all have H.264 video acceleration (though you can’t always gain access to it as a developer). Many of them don’t even know how to spell VP8. This boils down to WebRTC implementations on mobile needing to implement VP8 using software.
Some developers ended up replacing VP8 with H.264 on mobile just because of this reason. Especially for mobile only products.
While I am sure support for VP8 is improving in new chipsets, there’s this pesky issue of supporting the billion and more devices that are already out there. And now that all browsers support H.264 in one way or another, what incentive do developers needing to support mobile apps have to use VP8?
#3 – Legacy video systems
All them video conferencing systems? They use H.264. Most don’t have VP8. Not even in their latest released products. The way they end up supporting WebRTC until today is via a specialized gateway, on the MCU or not at all.
Transcoding was one of the main barriers to getting WebRTC to legacy video systems. It just costs a lot. It would have been easier to just go H.264 all the way. Which is what is now available.
It is one of the reasons why Cisco first worked on Firefox with Spark. It made a decision to use H.264 for WebRTC instead of transcoding from VP8.
#4 – Streaming
Over 60% of the Internet traffic is video. Most of it isn’t real time video, but rather the YouTube or Netflix kind. Passive consumption.
Video streaming today is predominantly H.264 based, and at times VP9 (=YouTube whenever possible).
To get video content on an iPhone device, HLS is required, and that again means H.264.
So again we are left with the alternative of either transcoding our WebRTC generated content to H.264 when we want to stream it out – or to create it using H.264 to begin with.
Do you even care?
If your service is a 1:1 calling service with no server side media processing, then you shouldn’t even care. In such a case, whatever the browsers end up negotiating will be good enough for you (and most probably the best alternative for that specific situation).
Those who invested in server side media processing, be it recording, mixing, routing – have made investments that are targeted at VP8. Modifying these to work with H.264 as well may not be trivial. For them, the decision of switching to H.264 is a harder one to make, but one that needs to be addressed.
The Future of Video Coding in WebRTC
Once we step into the future, we see VP9. And the SVC flavor of VP9.
And then there’s the Alliance of Open Media and the work they are doing towards a widely accepted next gen royalty free video codec. So much so that we now have a new video codec war in WebRTC: AV1 vs HEVC.
For the record, I rather hate H.264 and what it stands for. But now I must accept that it is here to stay and grow with WebRTC.
So… which of these video codecs should you use in your application? Here’s a free mini video course to help you decide.
Excellent write-up, thank you for doing this!
I disagree slightly. Mostly on the impact this has. Yes, this allows video calls between Edge and Firefox/Chrome. That turned out to be a no-brainer, took me less than a day between the announcement and actually running this. The impact will be a lot less than you’d expect.
First, there is a lot of calls using 1-1. For these and ignoring things like recording, codecs are negotiated. H264 will be the interop baseline. I do expect the majority of calls to continue to be using VP8, with VP9 ramping up slowly. How fast Edge interop is going to ramp here is a question, this also depends on Chrome unflagging H264 support.
Then there is multiparty. Just because H264 is the baseline all participants will support that does not mean the majority of your calls will be H264 either. Switching dynamically based on the best possible codec supported by everyone has been something at least I have worked on for quite a while now. Given the bugs i’ve reported in that area I seem to be the only person doing this however.
Advanced multiparty (read: simulcast) is a different beast. Despite the theory of SFUs being codec-agnostic, in reality you need to know _something_ about the codec. Like for example whether a packet contains a keyframe. Or the temporal layer. That requires some effort but then this is mostly about teaching your SFU those tricks and then doing them regardless of codec.
Fippo, as always – thanks for your insights – and you are of course correct 🙂
In quality comparisons, I’ve seen vp8/vp9 outshine h.264 even when hardware supported, particularly on latency, which is particularly annoying when that delays the audio. What was your experience when comparing h.264 with edge and chrome to a chrome or Firefox only call, fippo?
https://twitter.com/HCornflower/status/736456618758574080 — even though I suspect that was a CPU issue.
I’ve seen bad quality in Chrome-Chrome/Firefox H264 calls while performing some of the https://github.com/fippo/testbed tests.
It will be very interesting to do some large-scale A/B testing obviousy 🙂
I totally agree that consolidation around h.264 will be a huge thing for live streaming apps over WebRTC. Smartphones as you rightly pointed out, have native hardware encoding for the protocol, and HLS is a legitimate way to deliver the content.
However regarding HLS, what we’ve been seeing with our customers is a move away from it as a delivery protocol simply because the high amount of latency it introduces. This ends up meaning that WebRTC is the preferred protocol for one to many delivery when you want to have interactive features that require low latency. Some examples of this are a live text chat along with the stream and multiparty conference style broadcasts (like Blab).
Our approach with the upcoming release of Red5 Pro (http://red5pro.com) is to provide a JavaScript SDK which has a series of fallbacks depending on the capabilities of the browser: first WebRTC, then RTMP/Flash, and finally if all others aren’t there, HLS. I will write up a blog post today on this, as I think it’s relevant to the conversation.
Thanks as always for your insights Tsahi!
Thanks Chris
While I am following the Red5 Pro blog, please comment here back again with a link to that piece once written 🙂
Awesome! Yes, here’s my response on the Red5 Pro blog.
https://blog.red5pro.com/webrtc-the-delivery-protocol-for-live-streams-of-the-future/
H264 wins only in theory. In real world it is different. H264 with webrtc is not supported, if you have no hw decoder. But even you have, it is not used due to licence reasons. And if it works, it is unstable almost on every mobile and you can shoot down the browser. On desktop, it works well. We have switched from h264 to vp8 and we are happy now even we if loose safari support.
Thanks for sharing David!