WebRTC Codec Wars: Rebooted

03/09/2015

The beginning of the end of HEVC/H.265 video codec.

WebRTC's codec wars

On September 1st the news got out. There’s a new group called Alliance for Open Media. There have been some interesting coverage about it in the media and some ramifications that already got published. The three most relevant pieces of news I found are these:

I’ve written about the pending codec wars just a week ago on SearchUC, concluding that all roads lead to a future with royalty free video codecs. This was before I had any knowledge on the announcement of the open media alliance. This announcement makes this future a lot more possible.

What I’d like to do here, is to cover some aspects of where this is headed and what it tells us about the players in this alliance and the pending codec wars.

The Press Release

Let’s start off with the alliance’ initial press release:

This initial project will create a new, open royalty-free video codec specification based on the contributions of members, along with binding specifications for media format, content encryption and adaptive streaming, thereby creating opportunities for next-generation media experiences.

So the idea is to invent a new codec that is royalty-free. As Chris pointed out, this is hard to impossible. Cisco in their own announcement of their new Thor codec made it quite clear what the main challenge is. As Jonathan Rosenberg puts it:

We also hired patent lawyers and consultants familiar with this technology area. We created a new codec development process which would allow us to work through the long list of patents in this space, and continually evolve our codec to work around or avoid those patents.

The closest thing to a “finished good” here is VP9 at the moment.

Is the alliance planning on banking on VP9 and use it as their baseline for the specification of this new codec, or will they be aiming at VP10 and a clean slate? Mozilla, a member company in this alliance, stated that they “believe that Daala, Cisco’s Thor, and Google’s VP10 combine to form an excellent basis for a truly world-class royalty-free codec.”

Daala takes a lot of its technologies from VP9. Thor is too new to count, and VP10 is just a thought compared to VP9. It makes more sense that VP9 would be used as the baseline; and Microsoft’s adoption of VP9 at that same timeframe may indicate just that intent. Or not.

The other tidbit I found interesting is the initial focus in the statement:

The Alliance’s initial focus is to deliver a next-generation video format that is:

  • Interoperable and open;
  • Optimized for the web;
  • Scalable to any modern device at any bandwidth;
  • Designed with a low computational footprint and optimized for hardware;
  • Capable of consistent, highest-quality, real-time video delivery; and
  • Flexible for both commercial and non-commercial content, including user-generated content.

Would be easier to just bio-engineer Superman.

Jokes aside, the bulleted list above are just table-stakes today:

  • Interoperable and open
    • Without interoperability a codec has no life
    • Openness is what you do in an initiative like this one
  • Optimized for the web
    • People consume video over IP these days. This is where the focus should be
    • It also hints for embedability in web browsers, and having Google, Microsoft and Mozilla on this alliance couldn’t hurt
  • Scalable to any modern device at any bandwidth
    • Scalability here means many things. SVC for one, but that’s just a single feature out of the list of necessary needs
    • Modern devices means that anything that is built probably before 2012 or even 2014 is going to be ignored. With current lifecycle of smartphones, that seems reasonable
    • Any bandwidth means it needs to support crappy internet connections but also 4K resolutions and above
  • Designed with a low computational footprint and optimized for hardware
    • This one is going to be tough. Each codec generation takes 2-3 times the computational footprint of its predecessor. I am not sure this can be met if the idea is to displace something like H.265
    • Optimized for hardware is a wink to hardware vendors that they need to support this as well. Having Intel is nice, but they are almost the non-player in this market (more on that later)
  • Capable of consistent, highest-quality, real-time video delivery
    • Guess what? Everyone wants that for any time of a video codec
  • Flexible for both commercial and non-commercial content, including user-generated content
    • This talks about licensing and royalties. Free should be the business model to aim for, though the language may well translate to royalty payments, though at a lower rate than what MPEG-LA and HEVC Advance are trying to get

High goals for a committee to work on.

It will require Cisco’s “cookbook”: a team comprised of codec engineers and lawyers.

The Members

What can we learn from the 7 initial alliance members? That this was an impossible feat and someone achieved just that. Getting these players into the same table while leaving the egos out of the room wasn’t easy.

Amazon

Amazon is new to video codecs – or codecs and media. They have their own video streaming service, but that’s about it.

Their addition into this group is interesting in several aspects:

  • The Amazon Instant Video service has its place. Not the dominant service, but probably big enough so it isn’t ignored. Added to Netflix and YouTube, it adds its weight
  • More interestingly, how will AWS be affected? Their Amazon Elastic Transcoder for example, or the ability to host real time media processing services on top of AWS

Cisco

Cisco is a big player in network gear and in unified communications. It has backed H.264 to date, mainly due to its own deployed systems. That said, it is free to pick and choose next generation codecs. While it supports H.265 in its high-end telepresence units, it probably saw the futility of the exercise continuing down this path.

Cisco though, has very little say over future codecs adoption.

Google

Google needs free codecs. This is why it acquired On2 in the first place – to have VP8, VP9 and now VP10 compete with H.26x. To some extent, you can point the roots of this alliance to the On2 acquisition and the creation as webm as the first turning point in this story.

For Google, this means ditching the VPx codec branding, but having what they want – a free video codec.

The main uses for Google here are first and foremost YouTube and later on WebRTC. Chrome is the obvious vehicle of delivery for both.

I don’t see Google slowing down on their adoption of VP9 in WebRTC or reducing its use on YouTube – on the contrary. Assume the model played out here will be the same one Google played with SPDY and HTTP/2:

  • SPDY was Google’s proprietary transport mechanism to replace HTTP/1.1. It was later used as the baseline of HTTP/2
  • VP9 is Google’s proprietary video codec to replace H.26x. It is now being used as the baseline of the next generation video codec to displace H.265

To that end, Google may well increase their team size to try and speed up their technology advancement here.

Intel

Intel is trying for years now to conquer mobile with little to show for its efforts. When it comes to mobile, ARM chipsets rule.

Intel can’t really help with the “any modern device” part of the alliance’s charter, but it is a good start. They are currently the only chipset vendor in the alliance, and until others join it, there’s a real risk of this being a futile effort.

The companies we need here are ARM, Qualcomm, Broadcom and Samsung to begin with.

Microsoft

Microsoft decided to leave the H.26x world here. This is great news. It is also making the moves towards adopting WebRTC.

Having Google Chrome and Microsoft Edge behind this initiative is what is necessary to succeed. Apple is sorely missing, which will most definitely cause market challenges moving forward – if Apple doesn’t include hardware acceleration for this codec in their iOS devices, then a large (and wealthy) chunk of the consumer market will be missing.

Every day that passes it seems that Microsoft is acting like a modern company ready for this day and age as opposed to the dinosaur of the 90’s.

Mozilla

Mozilla somehow manages to plug itself into every possible initiative. This alliance is an obvious fit for a company like Mozilla. It is also good for the alliance – 3 out of 4 major browser players behind this initiative is more than we’ve seen for many years in this area.

Netflix

Netflix started by adopting H.265 for their 4K video streaming. It seemed weird for me that they adopted H.265 and not VP9 at the time.  I am sure the latest announcements coming out of HEVC Advance about licensing costs for content streaming has caused a lot of headache at Netflix and tipped the scale towards them joining this alliance.

If you are a content provider operating at Netflix scale with their margins and business model, the greedy %0.5 gross revenue licensing of HEVC Advance becomes debilitating.

With YouTube, Amazon and Netflix behind this alliance, you can safely say that web video streaming has voiced their opinion and placed themselves behind this alliance and against HEVC/H.265.

Missing in Action

Who’s missing?

We have 3 out of 4 browser vendors, so no Apple.

We have the web streaming vendors. No Facebook, but that is probably because Facebook isn’t as into the details of these things as either Netflix or Google. Yet.

We don’t have the traditional content providers – cable companies and IPTV companies.

We don’t have the large studios – the content creators.

We don’t have the chipset vendors.

Apple

Apple is an enigma. They make no announcements about their intent, but the little we know isn’t promising.

  • They have devices sold to think of. These devices support H.265 hardware acceleration, so they are somewhat committed to it. Hard to switch to another horse as a vertical integrator
  • Safari and WebKit are lagging behind when it comes to many of the modern web technologies – WebRTC being one of them
  • Apple owns patents in H.265 and are part of MPEG-LA. Would they place their bets in another alliance? In both at the same time? Contribute their H.265 patents to the Alliance for Open Media? Probably not

Once this initiative and video codec comes to W3C and IETF for standardization, will they object? Join? Implement? Ignore? Adopt?

Content providers

Content providers are banking around H.265 for now. They are using the outdated MPEG2 video codec or the current H.264 video codec. For them, migrating to H.265 seems reasonable. Until you look at the licensing costs for content providers (see Netflix above).

That said, some of them, in Korea and Japan, actually own patents around H.265.

Where will they be headed with this?

Content creators

Content creators wouldn’t care less. Or they would, as some of them are now becoming also content providers, streaming their own content direct-to-consumer in trials around unbundling and cord cutting.

They should be counting themselves as part of the Alliance for Open Media if you ask me.

Chipset vendors

Chipset vendors are the real missing piece here. Some of them (Samsung) hold patents around H.265. Will they be happy to ditch those efforts and move to a new royalty free codec? Hard to say.

The problem is, that without the chipset vendors behind this initiative it will not succeed. One of the main complaints around WebRTC is lack of support for its codecs by chipsets. This will need to change for this codec to succeed. It is also where the alliance needs to put its political effort to increase its size.

The Beginning of the End for HEVC/H.265

This announcement came as a surprise to me. I just finished writing my presentation for an upcoming TechTok with the same title as this post: WebRTC Codec Wars Rebooted. I will now need to rewrite that presentation.

UPDATE: Here’s the TechTok recording

This announcement if played right, can mean the end of the line for the H.26x video codecs and the beginning of a new effort around royalty free video codecs, making them the norm. The enormity of this can be compared to the creation of Linux and its effect on server operating systems and the Internet itself.

Making video codecs free is important for the future of our digital life.

Kudos for the people who dared dream this initiative and making it happen.

So… which of these video codecs should you use in your application? Here’s a free mini video course to help you decide.

Responses

Jeff - VoipDIY says:
September 3, 2015

Very interesting article, Tsahi. Lots of good points.

Reply
Yoav Nativ says:
September 3, 2015

Great information – very interesting!

Reply
Lawrence Byrd says:
September 4, 2015

Tsahi – Good article following your prescience previously at SearchUC! But FYI your three links in the initial bullets all go to the same CNet article (robbing Chris to pay Stephen…).

Reply
UserName says:
September 7, 2015

[quote]Daala takes a lot of its technologies from VP9[/quote]

This statement sounds untrue to me. AFAIK, one of Daala’s main selling points is the fact that it uses completely different (barnd-new or old-but-never-really-used-for-video) technologies from mainline codecs (including VPx and H.26x), thus simultaneously having a chance to deliver better compression without increasing complexity *and* avoiding whole swaths of patents. I’ve never heard one word said about any VPx connections in Daala.

Reply
Roey says:
September 8, 2015

Hi Tsahi,

great article!

I wonder how this subject affect companies who build their own webRTC networks.

also, if i understand right, google already own V8 and everyone may use it… so why the sudden need in a new open source codec?

Reply
    Tsahi Levent-Levi says:
    September 9, 2015

    Thanks Roey.

    VP8 is nice, but it is on par with H.264 for the most part.
    The next generation of codecs are better – while they eat more CPU, they compress more so enable you to send video with better quality for less bitrate.

    The challenge with VP8 is hardware acceleration today – support for it is still scattered. If we want the next video codec to work, we need hardware support for it from the start.

    Reply
      Roey says:
      September 9, 2015

      Do other propriety codecs are supporting hardware acceleration?
      if so, how come they are patent protected? how can someone write a patent on an algorithms that were already implemented by hardware vendors?

      Reply
        Tsahi Levent-Levi says:
        September 9, 2015

        A group effort.

        Patents can usually find their way into standards if the patent owner is willing to share it via a mechanism called FRAND – essentially saying he will license that patent to anyone who wants it at a fair price (fair=not discriminating one company versus others).

        H.26x is a hugely popular family of codecs that have patents on them. They are also supported by most hardware acceleration of video codecs out there.

        Up until recently, the mere idea of a patent-free or non-royalty-bearing video codec was ridiculous at best.

        Reply
Lawrence Byrd says:
September 10, 2015

Tsahi gave a great Meetup talk on this on 9-Sep at TokBox and has posted his (new!) slides on this at: http://www.slideshare.net/tsahil/webrtc-codec-wars-rebooted

FYI, I also found the following external article on the topic interesting: http://www.streamingmedia.com/Articles/Editorial/Featured-Articles/Commentary-The-Streaming-Industry-Gangs-up-on-HEVC-with-the-Alliance-for-Open-Video-106115.aspx

Reply
What to hope for from the Alliance for Open Media - Upperside Blog says:
September 17, 2015

[…] The Alliance for Open Media is a surprising and positive development. Who would have thought these rivals could come to terms about about something like video coding, doing it with a free royalties model attached to it. In order to make a dent, this alliance must attract the chipset manufacturers – they are the ones with the real work here as they need to include hardware acceleration for this new video codec in the process – something that Google’s VP8 was always criticized of lacking. I wrote a longer analysis of this topic here: https://bloggeek.me/webrtc-codec-wars-rebooted/ […]

Reply
The WebRTC Codec Conundrum - Corporate Blog - Dialogic - Dialogic Exchange Network says:
September 18, 2015

[…] a prolific advocate for WebRTC, wrote in his BlogGeek.me blog this last week about the recent resurgence of the “codec wars”, […]

Reply
nwgat says:
July 19, 2016

AMD, NVIDIA and ARM is now apart of the rebel alliance :=)
so thats basicly whole x86 computer market (all cpus in your computer that have intergrated graphics should play AV1 in the future)

and the whole graphics card market (AMD and NVIDIA)

also ARM is pretty huge (if they can make it simple to implement hw designs for oems)

btw why mozilla is a part of the rebel alience? well many of the xiph people aka those that does theora, vorbis, flac and opus (yes that one with skype/microsoft) are employed by mozilla

Reply

Comment