WebRTC Codec Wars: Rebooted

The beginning of the end of HEVC/H.265 video codec – or maybe a redefinition of the war towards HEVC vs AV1?

On September 1st the news got out. There’s a new group called Alliance for Open Media. There have been some interesting coverage about it in the media and some ramifications that already got published. The three most relevant pieces of news I found are these:

• Stephen Shankland over at CNET makes the most intelligent and thorough review of this announcement out of all the large media outlets. Well worth the read
• Chris Kranky decided to pitch in and explain how patents work with codecs, and how this alliance is doomed to fail. I disagree with the conclusions, but the story is solid
• Microsoft already plans on supporting VP9 video codec in their Edge browser

I’ve written about the pending codec wars just a week ago on SearchUC, concluding that all roads lead to a future with royalty free video codecs. This was before I had any knowledge on the announcement of the open media alliance. This announcement makes this future a lot more possible.

For the time being, it will be interesting to see what will people do. Wait it out until AV1 becomes a reality or adopt VP9 codec for their WebRTC application.

What I’d like to do here, is to cover some aspects of where this is headed and what it tells us about the players in this alliance and the pending codec wars.

The Press Release

Let’s start off with the alliance’ initial press release:

This initial project will create a new, open royalty-free video codec specification based on the contributions of members, along with binding specifications for media format, content encryption and adaptive streaming, thereby creating opportunities for next-generation media experiences.

So the idea is to invent a new codec that is royalty-free. As Chris pointed out, this is hard to impossible. Cisco in their own announcement of their new Thor codec made it quite clear what the main challenge is. As Jonathan Rosenberg puts it:

We also hired patent lawyers and consultants familiar with this technology area. We created a new codec development process which would allow us to work through the long list of patents in this space, and continually evolve our codec to work around or avoid those patents.

The closest thing to a “finished good” here is VP9 at the moment.

Is the alliance planning on banking on VP9 and use it as their baseline for the specification of this new codec, or will they be aiming at VP10 and a clean slate? Mozilla, a member company in this alliance, stated that they “believe that Daala, Cisco’s Thor, and Google’s VP10 combine to form an excellent basis for a truly world-class royalty-free codec.”

Daala takes a lot of its technologies from VP9. Thor is too new to count, and VP10 is just a thought compared to VP9. It makes more sense that VP9 would be used as the baseline; and Microsoft’s adoption of VP9 at that same timeframe may indicate just that intent. Or not.

The other tidbit I found interesting is the initial focus in the statement:

The Alliance’s initial focus is to deliver a next-generation video format that is:

• Interoperable and open;
• Optimized for the web;
• Scalable to any modern device at any bandwidth;
• Designed with a low computational footprint and optimized for hardware;
• Capable of consistent, highest-quality, real-time video delivery; and
• Flexible for both commercial and non-commercial content, including user-generated content.

Would be easier to just bio-engineer Superman.

Jokes aside, the bulleted list above are just table-stakes today:

• Interoperable and open
  • Without interoperability a codec has no life
  • Openness is what you do in an initiative like this one
• Optimized for the web
  • People consume video over IP these days. This is where the focus should be
  • It also hints for embedability in web browsers, and having Google, Microsoft and Mozilla on this alliance couldn’t hurt
• Scalable to any modern device at any bandwidth
  • Scalability here means many things. SVC for one, but that’s just a single feature out of the list of necessary needs
  • Modern devices means that anything that is built probably before 2012 or even 2014 is going to be ignored. With current lifecycle of smartphones, that seems reasonable
  • Any bandwidth means it needs to support crappy internet connections but also 4K resolutions and above
• Designed with a low computational footprint and optimized for hardware
  • This one is going to be tough. Each codec generation takes 2-3 times the computational footprint of its predecessor. I am not sure this can be met if the idea is to displace something like H.265
  • Optimized for hardware is a wink to hardware vendors that they need to support this as well. Having Intel is nice, but they are almost the non-player in this market (more on that later)
• Capable of consistent, highest-quality, real-time video delivery
  • Guess what? Everyone wants that for any time of a video codec
• Flexible for both commercial and non-commercial content, including user-generated content
  • This talks about licensing and royalties. Free should be the business model to aim for, though the language may well translate to royalty payments, though at a lower rate than what MPEG-LA and HEVC Advance are trying to get

High goals for a committee to work on.

It will require Cisco’s “cookbook”: a team comprised of codec engineers and lawyers.

The Members

What can we learn from the 7 initial alliance members? That this was an impossible feat and someone achieved just that. Getting these players into the same table while leaving the egos out of the room wasn’t easy.

Amazon

Amazon is new to video codecs – or codecs and media. They have their own video streaming service, but that’s about it.

Their addition into this group is interesting in several aspects:

• The Amazon Instant Video service has its place. Not the dominant service, but probably big enough so it isn’t ignored. Added to Netflix and YouTube, it adds its weight
• More interestingly, how will AWS be affected? Their Amazon Elastic Transcoder for example, or the ability to host real time media processing services on top of AWS

Cisco

Cisco is a big player in network gear and in unified communications. It has backed H.264 to date, mainly due to its own deployed systems. That said, it is free to pick and choose next generation codecs. While it supports H.265 in its high-end telepresence units, it probably saw the futility of the exercise continuing down this path.

Cisco though, has very little say over future codecs adoption.

Google

Google needs free codecs. This is why it acquired On2 in the first place – to have VP8, VP9 and now VP10 compete with H.26x. To some extent, you can point the roots of this alliance to the On2 acquisition and the creation as webm as the first turning point in this story.

For Google, this means ditching the VPx codec branding, but having what they want – a free video codec.

The main uses for Google here are first and foremost YouTube and later on WebRTC. Chrome is the obvious vehicle of delivery for both.

I don’t see Google slowing down on their adoption of VP9 in WebRTC or reducing its use on YouTube – on the contrary. Assume the model played out here will be the same one Google played with SPDY and HTTP/2:

• SPDY was Google’s proprietary transport mechanism to replace HTTP/1.1. It was later used as the baseline of HTTP/2
• VP9 is Google’s proprietary video codec to replace H.26x. It is now being used as the baseline of the next generation video codec to displace H.265

To that end, Google may well increase their team size to try and speed up their technology advancement here.

Intel

Intel is trying for years now to conquer mobile with little to show for its efforts. When it comes to mobile, ARM chipsets rule.

Intel can’t really help with the “any modern device” part of the alliance’s charter, but it is a good start. They are currently the only chipset vendor in the alliance, and until others join it, there’s a real risk of this being a futile effort.

The companies we need here are ARM, Qualcomm, Broadcom and Samsung to begin with.

Microsoft

Microsoft decided to leave the H.26x world here. This is great news. It is also making the moves towards adopting WebRTC.

Having Google Chrome and Microsoft Edge behind this initiative is what is necessary to succeed. Apple is sorely missing, which will most definitely cause market challenges moving forward – if Apple doesn’t include hardware acceleration for this codec in their iOS devices, then a large (and wealthy) chunk of the consumer market will be missing.

Every day that passes it seems that Microsoft is acting like a modern company ready for this day and age as opposed to the dinosaur of the 90’s.

Mozilla

Mozilla somehow manages to plug itself into every possible initiative. This alliance is an obvious fit for a company like Mozilla. It is also good for the alliance – 3 out of 4 major browser players behind this initiative is more than we’ve seen for many years in this area.

Netflix

Netflix started by adopting H.265 for their 4K video streaming. It seemed weird for me that they adopted H.265 and not VP9 at the time.  I am sure the latest announcements coming out of HEVC Advance about licensing costs for content streaming has caused a lot of headache at Netflix and tipped the scale towards them joining this alliance.

If you are a content provider operating at Netflix scale with their margins and business model, the greedy %0.5 gross revenue licensing of HEVC Advance becomes debilitating.

With YouTube, Amazon and Netflix behind this alliance, you can safely say that web video streaming has voiced their opinion and placed themselves behind this alliance and against HEVC/H.265.

Missing in Action

Who’s missing?

We have 3 out of 4 browser vendors, so no Apple.

We have the web streaming vendors. No Facebook, but that is probably because Facebook isn’t as into the details of these things as either Netflix or Google. Yet.

We don’t have the traditional content providers – cable companies and IPTV companies.

We don’t have the large studios – the content creators.

We don’t have the chipset vendors.

Apple

UPDATE: Apple have joined the Alliance of Open Media.

Apple is an enigma. They make no announcements about their intent, but the little we know isn’t promising.

• They have devices sold to think of. These devices support H.265 hardware acceleration, so they are somewhat committed to it. Hard to switch to another horse as a vertical integrator
• Safari and WebKit are lagging behind when it comes to many of the modern web technologies – WebRTC being one of them
• Apple owns patents in H.265 and are part of MPEG-LA. Would they place their bets in another alliance? In both at the same time? Contribute their H.265 patents to the Alliance for Open Media? Probably not

Once this initiative and video codec comes to W3C and IETF for standardization, will they object? Join? Implement? Ignore? Adopt?

Content providers

Content providers are banking around H.265 for now. They are using the outdated MPEG2 video codec or the current H.264 video codec. For them, migrating to H.265 seems reasonable. Until you look at the licensing costs for content providers (see Netflix above).

That said, some of them, in Korea and Japan, actually own patents around H.265.

Where will they be headed with this?

Content creators

Content creators wouldn’t care less. Or they would, as some of them are now becoming also content providers, streaming their own content direct-to-consumer in trials around unbundling and cord cutting.

They should be counting themselves as part of the Alliance for Open Media if you ask me.

Chipset vendors

UPDATE: Most chipset vendors are part of the alliance now. Qualcomm still isn’t.

Chipset vendors are the real missing piece here. Some of them (Samsung) hold patents around H.265. Will they be happy to ditch those efforts and move to a new royalty free codec? Hard to say.

The problem is, that without the chipset vendors behind this initiative it will not succeed. One of the main complaints around WebRTC is lack of support for its codecs by chipsets. This will need to change for this codec to succeed. It is also where the alliance needs to put its political effort to increase its size.

The Beginning of the End for HEVC/H.265

This announcement came as a surprise to me. I just finished writing my presentation for an upcoming TechTok with the same title as this post: WebRTC Codec Wars Rebooted. I will now need to rewrite that presentation.

UPDATE: Here’s the TechTok recording

This announcement if played right, can mean the end of the line for the H.26x video codecs and the beginning of a new effort around royalty free video codecs, making them the norm. The enormity of this can be compared to the creation of Linux and its effect on server operating systems and the Internet itself.

Making video codecs free is important for the future of our digital life.

Kudos for the people who dared dream this initiative and making it happen.