Managing Client side CPU in WebRTC

Understand the significance of WebRTC metrics in optimizing client-side performance, focusing on CPU and network issues.

Client-side CPU management in WebRTC isn’t common. You’ll find it in “high end” applications but not much elsewhere. Why? Because we all focus on network issues, trying to optimize our way there. It is assumed that once the CPU gets overloaded, the network statistics will also suffer, and then self correct.

It happens.

But not always.

And not quickly enough.

Let’s go down this rabbit-hole of client side CPU management in WebRTC.

TL;DR video explainer

If you’d rather watch an explanation of this, or something similar, then my recent chat with Arin Sime was all about CPU in WebRTC:

WebRTC client resources to look after

If we want to understand the health and behavior of a WebRTC client, then the main areas to look at are network, CPU and memory. In this order.

💩 If you’re sending too much over the network – you’re screwed.

🗑️ If you’re sending too little over the network – you’re missing an opportunity for improved media quality.

Most of us are likely to stop their journey here. And we’re ok with how WebRTC transparently adapts to the available bandwidth.

Why? Because it is darn hard to fine tune and optimize for the network alone, and oh – when you do that – you also inadvertently also “fix” CPU and memory issues. Simply because the same solutions on the network (reducing bitrate) is going to assist with CPU and memory issues as well.

There are a few problems with this approach though:

• The sooner you know of a problem the sooner you can fix it
  • The “time to fix” is also the time the user will notice problems
  • So if you find about a problem indirectly, it is likely to occur after the problem started, and later than you could have caught it if you were looking for it directly
  • End result? A user experience that is subpar to what can be achieved in the industry
• Sometimes, network statistics isn’t going to find or enable you to fix the problem. Not really
  • In such edge cases, you will simply have bad media quality
  • And users will complain
  • It will take you longer to reach a conclusion that there is a real problem

Due to this, my approach is to do the following:

💡 Actively monitor fo both network and CPU

💡 Make sure from the get go that the network isn’t overburdened because of a mindset of overabundance – I try to balance the video sent and received before starting the optimization journey

Figuring out the network is… simple (?)

When it comes to figuring out client side WebRTC network issues, things are rather simple – at least to begin with.

You can check the big 3: packet loss, round-trip time and jitter

Do that for all incoming and outgoing media streams. From these, you can deduce the media quality for the most part.

Here’s the thing though – you need to remember two nagging gaps here:

1. You now know there’s a problem related to the network and that media quality is suffering. But you don’t know the root cause – so how are you going to solve this for the user?
2. If you could know slightly earlier that there might be a problem – wouldn’t it make sense to start changing behavior so as to fix issues before they manifest?

Which is why network metrics are great, but not enough.

What you need is more data. Some of that data is going to be in the form of understanding the CPU’s load.

Why we can’t get WebRTC CPU state off a web browser

Privacy.

Web browsers are just too powerful. When you use a browser, the server you are connected to knows quite a lot about you. It knows the location you’re at, the operating system you are using, the size of your screen’s resolution, etc.

There’s a cat and mouse game played between vendors who want to know more about their users and to be able to fully identify them versus web browser vendors who want to keep their users as anonymous as their users choose to be, sharing only what they must at each step.

The idea goes that every new browser capability gives vendors more insights and more ability to identify the user. So there’s a balance that is being played out here. Like the time when an exploit in WebRTC was found enabling anyone to start mapping local networks – that one was clamped down a bit by moving to mDNS addresses where it mattered.

CPU is similar in a way. It is assumed that knowing the percentage of CPU that is being used isn’t good for the user’s privacy. This is one of the main reasons why detailed CPU information isn’t passed directly via Javascript to developers.

Instead, we have to make do with lower granularity solutions where we glean estimates of the CPU’s state. And as usual in these things, getStats() is going to be our best friend.

3 ways for estimating WebRTC client-side CPU state

Lacking a direct approach, we’re left with estimating the CPU load. There are a few ways to do that. I’ll share here the 3 most common approaches, which can also be viewed as best practices for this.

For the most part, you’re going to need to use all of them…

Remember – we’re not going to know that “we’re at 90% CPU load”, but rather that the CPU is likely “working harder than we want”. Estimating…

Video encode time

The most resource intensive task for a WebRTC client is video encoding (I am ignoring application related AI fairy dust here). This is why it is the easiest one to use to figure out CPU load.

The idea is this – if we know how long it takes to encode a video frame, we know how hard it is for the CPU. We’re not really interested in the percentage of CPU used, but rather in how much of what we have gets invested in video encoding.

How do we do this?

Using getStats, we take the totalEncodeTime and divide it by framesEncoded. The result is going to be the average time to encode a single video frame.

Let’s make use of this –

• If we are doing 30 fps, then we are aiming to send a new frame out to the network every ~33 milliseconds (1000/30 ~= 33)
• If it takes longer than 33 milliseconds to encode a video frame, then we’re out of our league – we’re encoding the frames slower than we need to send them to the network
• We also want to “leave” some of that CPU power for things other than encoding. So we want the encode time per frame to be considerably lower than that 33 milliseconds here
• As a rule of thumb, for most use cases, a third of the time we have should be a good measurement. So anything that is… say… 12 milliseconds or less for single frame encode time would be good for us and anything higher – a red flag (12 milliseconds because 33/3 = 11)

Video decode time

The other part of video processing is video decoding.

So we do the same for video decoding – we take the totalDecodeTime and divide it by framesDecoded. And the result is the average time to decode a single video frame.

Similarly to what we did for the encoding, we are going to calculate what our intent is based on the frames per second. Here though, it makes more sense to divide by 4 and not 3 – the decoder needs less resources than the encoder.

For a “normal” 30 fps stream, we’d keep an eye on anything above 9 milliseconds (33/4 = 8.25).

There are a few important things to say here:

• The decoder is helpless if video is shoved down its throat. So if you want to reduce bitrate, framerate or resolution – you’ll need to ask the remote side to do so – either as part of the mechanisms WebRTC gives you or out of scope of WebRTC, via signaling
• You may have more than a single decoder in SFU group calls. Accommodate for that in your assumptions and calculations here
• Sometimes, there’s no video encoder. In such a case, the video decode time is going to be your best friend to figure out CPU load

Quality Limitation

I’ve placed this one last:

On one hand, it is the most straightforward one with a clear indicator that things are getting hairy due to the CPU.

On the other hand, you can only use it for scenarios where there’s outbound video. Oh – and if your video is muted, the metrics also don’t work.

I am obviously referring here to the qualityLimitationReason metric in getStats.

It is rather straightforward to use – per video frame encoded, it indicates why that frame was “limited” (read that as “hey, this is libWebRTC here, I had to reduce the bitrate of the video encoder because of X). The reasons?

• none – nothing got limited in any way – you’re living the life (this is how we like things around here with WebRTC)
• network – the network isn’t great, so we had to reduce bitrate (that’s a bandwidth estimation thing)
• cpu – the reason you’re here reading this. The user’s CPU isn’t having a great time and needs less stress in its life
• other – a generic catchall that I don’t remember seeing

What do you do? Just check if cpu was the reason and make sure it isn’t the reason too often. Assume it will/should flare out in real life scenarios (because you don’t control what devices do in their own free time), but keep this monitored properly, to be sure this isn’t because of some coding logic on your end (like an AI feature, trying too hard to please the user only to ruin his day by overloading the CPU).

The future of CPU load monitoring in WebRTC

What we’ve got so far is estimates. Trying to figure what’s going on through heuristics and hints that we glean out of getStats(). Wouldn’t it be nice if there was a better, more straightforward way?

Of course…

But life isn’t like that. Yet.

There’s an origin trial that Google Chrome has been running for quite some time now known as Compute Pressure API. The intent behind it is to allow web applications to register to a specialized compute pressure callback that would indicate the state of the CPU.

The main use cases, based on the trial’s document page? Video conferencing and video games.

It is too early for most of us to start using this new API – and it is too early to tell if this would end up as an official API across all web browsers. But it is definitely something to track and wait for to happen.

Until then? Use the metrics in getStats() and build your own strategy on how to deal with them.

The rtcStats approach to CPU load troubleshooting

With rtcstats.com, what we’re doing here for CPU load is rather straightforward:

1. We collect all getStats() metrics, so you should be covered
2. There are Observations that cater for all 3 techniques mentioned in this article: high decode time, high encode time and high CPU limitation
3. Other Observations we have cover many more video related quality issues, making it easier to correlate and find the root causes
4. We’re tracking that Compute Pressure API, so once it makes sense, we’ll add support for it as well

Did I mention that most of what we do at rtcStats is open source and free to use?

Now that you know the problem, how do you solve it?

Ok. CPU. We need to look at it. And not only rely on network stats.

What is it that you’re going to do about this from now on?

Need any help? Leave me a comment or reach out to me 😉