WebRTC Debugging & Monitoring

WebRTC troubleshooting is a mess. Stop delaying, start debugging! Discover the right tools and client-side observability to find the root cause, fast.

Have you ever tried debugging WebRTC applications? How about monitoring them? For many, monitoring is non-existent or just a line on the roadmap that gets postponed from one release to the next. And debugging? That’s what you do when there’s a problem from a customer. And then you muck around with delaying tactics and later by trying to look at metrics and decipher them.

Time to put some order into that thing called WebRTC debugging & monitoring…

Why WebRTC requires client-side debugging

When debugging WebRTC applications, much of your focus and effort is going to be placed on the client-side application.

Why?

Because that’s where the users reside and that’s where they are going to experience your service – an experience that can cause them to become your biggest fan or make them churn.

In WebRTC, the first question with every user issue is who is to blame?

• Is that the user’s device
• Or maybe the network he is using
• It might be our infrastructure. Maybe

How do you make that distinction? Especially when the knee-jerk reaction of the user is to blame the service and the knee-jerk reaction of the IT is to blame the user’s device or network…

The only real way to get to the bottom of an issue is by debugging the data directly from the user’s device. From there you can work your way towards finding the root cause and finger pointing at the real culprit.

Here’s how you get to debug WebRTC on various client-side entities 👇

WebRTC debugging on web browsers

Different browsers offer slightly different tools for debugging WebRTC. Olivier has a good overview of that. Me? I’ll focus on Chrome. That’s because of multiple reasons:

• Most users end up using Chrome
• Chrome has the best tooling in place for WebRTC debugging
• “Packaged” alternatives on both PC and mobile such as Webview and Electron are based on Chromium (which is Chrome for the purpose of understanding WebRTC)
• rtcstats and similar tools simulate Chrome’s collection closely, which means you can get to the same results across all web browsers

For web browsers, debugging means understanding how the application invokes and interacts with the WebRTC APIs as well as how the media engine of the WebRTC implementation in the browser is behaving. Anything more is a bonus.

The main challenge is to get the data from the browser to the engineer that needs to troubleshoot the issue. That’s because the information in the browser that is needed for debugging is transient in nature and isn’t collected by default in any way.

We will touch on solving the transient nature of webrtc-internal in a second, when we get to discuss the tools at your disposal.

WebRTC debugging in native applications

Native applications are trickier.

Here I’ll bunch together the following types:

• Applications built by using Google’s libWebRTC library directly, ported to your target operating system. Most likely iOS and Android
• Client-side applications built using Pion. A lot of these end up being server-side rendering (think cloud gaming), surveillance, IOT edge devices, etc
• Other 3rd party libraries – open source or proprietary – that are meant to interoperate with WebRTC in the browser

The challenge here is that some might not have the same metrics collected at all, or similar enough APIs to collect them through. The end result is needing other means of collection and analysis here.

Our suggestion? Stick to something that is libWebRTC or that closely resembles it when it comes to the interface given to the developers. That will make the debugging and monitoring piece later on a lot easier to manage across all the devices and environments you are likely to have.

WebRTC debugging: The tools at your disposal

The WebRTC ecosystem is a bit lacking in debugging tools. That said, there are a few out there for you to use 👇

webrtc-internals and getStats() as the ground truth

First things first. WebRTC has a great tool for us to use – getStats(). It gives us access to the innards of WebRTC by collecting metrics from all over the place.

Then there’s webrtc-internals – the Chrome debugging pane for everything WebRTC right inside the browser. It is great, but limited to the “here and how”.

Use these as your ground truth – a way to understand what is really going on with a WebRTC client implementation. Anything else is going to be a cheaper compromise and you really don’t want that – not if debugging and troubleshooting is top of mind for you.

Visualizing the data

Now that we can collect the metrics, it is time to visualize it.

The way I see it, there are only 4 tools out there:

1. webrtc-internals in Chrome has some really rudimentary implementation
2. The dump-importer which you can download and setup somewhere for your own use. The charts are better, but not fancy. They give you the data but don’t think about any creature comforts. This tool was built by Philipp for Philipp
3. Cyara’s testRTC has good visualization (I know… I built it). But you must be a paying customer to use it even if you want nothing else
4. rtcstats.com which frankly, has the best visualization for WebRTC debugging in the world. And you can use it freely

Other tools may offer solutions such as sending metrics to Grafana so you can use their dashboarding capabilities to chart stuff, but that’s no real visualization that is geared towards WebRTC.

Making real Observations and getting to root causes

Visualization is nice, but wouldn’t it be much nicer to have something that helps you in figuring out what the problems are by just saying it?

Here, I know of only 2 tools that do this:

1. Cyara’s testRTC which has its own capabilities in this space
2. rtcstats.com Observations and AI engine, offering immediate root cause analysis and powerful observations. What we did here was pour our knowledge and expertise into the tool, and then used that to train our AI engine

Visualization is great for an expert developer, but in many cases not as good for novice users. Having observations and root causes at your fingertips means democratising the debugging process itself and opening it up to more of your employees 👉 It frees your top WebRTC experts’ time…

WebRTC monitoring = client-side observability

Now that we looked at debugging tools, it is time to put some ground truth out there – if you want to monitor a WebRTC application for user experience you must use client-side observability.

Looking at server metrics, even if they do arrive from RTCP reports to the server just isn’t enough. It gives you a glimpse to what might be wrong, but it almost never is enough to decide the root cause or it. And RTCP is limited to network data so you get no information about decoding or CPU issues. It won’t direct you towards a resolution, only tell you there’s a potential problem.

When WebRTC came out, most monitoring solutions for WebRTC started from collecting metrics from the client-side, aggregating it on servers and looking at issues there.

Doing it in any other way means you’re missing out on your WebRTC monitoring experience.

What WebRTC metrics and logs should you collect

This is the big question.

WebRTC has tons of metrics. Lots of different metrics that can be collected at frequencies as low as every second. In some ways, that’s too much data. In other ways, not having parts of that information is going to leave you guessing.

Some services collect the minimal amount of information. Things like bitrate, packet loss, round trip time and latency. These services focus on telling you there’s a problem. But don’t give you the root cause or any path that will lead to solutions.

This means:

1. getStats() metrics. All the ones that can change throughout the duration of the session. In many ways, picking and choosing leading metrics is bound to fail as a technique to do effective troubleshooting in WebRTC
2. WebRTC events. All calls to the APIs and returned callbacks. Without these, being able to trace the logic of what took place in connectivity issues, or understanding the configuration for the session won’t be there for you
3. Machine data. Some machine data is available to you in the browser (and definitely in native applications). Collect it to give the bigger picture view. This is especially important when running after CPU and performance issues

Commercial solutions for WebRTC monitoring

There are a few commercial solutions out there that give you some monitoring capabilities:

• Cyara and Operata both offer monitoring solutions specific to call center agents work. Cyara also offers watchRTC for more generic WebRTC monitoring
• CPaaS and Video API vendors offer some monitoring or analytics. Usually these are geared towards their ability to troubleshoot issues and less towards your ability to understand what happened to users. It is more about showing network problems and less about showing WebRTC behaviors
• rtcstats.com has its own commercial model (check us out)

Open source alternatives to WebRTC monitoring

There are also a few open source alternatives when you want to implement your own WebRTC monitoring.

I suggest using the rtcstats client and server open source combo. It gives you the most amount of flexibility in my mind, along with the commercial and SaaS freemium to complement it. You own the data.

You should also look at ObserveRTC and Peermetrics. As far as I am aware, both are in the business of collecting partial data.

What about WebRTC server-side

My focus in WebRTC monitoring is the client-side. That’s because I look at user experience. WebRTC server-side monitoring is akin to the more general purpose APM (Application Performance Monitoring) market, where you don’t only look at CPU and memory, but also check for performance indicators that are application specific.

The only issue is that here, the application specific metrics are specific to the server and not the user. So you end up doing health checks and monitoring your server infrastructure, which is important, but it isn’t monitoring user experience. Not really.

This is why I won’t be expanding on this here – each WebRTC media server vendor/framework has its own set of health metrics it shares with the world, but up until today, I haven’t had any substantial case where a vendor asked to merge that data with client-side metrics collection in WebRTC to gain more user experience related insights…

When to start with WebRTC debugging & monitoring

The big question is where to start investing in WebRTC debugging and monitoring.

I’ll split my answer into two parts:

1. Put the underlying basic infrastructure on day one. That’s easy – just include rtcstats-js in your code and put up your own rtcstats-server installation in front of it – or not – just integrate it with a rudimentary ticketing system. This will give you debugging capabilities
2. Once you have your first paying customer – that’s when you should lean into systematic WebRTC monitoring. This is when I’d really go and make sure all sessions are properly collected and accounted for

If you are past the point of having paying customers and you aren’t monitoring WebRTC from the client-side then you’re running blind. When users complain – it starts a never ending saga of back and forth until the user gives up and churns or until you get what you need from them – all because you haven’t done the simple step of collecting it in advance.

rtcstats as the future of WebRTC debugging & monitoring

For me, rtcstats.com is the future of how WebRTC debugging and monitoring looks like:

• Client-side data collection at all times
• Telemetry owned by you and no one else
• Great visualization with powerful observations and an AI engine

Need any help?

If you’re looking into debugging, monitoring or optimizing a WebRTC application to get the best possible user experience and you don’t know where to start – contact me. If anything, I can give you a tip or two specific to your current challenge.