WebRTC test: How to go about testing WebRTC?

09/03/2020

WebRTC testing has many aspects to it. Many don’t systematically test their WebRTC applications while others test the wrong things. Here’s what you need to know about testing WebRTC.

WebRTC test

WebRTC has many moving parts to it. You have the user devices, signaling servers, the application server, TURN and STUN servers, sometimes media servers. The areas you need to focus in your WebRTC testing will be different than those of someone else and would depend on both the use case and the architecture chosen. In this article, I’ll try to list the various aspects you need to think about when building a test plan for your WebRTC application.

End to end WebRTC testing

At the end of the day, your users are going to interact with your service as a whole. This means that much of your testing should focus on end to end WebRTC testing.

End to End WebRTC test

Such tests need to be conducted in a fashion that is as close as possible to how your real users interact with your service.

Consider the following:

  1. Manual testing – your QA engineers testing the service as end users will. But bear in mind that you will need to handle aspects of different network conditions and locations, which greatly affect how WebRTC behaves
  2. Automated testing – you can automate some of the end to end tests using Selenium or SaaS platforms like testRTC (I am a co-founder there). These offer the ability to cover 80% of the scenarios you need to deal with at 20% of the time in a predictable and reproducible manner
  3. User testing – this happens a lot in the consumer space but also elsewhere. What I mean here is releasing the product to users or to a small percentage of the users and seeing what people complain about. Effectively using users as guinea pigs and cheap testing labor. It can work for some, but not for all products

BTW – using a CPaaS provider (=consuming WebRTC as an API from a third party) doesn’t mean you don’t need to conduct your own testing. You can never know if the use case you implemented is in one of the main scenarios that CPaaS provider is focusing on.

P2P testing

p2p testing as part of the webrtc test

Some services are simpler than others. If your service provides only 1:1 calls conducted in a peer-to-peer fashion without any media servers in the middle. This reduces the testing needed by quite a bit.

In P2P, the main concern you will have is the scale and uptime of your signaling server (the other important one is interoperability, but on that – later).

Since actual media doesn’t touch your infrastructure other than your TURN servers, testing it would be nice, but you’d be just doing the browser vendors’ job if that’s your main focus.

Check here for more on WebRTC P2P testing.

Mobile testing

mobile testing

Mobile is slightly different when it comes to WebRTC. And by that I mean non-browser testing – no mobile and elsewhere.

When you use a browser, someone else is in charge of the WebRTC implementation on the device. When you use your own application, on mobile, embedded or desktop – you are the one in charge of that WebRTC implementation. You are the one picking the WebRTC stack used, modifying it, updating it, etc.

This means you will need to spend more time in testing your mobile application as well, and sadly, there are no easy ways to automate these tests. WebRTC uses too much of the device to make it simple to automate mobile testing by the use of simulators or even by the use of real devices.

A lot of your mobile testing will be conducted manually when it comes to the communications part of it.

TURN server testing

turn server testing

TURN server testing requires forcing networks to block certain types of traffic so that STUN won’t work well. You will either need to change the peer connection configuration to enable that or to configure the machines used to work in a certain fashion, such as blocking UDP traffic.

Now, if you are outsourcing your TURN servers to a third party managed service provider, then you don’t need to have such extensive testing to it (assuming the provider you use is reputable enough that you can trust his testing).

Media server testing

media server testing

WebRTC media server testing can be isolated from the rest of the system. The reason to do that is in order to test behavior in different network conditions or when you want to stress test the service.

You may want to stress test your server by using synthetic traffic not originating from browsers (such as wiring one media server to another and sending traffic directly between them) in order to increase the scale of your tests while keeping your costs low. At other times, you may prefer using automated headless browsers for that purpose, to get as close as possible to real life conditions in your stress testing.

A word about stress testing –

These can be conducted in the end to end fashion or in a media server unit testing fashion. Both are valid. Aiming for the media server unit testing alternative makes sense if the application is complex in terms of its UI – you wouldn’t want to fail tests due to failures in automating the UI itself. Aiming for end to end testing gives you a higher confidence level as to potential bottlenecks throughout your system and not only in your media servers.

Functional and coverage testing

functional coverage testing

Dealing with all the edge cases and functionality of an application is important yet daunting. When many of these tests occur with no actual WebRTC session taking place, it is important to ask how do we treat these tests?

We’re 8 years into WebRTC. It is still a small niche, which means that most of the testing tools available out there don’t focus on WebRTC at all. If you plan to use a generic testing tool for all of your testing, including the WebRTC ones, then you will lose a lot of the functionality required for WebRTC (things like geographic spread, machine configurations, network conditions, etc).

If you only use a WebRTC focused testing tool, you will miss out on other features and requirements you need to handle.

Today, you need to use multiple tools to handle the full coverage of your application if you use WebRTC.

WebRTC Interoperability testing 

interoperability testing as part of the webrtc test

WebRTC interoperability testing is tricky. Especially if you come from the VoIP world.

Interoperability means taking products/entities/devices from different vendors and have them speak with each other.

In the domain of WebRTC, this is split into two parts:

Browsers interoperability – how browsers communicate with each other? Something that browser vendors need to deal with and NOT you (while you may well find bugs there, it isn’t your purpose in life). Today, browser vendors do it by using KITE, an open source framework designed and created for browser interoperability testing. It was later extended to support other scenarios.

Service interoperability – your service needs to work well in front of browsers. How do you test for that? Most of it is about keeping pace with browser vendors.

Fuzzing WebRTC

fuzzing webrtc

Fuzzing deals with changing what goes on the network to ruin your day. The idea is to create slightly modified messages so that the software that needs to parse them gets confused, hopefully breaks – or even better – create a backdoor for hackers.

WebRTC is no different and recently, fuzzing got its publicity.

It is interesting, but for most WebRTC services, something that you can park for now.

Check here to learn more about WebRTC fuzzing.

How do you plan to test your WebRTC service?

Here’s what you should be doing:

  • Sit down with the team
  • Check for the items here which ones you are covering today in your test plans
  • Identify the areas you aren’t covering in your WebRTC testing based on the list in this article – and decide which ones are important for you to add
  • Think what tests conducted today are useless, or less important. You might be able to stop conducting them or to invest less time and energy on them

But most of all, understand better your testing strategy when it comes to your WebRTC service.

Want to learn more about WebRTC? Check out my online WebRTC courses.

Interested in working on a test plan for your WebRTC application? Contact me AND check testRTC out!

Responses

Steve Fox says:
March 9, 2020

The article could be more detailed; now it’s just a brief overview of the subject.
For example, just the interoperability section deserves the article on it’s own,
demonstrating issues and challenges of a) pushing content generated by various encoding hardware to WebRTC, and b) pulling the streams from WebRTC into other multimedia domains – into ffmpeg, DirectShow, decoding STBs, transcoding clowds etc… Kind of the stuff that Google developers never cared about – they only really care about browser-to-browser WebRTC. Huge demand for interoperability exists; look, for example, at this Stackoverflow thread: https://stackoverflow.com/questions/59865405/use-webrtc-getusermedia-stream-as-input-for-ffmpeg/60143788#60143788
There is not a lot of supply meeting that demand. And yes, lots of interoperability issues do exist. Let me give you one typical example: Intel QuickSync H264 encoder produces a stream with 3-bytes and 4-bytes startup codes in front of H264 NALUs. But H264 decoder used for WebRTC in Chrome (mojovideodecoder), does not accept 3-bytes startup codes, despite that standard allows 3-bytes startup codes. So that video stream will not play in Chrome, when sent via WebRTC.
We send numerous bug reports to Chrome developers, and they are usually very responsive and try to fix these bugs, but the real issue is – WebRTC interoperability is never even on a radar of Google’s developers, they don’t think about it.

Reply
    Tsahi Levent-Levi says:
    March 9, 2020

    Thanks for sharing this Steve.

    I had to cut the length of this article at some point (it was getting longer as it is).

    Interoperability is a never ending testing effort for browser vendors. My assumption is that their focus is on interoperablity across the browsers as they are trying to hit the WebRTC 1.0 milestone at the W3C, coupled with the effort of introduction of things like AV1 (at some point) and making more use of VP9 in actual services.

    Reply
Amit Verma says:
September 24, 2020

Hi

Is there any tool which could test my WebRTC app feature which is for Video Voice calling and check its Quality as well.

Please let me know if you can help.

Reply
    Tsahi Levent-Levi says:
    September 24, 2020

    Amit,

    testRTC (I am co-founder and CEO there) offers a commercial service for that.

    KITE is an open source tool that does similar things.

    Reply

Comment