WebRTC test: How to go about testing WebRTC?

WebRTC testing has many aspects to it. Many don’t systematically test their WebRTC applications while others test the wrong things. Here’s what you need to know about testing WebRTC.

WebRTC has many moving parts to it. You have the user devices, signaling servers, the application server, TURN and STUN servers, sometimes media servers. The areas you need to focus in your WebRTC testing will be different than those of someone else and would depend on both the use case and the architecture chosen. In this article, I’ll try to list the various aspects you need to think about when building a test plan for your WebRTC application.

https://www.youtube.com/watch?v=1-P6uZ7qMfk

End to end WebRTC testing

At the end of the day, your users are going to interact with your service as a whole. This means that much of your testing should focus on end to end WebRTC testing.

Such tests need to be conducted in a fashion that is as close as possible to how your real users interact with your service.

Consider the following:

1. Manual testing - your QA engineers testing the service as end users will. But bear in mind that you will need to handle aspects of different network conditions and locations, which greatly affect how WebRTC behaves
2. Automated testing - you can automate some of the end to end tests using Selenium or SaaS platforms like testRTC (I am a co-founder there). These offer the ability to cover 80% of the scenarios you need to deal with at 20% of the time in a predictable and reproducible manner
3. User testing - this happens a lot in the consumer space but also elsewhere. What I mean here is releasing the product to users or to a small percentage of the users and seeing what people complain about. Effectively using users as guinea pigs and cheap testing labor. It can work for some, but not for all products

BTW - using a CPaaS provider (=consuming WebRTC as an API from a third party) doesn’t mean you don’t need to conduct your own testing. You can never know if the use case you implemented is in one of the main scenarios that CPaaS provider is focusing on.

P2P testing

Some services are simpler than others. If your service provides only 1:1 calls conducted in a peer-to-peer fashion without any media servers in the middle. This reduces the testing needed by quite a bit.

In P2P, the main concern you will have is the scale and uptime of your signaling server (the other important one is interoperability, but on that - later).

Since actual media doesn’t touch your infrastructure other than your TURN servers, testing it would be nice, but you’d be just doing the browser vendors’ job if that’s your main focus.

Check here for more on WebRTC P2P testing.

Mobile testing

Mobile is slightly different when it comes to WebRTC. And by that I mean non-browser testing - no mobile and elsewhere.

When you use a browser, someone else is in charge of the WebRTC implementation on the device. When you use your own application, on mobile, embedded or desktop - you are the one in charge of that WebRTC implementation. You are the one picking the WebRTC stack used, modifying it, updating it, etc.

This means you will need to spend more time in testing your mobile application as well, and sadly, there are no easy ways to automate these tests. WebRTC uses too much of the device to make it simple to automate mobile testing by the use of simulators or even by the use of real devices.

A lot of your mobile testing will be conducted manually when it comes to the communications part of it.

TURN server testing

TURN server testing requires forcing networks to block certain types of traffic so that STUN won’t work well. You will either need to change the peer connection configuration to enable that or to configure the machines used to work in a certain fashion, such as blocking UDP traffic.

Now, if you are outsourcing your TURN servers to a third party managed service provider, then you don’t need to have such extensive testing to it (assuming the provider you use is reputable enough that you can trust his testing).

Media server testing

WebRTC media server testing can be isolated from the rest of the system. The reason to do that is in order to test behavior in different network conditions or when you want to stress test the service.

You may want to stress test your server by using synthetic traffic not originating from browsers (such as wiring one media server to another and sending traffic directly between them) in order to increase the scale of your tests while keeping your costs low. At other times, you may prefer using automated headless browsers for that purpose, to get as close as possible to real life conditions in your stress testing.

A word about stress testing -

These can be conducted in the end to end fashion or in a media server unit testing fashion. Both are valid. Aiming for the media server unit testing alternative makes sense if the application is complex in terms of its UI - you wouldn’t want to fail tests due to failures in automating the UI itself. Aiming for end to end testing gives you a higher confidence level as to potential bottlenecks throughout your system and not only in your media servers.

Functional and coverage testing

Dealing with all the edge cases and functionality of an application is important yet daunting. When many of these tests occur with no actual WebRTC session taking place, it is important to ask how do we treat these tests?

We’re 8 years into WebRTC. It is still a small niche, which means that most of the testing tools available out there don’t focus on WebRTC at all. If you plan to use a generic testing tool for all of your testing, including the WebRTC ones, then you will lose a lot of the functionality required for WebRTC (things like geographic spread, machine configurations, network conditions, etc).

If you only use a WebRTC focused testing tool, you will miss out on other features and requirements you need to handle.

Today, you need to use multiple tools to handle the full coverage of your application if you use WebRTC.

WebRTC Interoperability testing 

WebRTC interoperability testing is tricky. Especially if you come from the VoIP world.

Interoperability means taking products/entities/devices from different vendors and have them speak with each other.

In the domain of WebRTC, this is split into two parts:

Browsers interoperability - how browsers communicate with each other? Something that browser vendors need to deal with and NOT you (while you may well find bugs there, it isn’t your purpose in life). Today, browser vendors do it by using KITE, an open source framework designed and created for browser interoperability testing. It was later extended to support other scenarios.

Service interoperability - your service needs to work well in front of browsers. How do you test for that? Most of it is about keeping pace with browser vendors.

Fuzzing WebRTC

Fuzzing deals with changing what goes on the network to ruin your day. The idea is to create slightly modified messages so that the software that needs to parse them gets confused, hopefully breaks - or even better - create a backdoor for hackers.

WebRTC is no different and recently, fuzzing got its publicity.

It is interesting, but for most WebRTC services, something that you can park for now.

Check here to learn more about WebRTC fuzzing.

How do you plan to test your WebRTC service?

Here’s what you should be doing:

• Sit down with the team
• Check for the items here which ones you are covering today in your test plans
• Identify the areas you aren’t covering in your WebRTC testing based on the list in this article - and decide which ones are important for you to add
• Think what tests conducted today are useless, or less important. You might be able to stop conducting them or to invest less time and energy on them

But most of all, understand better your testing strategy when it comes to your WebRTC service.

Want to learn more about WebRTC? Check out my online WebRTC courses.

Interested in working on a test plan for your WebRTC application? Contact me AND check testRTC out!