Yes, a P2P test is needed.
It is a question I get from time to time, especially now, that I am a few months into the WebRTC testing venture as a co-founder with a few partners – testRTC.
The logic usually goes like this: the browsers already support WebRTC. They do their own testing, so what we end up getting is a solid solution we can use.
If life was that easy…
Fippo says:
“if you look at the tests done by Google you might notice that they just test the apprtc use case.”
I just think you can’t cover everything. Here are a few things you need to take care of when it comes to testing the most simple of WebRTC services:
#1 – Future proofing browser versions
Guess what? Things break. They also change. Especially when it comes to WebRTC.
A few interesting tidbits for you:
- Google is dropping HTTP support for GetUserMedia, so services must migrate to HTTPS. Before year end
- The echo canceller inside WebRTC? It was rewritten. From scratch. Using a new algorithm. That is now running on a billion devices. Different devices. And it works! Most times
- WebRTC’s getStats() API is changing. Breaking its previous functionality
And the list goes on.
WebRTC is a great technology, but browsers are running at breakneck speeds of 6-8 weeks between releases (for each browser) – and every new release with a potential to break a service in multitude of ways – either because of a change in the spec, deprecation of capability or just bugs.
Takeaway: Make sure your service works not only on the stable version of the browsers, but also on their beta or even dev versions as well.
#2 – Media relay
Your service might be a P2P service, but at times, you will need to relay media through TURN servers.
The word on the street is that around 15% of sessions require relay. To some it can be 50% and to others 8% (real numbers I heard from running services).
Media relay is tricky:
- You need to configure it properly (many fall at this one)
- You need to test it in front of different firewall and NAT configurations
- You need to make it close to your users (you don’t want a local session in Paris to get relayed through a server in San Francisco)
- You need to test it for scale (check the next point for more on that)
Takeaway: Don’t treat WebRTC as a browser side technology only, or something devoid of media handling. Even if the browser does most of the heavy lifting, some of the effort (and responsibility) will lie on your service.
#3 – Server scale
Can your server cater for 200 sessions in parallel to fit that contact center? What about a 1000?
What will happen if you’ll have a horde effect due to a specific event? Can you handle that number of browsers hitting your service at once? Does your website operate in the same efficiency for the 1000th person as it does for the first?
This relates to both your signaling server, which is no part of WebRTC, but is there as part of your service AND your media server from my previous point.
Takeaway: Make sure your service scales to the capacities that it needs to scale. Oh – and you won’t be able to test it manually with the people you have with you in your office…
#4 – Service uptime
You tested it all. You have the perfect release. The service is up and running.
How do you make sure it stays running?
Manually? Every morning come in to the office and run a session?
Use Pingdom to make sure your site is up? Go to the extreme of using New Relic to check the servers are up, the CPUs aren’t over loaded and the memory use seems reasonable? Great. But does that mean your service is running and people can actually connect sessions? Not necessarily.
Takeaway: End-to-end monitoring. Make sure your service works as advertised.
The ugly truth about testing
The current norm in many cases is to test manually. Or not test at all. Or rely on unit testing done by developers.
None of this can work if what you are trying to do is create a commercial service, so take it seriously. Make testing a part of your development and deployment process.
And while we’re at it…
Check us out at testRTC
If you don’t know, I am a co-founder with a few colleagues at a company called testRTC. It can help you with all of the above – and more.
Leave us a note on the contact page there if you are interested in our paid service – it can cater to your testing needs with WebRTC as well as offering end-to-end monitoring.
Need to test WebRTC? Go to testRTC.
Want to learn more about WebRTC testing?