The Challenges in Porting WebRTC to Mobile Devices

Ever thought of how WebRTC fits into mobile? [Arik Halperin from SoftSkills Solutions had his share of mobile development. Recently, he ported WebRTC to Android and iOS. Here is his experience around that activity]

When people think of WebRTC, they usually think about Chrome or Firefox. But what about mobile applications? WebRTC above all is a great media engine, the source code is open and free and integrating into a mobile app should be a piece of cake, at least this is what I thought...

As I found out, building mobile applications around WebRTC can be a bit tricky. WebRTC indeed comes with ports for Windows, Mac, Android and iOS (Audio only) and Google is doing an amazing job, but in order to be able to make a truly usable mobile app there are a few challenges that still need addressing:

1. Device Challenges: Challenges which emerge from the fact that mobile devices vary in form, Hardware configuration and computation resources. Included in this are processor architecture, CPU power, screen and camera resolution, audio acceleration support, different sensors and battery.
2. Medium Challenges: Medium challenges are challenges which are a product of the mobile environment instability. For example: Changes in connectivity (WiFi, 3G, CDMA), changes in network profile (Network load, interference), ambient noise, changes in lighting etc.

Device Challenges

• Form factor differences: Form factor differences means one device’s image will have a different look than another in aspect ratio and resolution. What should happen if I use an iPad 4 and a Galaxy S4 mini phone? Or maybe a Chrome browser connecting to a mobile app? How should these varitions be treated?
• Audio Hardware configuration: Different devices have different implementation for the built in speaker and Bluetooth. Especially in Android there are many differences between devices that require special handling.
• Screen & camera resolutions: Different camera and screen resolutions can affect the quality of the image in the video stream.
• Processor architecture: To support WebRTC in video, a CPU which has at least ARMv7 architecture with its NEON extension. Audio can be used in processors without NEON (You will not have enough CPU power for video).
• CPU Power: CPU power can be a limiting factor in frame rate and image resolution, especially image capture, due to the complexities of the video encoding.
• Audio acceleration support: Not all devices support audio acceleration. In a device that supports Audio Acceleration (iPhone 4 and above for example) it can be utilized to save the CPU power used by WebRTC audio processing.
• Sensors: Mobile devices can change orientation, which will affect the image generated by them and the one displayed on their screen.
• Battery: Left it to the end, because it’s a real pain. Running a WebRTC video call kills your battery! There could be lots of innovation in this one, both from chip vendors and the developer community.

Medium Challenges

• Changes in connectivity: Changes in connectivity can create a problem when they happen inside a session. You need to have a strategy for handling such connectivity changes, for instance if a device can change its connection from WiFi to 3G within a session, then you should take the correct measures to keep the session alive (Block 3G handoff, recreate the session after handoff etc.)
• Network profile: In mobile the network profile can change much more dynamically than in a stationary PC environment. For example: User may change position and WiFi or 3G jitter and bandwidth can change as a result. If you are in a car cell handoffs can occur or there could be glitches in connectivity (Better not be the driver though, if doing video). The configuration used for WebRTC in a mobile app needs to take these into account in elements like bandwidth, frame rate and resolution adaptation.
• Changes in lighting: The user may move with the phone to places where light is more dim and this could affect the video display on the receiving side.
• Ambient noise and use of speaker: A mobile profile usually demands using more rigorous noise suppression and echo cancellation algorithms.
• Network challenges: In a mobile environment especially, NATs and firewalls are a real pain (Especially if you want your app to work in corporate environments).

Summary

WebRTC is a great platform for making live video chat mobile applications. However, it requires some tuning. There are inherent challenges which exist in a mobile application, not found in a stationary PC environment. If you want create a great WebRTC mobile app, I will be happy to assist.