Fitting WebRTC in the brave new world of webcams, security, surveillance and visual intelligence

WebRTC has its place in surveillance and security applications. It isn’t core to these industries, but it is critical in many deployments.

Surveillance has become near and dear to my heart. I had a few vendors consult with me in the past. There are a few using testRTC. And then there’s the personal level. The system we have in our apartment building.

This got me to think quite a lot about WebRTC in surveillance tech lately.

Why my interest in surveillance cameras (and WebRTC)?

I live in an apartment building here in Israel:

🏢 23 floors

🤼 91 apartments

🚪 2 main entrances (and another side one)

🛗 3 elevators

🅿️ 3 levels of underground parking

…

And yes. We have a surveillance camera system. Like all of the other apartment buildings in my neighborhood:

A year ago, I was in charge of the vendor selection and upgrade process of our cameras. We switched from an analog system into a hybrid analog/IP one.

This month, we’re looking into upgrading an elevator camera to an IP one, as well as adding WiFi to our underground parking. Having a chat with one of the vendors we’re reaching out to, he was fascinated with my work on WebRTC and the potential of using it for application-less viewing of cameras.

I’ve had my share of meetings and dealings with vendors building different types of surveillance and security solutions. From private security solutions to large scale, enterprise visual intelligence ones. Obviously, the matter of these interactions were around WebRTC.

⏩ I am not an expert in surveillance, so take the market overview with a grain of salt

⏩ That said, I do know my way with WebRTC and where it fits nicely

⏩ Here are some of the things I learned over the years

Security and surveillance use cases in WebRTC

I’ll start with the obvious – cameras, security and surveillance have multiple use cases. Some of them can be seen as classic to this domain while others slightly newer or a specialized niche. Each of these use cases is a world onto its own with its requirements from WebRTC and the types of solutions emerging in it.

Small scale / cheap multiple surveillance cameras

This is where I’d frame my own experience of our apartment building. A system that requires 32 or less video cameras, spread across the location, connected to a DVR (Digital Video Recorder) or an NVR (Network Video Recorder).

In essence, you go install the cameras in sensitive locations, wire them up (with an analog cable, IP or even wireless) to the media server that is located onsite as well. That media server is a DVR if it is a closed loop system or an NVR if you’re living in modern times. I’ll just refer to these two as xVR from here on.

Once there, you hook’em up to a local monitor that nobody goes and look at, as well as let the owner connect remotely from his PC or mobile phone.

Is WebRTC needed here? Not really.

Surveillance cameras today use RTP (and sometimes also RTSP). These are the new ones. Old ones are pure analog. They connect to that xVR media server, which handles them quite well today. It did so also before WebRTC came to our lives. The user then accesses the system to play the videos remotely using a dedicated application, which again, existed before WebRTC.

Since there’s no specific requirement to access this through a web browser, the use of WebRTC here is questionable.

You might say WebRTC would make things easier, but hey – if it ain’t broken, don’t fix it

These solutions are purchased from local vendors that install such systems. The buyer will usually reach out to an installer that will pick and choose the cameras and the surveillance system for the buyer. The buyer cares less about the technology and more about the local vendor’s ability to install and maintain the system when needed.

Enterprise / large scale surveillance

Large scale surveillance systems for enterprises is more of the same as the small scale ones, but with a few main differences:

1. There are more cameras
2. There are also more sensors which we want to control and manage, likely using the same system. Think doors and managing employee entrance using keycards for example. While this is about surveillance and security, it is also about building automation
3. This can go from a small scale building to as large as smart cities with lots of cameras – anywhere in-between that I bunch here are most likely multiple different markets with slightly different requirements
4. We are likely to have a NOC, where security guards look at screens. Just like in the movies…

The two things that are making headways in this industry?

• Using AI to reduce the amount of people needed to look at surveillance monitors. This is done by adding vision smarts into cameras and the media servers (local or in the cloud), so that events and alerts can be filtered better
• To some extent, there’s also a requirement to use WebRTC in the NOC to be able to view in real time camera feeds without installing anything

Like the small scale solutions, here too the buyer will look for local installers. These will be the local integrators who bring the systems and install them. At times, the decision of brand will come from the buyer, though this is less likely. It is important to remember that a considerable part of the cost goes towards the setup and installation and not necessarily to the cost of the equipment itself.

Personal/home surveillance

This one is the residential one. It is a B2C space where the buyer is a person buying a camera for his own home security. The decision is made on price or brand mostly.

Here you’ll find also solutions that make use of old smartphones and tablets as cameras, or something like the one we purchased a few years back when our kids were younger:

Having the ability for them to see who is outside our door when they were shorter.

Here too, the market is going into multiple directions:

• Home automation, connecting more sensors and devices in the home, some of them have cameras in them
• Surveillance and security, where today it seems at least here in Israel, that fingerprint door locks are all the rage

Where does WebRTC play here? It might make things smoother to develop for the companies, but this doesn’t seem to be the case.

One thing that goes through all use cases above, is the existence of another solution – the video doorbell. Taken into buildings, this becomes an intercom system, which again – can make use of WebRTC. And why? Because it needs bidirectional support for audio at the very least, making WebRTC a suitable alternative.

Personal security

A totally different niche is the one of personal security.

This manifests itself as apps (and services) people can use to increase their security while going about in their daily tasks. Some of these apps connect you to friends and family while others to personal security agents. The WebRTC requirement here is the same for all cases – be able to conduct voice and video calls in real time.

Taken more broadly from the personal level, the same can be implemented in campuses, cities, events, etc.

Unique (?) challenges for WebRTC with camera hardware

There are some unique challenges for WebRTC when it comes to the surveillance space, and that’s mostly a matter of hardware.

• Costs
  • Hardware costs money. Not just the devices themselves, but their installation. This also means that hardware costs needs to be kept low in most of these systems, which means less processing power available on the cameras themselves or the xDR devices
  • To drive costs down, CPUs won’t be as performant as the ones found in smartphones or PCs for example, and they would almost always rely heavily on hardware video encoding
• Maintenance
  • Many of these hardware systems come without subscription services. This means any firmware upgrades might or might not be available. It also means that such upgrades are sometimes clunky to get done on the devices, especially when they need to be handled remotely
  • There’s a lot of physical maintenance as well involved. Cleaning lenses of cameras for example
• Technology leaps
  • You purchase a system. It has cameras and a xDR. Time passes. A couple of years. You decide you need more cameras, replace an existing one, whatever
  • There’s improvements that took place. The system you have might not even be able to deal with the new cameras available today, and purchasing old ones might not be possible or economical anymore
  • We had this when the system in our residential building broke. The DVR had a hard drive malfunction – it didn’t record anything anymore
    • It was impossible to replace, and buying a new old system wouldn’t be the right approach
    • Some of the cameras lost quality due to their analog coax cables (I was told this is an issue), and the predicament was we’d lose more of these cables in the coming 2-5 years anyways
    • So we had to shift the whole system to an IP based one. A technology leap
    • While I don’t foresee a move away from IP, I am sure many of these systems will change in the coming years in ways that would leave some of the old hardware unusable
• Hybrid
  • There are hybrid alternatives in this space. We ended up getting one for our building
  • Due to the technology leaps, you end up with multiple types of sensors and cameras, from different generations and technologies
  • The systems that cobbles it all together (the xDR in our case), can be one that manages them all
  • Most installers won’t recommend it. It is mostly a necessary evil. Likely because it reduces the revenue of the installer and adds to the complexity of the installation and the system

Most of these issues won’t plague a software solution. But here, we end up in the real world simply because someone needs to go and install the physical cameras.

👉 When figuring out the hardware platform to use, it is important to think of future trends and technology improvements that affect your implementation

👉 In the case of surveillance, there’s WebRTC, future video codecs (AV1) and machine learning in the vision domain to think about. Probably also programmable photography that is bringing innovations to smartphones for a few years now

Ingress, egress and the concept of real time

Where to place WebRTC in the solution?

Since I write a lot about WebRTC, and this article is mostly about WebRTC in surveillance markets, it is THE biggest question to answer here.

There are two different places, and both are suitable, but not necessarily together in the same system.

Surveillance needs real time. Sometimes.

Egress

In our own residential building, I seldom care about the live feed from the cameras. It is to check if the front door to the building is open or not, or if there’s some area that got dirty (usually dog pee). Then most of the time is spent rewinding to figure out who caused the problem. Nothing here is considered real time in nature or requires sub second latency.

Elsewhere, real time might be critical on the viewer side (egress), which brings with it the question of whether WebRTC fits here well.

Ingress

Web cameras that directly stream out WebRTC to the world (or the xDR). Is that a benefit? What’s the value of it versus the existing camera technologies used?

I am not quite for or against this, as I am not really sure here. I’d say that a benefit here can be in the fact that it makes the whole technology stack simpler if you end up using WebRTC end-to-end instead of needing to switch protocols from the camera to the viewer. Just remember here that rewind and playback will likely require something other than WebRTC.

The main advantage of WebRTC here might be the removal of the need to transcode and translate across protocols and codecs. It makes xDR software simpler to write and reduces a lot of their CPU requirements, making the systems lighter and cheaper (the xDR – not the camera itself).

One more thing to think of is cameras that also require bidirectional audio. Because a security guard wants to announce or warn perpetrators, or because this is a video doorbell. There, WebRTC fits nicely, though again – not mandatory (I’d still try using it there more than elsewhere).

👉  Going to introduce WebRTC to a surveillance system? Great. Check first where exactly within the whole architecture WebRTC fits and ask yourself why

Mobile or desktop?

Another important aspect of a surveillance system is where people go to watch the videos.

When we installed our own system, we were told that the mobile app is better than the PC app. In both, these were applications. But somehow for the consumers, it meant using the smartphone. It sucks. But yes – it sucks more on the desktop. Which is crazy, considering that what you’re trying to do is watch output coming from 4K cameras in order to identify people.

Then again, who is your customer?

If this is a large enterprise, where there’s going to be a fancy video wall of video feeds with a bored security guard looking at it, then should this be an application or would it be preferable to use a web application for it, with the help of WebRTC? It seems that much of the industry on the client side is looking for lightweight solutions that require less software installations, favoring browsers and… WebRTC.

And if you’re already doing WebRTC for one egress destination, you can use it for all others – browser and app based.

One more thing to consider – it is easier today to develop a web application than it is a native PC application. Cheaper and faster. Which means that supporting WebRTC if the desktop is your primary viewing device might be the right decision to make.

👉 See if there’s a strong need for a zero-install or desktop viewing. This might well lead you towards WebRTC on the egress side

The age of Artificial Intelligence in surveillance tech

The biggest driver in this industry is machine learning and artificial intelligence. And not necessarily the Generative AI kind, but rather the kind that deals with object classification.

The challenge with surveillance is watching the damn cameras. You need eyeballs on screens. The good old motion detection removes a lot of noise (or more accurately, static), but it leaves much to be desired.

One of the elevators in my building, along with the video you get most hours of the day – empty. The bar at the bottom with the blue stripes marks when there’s actual movement.

Using machine learning, it will be easier to search for dogs, people, colors, items and other tidbits to figure out times of interest in the thousands of hours of boring videos, as well as act as “Google search” on recorded video feeds.

Doing all that in the cloud is possible, but expensive and tedious – how do you ship all the video, decode it, process it again, etc.

Doing it on the edge, on the device itself (the camera or the xDR) is preferable, but requires new hardware, so requires another technology leap and refresh.

WebRTC isn’t core for surveillance but it is critical

This is something to remember.

WebRTC isn’t core to surveillance. You don’t really need it to get surveillance cameras working, installed or connected to their xDR media servers. You don’t even need it to view videos – either “live” or as playback.

But, and that’s a big one – in some cases, having WebRTC is critical. Because your customer may want to be able to use web browsers and install nothing. He may want to be able to get bidirectional media. There might be a need to get video feeds that are at sub second latencies.

For these, WebRTC might not be a core competency, but they are critical to the successful delivery and deployment of your product. This translates into having a need to have that skill set in your team or be able to outsource it to someone with that skill set.

🔸🔹🔸🔹🔸

Where can I help, if at all?

🎯 Online WebRTC courses, to skill up engineers on this technology

🎯 Consulting, mostly around architecture decisions and technology stack selection🎯 Testing and monitoring WebRTC systems, via my role as Senior Director at Cyara (and the co-founder of testRTC)