IBM Research: First response made faster with video analytics

Drones have been all over the news lately, with futuristic promises to do everything from chasing criminals to delivering packages. The reality is that drones, also known as unmanned aerial vehicles, are already commonly used in complex systems. This includes systems that rely on obtaining large amounts of video information in real time, such as those used for managing large-scale emergencies, as well as ones used for improving farming techniques and traffic control.

My team at IBM Research – Haifa developed a framework for command and control (C2) applications that use mobile video feeds. We call it CARMEL, short for IBM Context-Aware Rich-Media Extensible middLeware.

These drones carry mobile sensors that can film large areas, providing good video coverage at a low cost. They come in all shapes and sizes, and have even made their way to the consumer market in the form of quad-copters. In disasters such as floods or fires, when road travel may not be an option, video from drones may be the only view we can get. Our aim is to help the people who need to interpret these types of videos.

Reducing information overload

As is often the case with new technology, these video sensors can create as many problems as they are attempting to solve. They provide such huge sets of data so quickly that sorting through them in time for practical use is difficult. And because the sensors are in motion as they photograph, figuring out when or where events took place can be nearly impossible.

CARMEL works by pairing video segments with the location information provided by metadata from the sensors’ geographic information system (GIS), annotating the videos with date and time information. Users can watch videos streamed in real time, understand what they are seeing, and then make decisions based on accurate, up-to-date information.

The software uses IBM middleware such as Websphere, MQ, and Informix. It is not integrated directly with drones. But once the feeds from the drone are received at the ground-station network in industry standard format, CARMEL can detect these feeds (scanning the relevant IP/Port ranges) and ingest the data along with geographical metadata into the system.

Collaboration capabilities

Today, emergency officials can get video feeds of the aftermath of a flood. Watching from a command center, away from the disaster, the videos come in faster than they can watch them, let alone analyze what they are looking at. Imagine that the officials see video of people stranded on their rooftops, yet they are not sure exactly where the houses are, or where rescue crew members are, or even if the roads between them are passable.

Using CARMEL, rescue teams in this situation can search for people who need help while simultaneously watching aerial video feeds and projecting their coordinates to their managers. The command center can then locate those in need and view their immediate surroundings to quickly and safely direct their rescue forces there.

CARMEL can also be configured to focus on a specific area, reducing information overload and potential distractions. The technology also provides instant-replay and collaboration capabilities while video continues to be captured. This allows CARMEL users to collaborate by sharing annotations on the viewed video in real-time In the case of disaster recovery, emergency managers can divide a region among rescue workers. These features work together to help streamline planning, evaluating, and acting — crucial in situations in which every minute counts.