Objects of interest

In the summer of 2017, a man in a white van was driving around Hartford, Connecticut. He slowed down to a halt alongside a ten-year-old girl and exposed himself. Within minutes police officers were able to identify his van, as it weaved through traffic, and he was subsequently arrested.

The girl raised the alarm, triggering a sequence of events that was fast, effective and – critically – preventative. It’s well understood that predatory offences of this nature are rarely isolated and can quickly spiral, so time is of the essence to ensure that the offender does not quickly move on to another victim. In this case, Hartford PD were able to identify the van in minutes and gather enough evidence to apprehend the suspect using a combination of information from officers, footage from their network of 545 cameras and BriefCam’s patented VIDEO SYNOPSIS® technology. BriefCam’s Video Content Analytics Platform allowed the police to scan huge amounts of footage in a fraction of the time that it would take for manual review. It even helped them pinpoint evidence of the offender tracking the child over a period of days. But how?

“Our technology essentially takes a video and breaks it into pieces,” explains Tom Edlund, Chief Technology Officer at BriefCam. “And the pieces are actual objects. So, we first conduct what we call ‘object extraction’, which detects and tracks an object at any given time. Then we add another technology called ‘background/foreground separation’, which gives us a static background of the scene.” The magic happens when the VIDEO SYNOPSIS® technology creates a completely new, condensed video, where all video objects are displayed simultaneously in the video – showing the full activity of the scene over the course of many hours all at once, rather than in linear time. This means that when Hartford PD were looking for their white van, they could ‘extract’ it as an object to locate its appearance across multiple cameras and video files and see its journey all at once, instead of ploughing through potentially hours of footage.

At the heart of this incredible technology is, of course, Artificial Intelligence. “Today, almost everything we do is deep learning based,” says Tom. “and to do this you need a vast amount of data, which must be processed and annotated. You then use this data to train the network, until it is able to repeat what we have trained it to do.” A good example might be in trying to locate a particular item of clothing in network surveillance footage. The training data, which is existing recorded video, will be cropped and prepared to present often hundreds of thousands of examples of an item – say, a rucksack – then the deep learning neural network uses this information to understand the qualities of a rucksack in all its variations in order to answer a query against actual footage: ‘identify all instances of people wearing rucksacks’. Perhaps you want to find all yellow rucksacks? In order to detect them, the deep learning algorithm will need to know what yellow is and will need to be fed thousands of data examples of this colour too. This is repeated ad infinitum for as many objects are required.

But detection alone is not enough. Once an object is detected, then it must also be tracked. “It’s very important that we’re able to accurately follow the object from the time it enters the scene to the time it leaves the scene, even if there is some occlusion or crowded scenario,” Tom explains. “Multiple hits, alerts or matches on the same person that will just create a lot of distraction for the operator.” Once the object is detected and tracked, then descriptive information (or ‘metadata’) can be applied to it to create a vast structured database of classified objects. When you consider the many hundreds of thousands of detections and classifications involved, it’s a vast, highly complex and resource intensive undertaking. But one that is ultimately essential when you consider how much time and how many officers would have been required to locate the ‘white van’ simply by watching the footage. For law enforcers, time is simply a luxury they do not have.

However, it’s not just the police who reap the benefits of BriefCam’s technology. It has multiple uses and is adaptable to all sorts of potential scenarios – before, during and after an event. Hartford PD used the reviewing capability to investigate an event that had already occurred, but real-time alerting can also be put in place, responding to events as they happen in order to identify objects of interest. For example, you can set a real-time alert for a specific camera at the entrance of a construction site to ensure that everyone who passes is wearing a hard hat. Or in the case of The Communauté De Communes Coeur Côte Fleurie in France, use it to detect traffic violations that may otherwise have gone unnoticed.

It’s no understatement to say that the data created from network cameras also has the potential to drive positive change in huge, life-enhancing ways. It can be used to analyse places and spaces in a way that can influence decision making across buildings, businesses – even entire cities. It’s a similar kind of business intelligence that we might be familiar with in department stores, where consumer behaviour is measured in order to create a more appealing shopping experience. Hartford PD have shown that network cameras can be used for law enforcement, but when video output is analysed, it can be used the output for forecasting and monitoring of huge events – such as understanding, predicting and managing the flow of crowds, so no one is injured. Business intelligence of this nature can even be used to raise the bar for customer service –tracking visitor numbers to spaces, so they are staffed appropriately and monitoring comings and goings at public restrooms, so they are cleaned after a measured number of visitors. Every intelligence dashboard is as unique as its use case.

This ‘review, respond and research’ approach has something of the virtuous circle about it and certainly sits in parallel to the fundamentals of smart city development, where seamless connectivity and its very human benefits are driven by a learning curve. At the peak of its effectiveness, all city dwellers will see is an environment that is just the same, only better and safer.

BriefCam, a Canon company, technology transforms how organizations use video surveillance, and has been recognised with a series of industry accolades, including The Wall Street Journal Technology Innovation Award, Security Products Magazine New Product of the Year, SIA New Product Showcase Best in Video Analytics Award, and CNBC Europe’s 25 Most Creative Companies.