Click here to Skip to main content
14,031,046 members
Click here to Skip to main content
Add your own
alternative version


7 bookmarked
Posted 22 Mar 2019
Licenced CPOL

Action Detection Using Deep Neural Networks: Problems and Solutions

Rate this:
Please Sign up or sign in to vote.
Learn more about the challenges we faced with dataset preparation and network configuration, and how these problems can be solved.


Action detection is one of the most challenging tasks in video processing. It can be useful in security systems and closed-circuit television (CCTV), emotion and gesture tracking, sports event analytics, behavior observation, statistics gathering, etc.

In one of our recent projects, we were challenged with the task of detecting a particular human action in a high-resolution video. In this article, we talk about the importance of finding a balance between accuracy and performance, the challenges we faced with dataset preparation and network configuration, and how these problems can be solved.


Using DNNs for action detection

Balancing accuracy and performance

Preparing the dataset

Configuring the network


Using DNNs for action detection

Deep learning is an area of machine learning that uses feature learning techniques instead of task-specific algorithms. It uses multi-layered artificial neural networks that work similarly to neural networks in the human brain. But unlike the brain, a neural network is divided into separate layers each with a defined direction of data processing. A network with more than two layers is called a deep neural network (DNN).

A DNN can achieve human-like accuracy in such tasks as image classification, object detection and classification, speech recognition, handwriting recognition, and computer vision. With proper training, a neural network can perform some tasks even more accurately than a human. For example, Google’s LYNA algorithm is 99% accurate in detecting cancer — a level of accuracy hard to achieve for a doctor.

convolutional neural network (CNN) is a subtype of DNN, a multilayered algorithm that resembles the work of the visual cortex. CNNs are mostly used for the image, video, and language processing. The region-based convolutional neural network (R-CNN) is a new category of CNN created specifically for object detection. There are several types of R-CNN neural networks (Fast R-CNN, Faster R-CNN, Mask R-CNN, etc.) created for different image processing tasks.

A convolutional neural network algorithm

A convolutional neural network algorithm

The development of video processing solutions has become one of the most popular use cases for convolutional neural networks. A CNN can recognize emotions, gestures, speech, and handwriting, detect and classify objects, and detect and recognize actions in the video.

During our project, we defined action detection as classifying motions based on a series of regions of interest (ROIs) for an unbroken sequence of images.

However, there are many challenges in using CNNs for recognizing and detecting actions in the video. Recently, we worked on a project where the major goal was to detect a specific human action in a video stream. While working on this project, we faced a few major challenges:

  • Finding a balance between accuracy and performance
  • Preparing a dataset with similar parameters to the one the network was supposed to be working with
  • Choosing a relevant CNN for the task and configuring it

Let’s find out how to solve these issues by means of deep neural networks.

Balancing accuracy and performance

Nowadays, detecting actions remains one of the most challenging tasks in terms of both accuracy and performance. 

For detecting actions, it’s crucial to deal with spatial features of images (what object recognition methods do) as well as temporal ones.

In terms of image processing, identifying temporal features means determining changes in other features over some period of time. For videos, this could be a timespan or a certain sequence of frames. Identifying spatial features means determining image properties (grayscale, pixel locations, etc.).

 As for performance, it needs to be taken into account that video processing solutions are often implemented for trimmed, untrimmed, and real-time videos. The problem is that today’s video codecs can produce videos with very high frame rates and resolutions (FullHD, UHD, etc.). Basically, the higher the quality of a video, the longer it takes to process the data. Therefore, high-quality videos become a real challenge for projects (such as action detection) where real-time or near real-time data processing is required.

The trade-off between accuracy and processing speed is especially noticeable in action detection because while a higher quality of video often means better accuracy, such videos usually take more time to process.

Our research showed that this problem can be solved by a low-level language implementation of a neural network such as Faster R-CNN. This region-based convolutional neural network is designed to provide fast data processing by moving region proposal calculations to another pretrained feature extractractor called a region proposal network (RPN). This way, Faster R-CNN performs only detection calculations.

Faster R-CNN architecture

Faster R-CNN architecture

Faster R-CNN can be implemented with frameworks like Caffe2 or OpenCV, along with the TensorRT Inference Server and the best GPU setup possible. We used Caffe2 for our calculations.

Preparing the dataset

Training a neural network model requires massive amounts of data. There are quite a few good datasets for image classification and detection, such as MS-COCO and ImageNet.

There are also a few ready-to-use datasets for detecting actions, such as UCF101 and THUMOS.

The problem is, these datasets are designed for action classification and mostly consist of low-resolution trimmed videos of certain actions. Since we were challenged to accurately detect a particular human action in high-resolution high frame rate videos, there was a need for additional training on high-resolution videos of this action.

With no ready-to-use training data for our task, we had to prepare our own dataset. Training data is prepared by making annotations. These annotations are often made with tools such as GEDI or VGG Image Online Annotator. Initial research showed that we needed to consider both spatial and temporal features of training data.

First, we focused on spatial features, since there are a lot of well-trained feature extractors at our service. Fortunately, they’re also good at detecting people, and that was what we needed. We used the popular state-of-the-art MS-COCO dataset on the pretrained Mask R-CNN framework to generate ROIs on every frame of the video.

Then, we investigated ways to annotate actions and found that UCF101 annotations consist of ROIs and action annotations in the form of a frame span using XML format:

<object framespan="1:64" name="human">
           <attribute name="Location">
               <data:bbox framespan="1:1" height="47" width="27"
                   x="131" y="68"/>
               <data:bbox framespan="64:64" height="85" width="34"
                   x="153" y="83"/>
           <attribute name="action">
               <data:bvalue framespan="1:32" value="false"/>
               <data:bvalue framespan="33:64" value="true"/>

This XML code describes an object that performs an action. The object’s attributes represent its locations and actions in every frame where the object occurs.

Location data for annotations were obtained from Mask R-CNN; we used a custom script that skipped masks and dumped ROIs in a file with frame annotations. In order to support action detection for multiple humans in a single frame, we used this format as our starting point and modified it with unique identificators.

Configuring the network

For our project, we implemented Faster R-CNN, even though Mask R-CNN is considered superior to Faster R-CNN in terms of accuracy. We had two major reasons for choosing Faster R-CNN over Mask R-CNN:

  • Faster R-CNN has better performance
  • We had no need for masks in our project

Adapting the Faster R-CNN architecture from object recognition to action detection required a significant amount of customization, however.

Inspired by research on multi-region two-stream R-CNN for action detection, we implemented a custom logic to combine spatial and temporal features from our dataset annotation into so-called tubes.

Tubes are three-dimensional structures that consist of ROIs. Xiaojiang Peng and Cordelia Schmid, the researchers who published the paper that inspired us, used the Viterbi algorithm to generate these tubes for frames. However, because of the nature of the action we needed to detect, we decided to use a simpler approach. We used approximated coordinates for all ROIs in a sequence, which led to fewer calculations. In the current project, we didn’t compare these two methods.

Also, we implemented scripts for training on frame sequences of different lengths.

In contrast to the original Faster R-CNN, we redesigned our network to process sequences of frames and detect only two classes (background + action). This was achieved by modifying the prototxt files of the Faster R-CNN solution. These files are used in Caffe2 to describe each layer of the neural network. A typical layer description in these files looks like this:

layers {
  bottom: "data"
  top: "conv1_1"
  name: "conv1_1"
  convolution_param {
    num_output: 64
    pad: 1
    kernel_size: 3

The bottom and top attributes of the layer represent its connections with other layers. The type attribute represents actions performed by the current layer, which is a convolution layer connected with the first input layer (the bottom attribute). It has 64 output neurons that each serve as an input for the next connected layer (the top attribute).

Additional testing showed that pretraining the network on trimmed low-resolution images had no significant impact on detection accuracy. This may be due to the significant difference between the features acquired by the DNN during the training process on low-resolution and high-resolution videos.


The ability to detect and recognize specific human actions in video streams or recordings shows huge potential in many areas, from sports analytics to cybersecurity and surveillance systems. However, in order to implement action detection functionality, you need to consider that:

  • to improve accuracy, you need to take advantage of spatial and temporal features;
  • an ROI-based neural network architecture needs to be merged into tubes to represent an action (separate ROIs don’t mean that any action occurs);
  • training requires the use of data with a resolution similar to that which your network is supposed to work with;
  • if high performance is the priority for your project, you shouldn’t use features you don’t need, as any extra calculations significantly influence performance with high-quality video.

With the approach we’ve described in this article, we achieved 63 percent action detection accuracy at the first testing stages. Adding new training data to the dataset and further network fine-tuning may lead to even better results.


This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


About the Authors

Apriorit Inc
Chief Technology Officer Apriorit Inc.
United States United States
ApriorIT is a software research and development company specializing in cybersecurity and data management technology engineering. We work for a broad range of clients from Fortune 500 technology leaders to small innovative startups building unique solutions.

As Apriorit offers integrated research&development services for the software projects in such areas as endpoint security, network security, data security, embedded Systems, and virtualization, we have strong kernel and driver development skills, huge system programming expertise, and are reals fans of research projects.

Our specialty is reverse engineering, we apply it for security testing and security-related projects.

A separate department of Apriorit works on large-scale business SaaS solutions, handling tasks from business analysis, data architecture design, and web development to performance optimization and DevOps.

Official site:
Clutch profile:
Group type: Organisation

33 members

Vadym Zhernovyi
Software Developer Apriorit
Ukraine Ukraine
No Biography provided

Comments and Discussions

-- There are no messages in this forum --
Permalink | Advertise | Privacy | Cookies | Terms of Use | Mobile
Web04 | 2.8.190419.4 | Last Updated 22 Mar 2019
Article Copyright 2019 by Apriorit Inc, Vadym Zhernovyi
Everything else Copyright © CodeProject, 1999-2019
Layout: fixed | fluid