C# Application to Create and Recognize Mouse Gestures (.NET)

• Download source files - 169.02 KB
• Download demo project - 121.07 KB

Introduction

It was about time to update the content of this article.
The original article was written in 2004, and the code provided was updated one year later, even if it was not available to download from this page. That said, this software is able to recognize mouse gestures, and was originally inspired by this article.

Even though I took inspiration from the article posted by Konstantin Boukreev, this application is quite different (different programming language, gesture management and neural network at least), even though it uses a similar graphic interface (I found it amusing!) and the same logic behind the computation of gesture features.

This update contains a lot of changes (to be honest, the code has been completely rewritten), and provides an easier integration and a more flexible and extensible implementation of the neural network, moreover the solution file is for Visual Studio 2008.
More information will be provided below.

NOTE: If you are not familiar with artificial neural networks (ANN), please have a look at this page, since explaining such subject in detail goes beyond the scope of this article.

Details

The complete solution is divided into four projects:

1. NeuralNetworks: A library allowing neural networks creation and manipulation
2. MouseGestures: A library allowing mouse gestures definition
3. GestureRecognizer: An executable, providing a possible implementation and integration of previous libraries
4. Test: A console application used for testing purposes

Regarding the first project, it contains a feed forward neural network implementation (composed of layers of neurons), some typical activation functions to be assigned to the neurons, an implementation of the Backpropagation algorithm and the PerformanceMonitor class, used to compute statistics about the recognizing performances of the system.
The network implementation assumes that input features are already normalized (in other words there is no input layer dedicated to such a task).

The second library contains the implementation of the Gesture class, which is mainly a sorted collection of bidimensional points, and the GestureSet class (a collection of gestures, used to store training, test and validation sets used by the neural network), both providing load/save capabilities (files are saved in XML format).

Other than the above classes, there are some helper functions used to manipulate the geometry of the gestures.

The main difference between this version and the previous one is the ability to extract a variable number of features given the same number of gesture points; such a task is achieved by adding or removing points from the gesture, trying to minimize the deformation of the gesture (refer to the ExtractFeatures(int count) function in the Gesture.cs file).

The executable project is probably the most interesting, since it gives an immediate visual feedback to the user.

Just like the previous version, the classificator is composed of a set of competitive neural networks, each one used to recognize just one gesture; the main difference lies in the ability to customize the morphology of the net, choosing the number of inputs, layers, neurons per layer and type of activation functions (refer to menu entry Neural net -> Neural Network).

To use the application properly, first define the gestures you want to recognize (Settings -> Manage gestures -> Create gesture), then create the training set (Settings -> Manage gestures -> Create training set) and finally create the test set (Settings -> Manage gestures -> Create test set).

In this version, the gesture sets creation can be performed manually or automatically, moreover it is possible to choose different 'seeds' during automatic creation.
Once gesture sets are created, it is possible to start the training phase (Neural net -> Training and, at the end of that, try out the recognizing capabilities of the system by hand (NeuralNet -> Verify) or by using the performance monitor (NeuralNet -> Performances).
Another news is the possibility to set a confidence threshold for each neural network, which could improve recognizing abilities of the system.

Gestures, gesture sets and neural networks can be saved and loaded as XML files in the APP_PATH\Gestures\ folder; the application uses the following convention (GESTURE is a placeholder for the gesture name):

1. APP_PATH\Gestures\GESTURE.xml: File used to store the path of the gesture, training set, test set and neural network files
2. APP_PATH\Gestures\GESTURE\: Folder containing gesture, training set, test set and neural network XML files
3. APP_PATH\Gestures\GESTURE_G.xml: File containing gesture points
4. APP_PATH\Gestures\GESTURE_S.xml: File containing gestures used as training set
5. APP_PATH\Gestures\GESTURE_VS.xml: File containing gestures used as test set
6. APP_PATH\Gestures\GESTURE_NN.xml: File containing neural network data

About the Neural Network

In the previous version of this article, I wrote that it could be interesting to test out the performances of a classificator based on Autoassociators, compared to a classificator based on Multilayer Perceptrons; well, the good news is that the current version can deal with this: when the neural network input number is the same as the output number, training and verify phases behave differently, allowing the identification of which classificator performs better.

Conclusions

This article introduced a method to recognize mouse gestures using feedforward artificial neural networks, for educational purposes.

Two libraries were provided, which could be used to implement a custom gesture recognizing system; an application, contained in the solution, shows not only a possible implementation, but even a simple program which can be used to develop and test custom recognizing systems.

History

• 21st October, 2004: Article posted
• 14th March, 2008: Article updated (code has been completely rewritten)