Click here to Skip to main content
15,860,972 members
Articles / Multimedia / GDI+
Article

C# Application to Create and Recognize Mouse Gestures (.NET)

Rate me:
Please Sign up or sign in to vote.
4.82/5 (39 votes)
17 Mar 2008CPOL5 min read 220.6K   8.1K   144   54
This program can create and recognize mouse gestures.
Sample Image - CMGBlade.jpg

Introduction

It was about time to update the content of this article.
The original article was written in 2004, and the code provided was updated one year later, even if it was not available to download from this page. That said, this software is able to recognize mouse gestures, and was originally inspired by this article.

Even though I took inspiration from the article posted by Konstantin Boukreev, this application is quite different (different programming language, gesture management and neural network at least), even though it uses a similar graphic interface (I found it amusing!) and the same logic behind the computation of gesture features.

This update contains a lot of changes (to be honest, the code has been completely rewritten), and provides an easier integration and a more flexible and extensible implementation of the neural network, moreover the solution file is for Visual Studio 2008.
More information will be provided below.

NOTE: If you are not familiar with artificial neural networks (ANN), please have a look at this page, since explaining such subject in detail goes beyond the scope of this article.

Details

The complete solution is divided into four projects:

  1. NeuralNetworks: A library allowing neural networks creation and manipulation
  2. MouseGestures: A library allowing mouse gestures definition
  3. GestureRecognizer: An executable, providing a possible implementation and integration of previous libraries
  4. Test: A console application used for testing purposes

Regarding the first project, it contains a feed forward neural network implementation (composed of layers of neurons), some typical activation functions to be assigned to the neurons, an implementation of the Backpropagation algorithm and the PerformanceMonitor class, used to compute statistics about the recognizing performances of the system.
The network implementation assumes that input features are already normalized (in other words there is no input layer dedicated to such a task).

The second library contains the implementation of the Gesture class, which is mainly a sorted collection of bidimensional points, and the GestureSet class (a collection of gestures, used to store training, test and validation sets used by the neural network), both providing load/save capabilities (files are saved in XML format).

Other than the above classes, there are some helper functions used to manipulate the geometry of the gestures.

The main difference between this version and the previous one is the ability to extract a variable number of features given the same number of gesture points; such a task is achieved by adding or removing points from the gesture, trying to minimize the deformation of the gesture (refer to the ExtractFeatures(int count) function in the Gesture.cs file).

The executable project is probably the most interesting, since it gives an immediate visual feedback to the user.

Just like the previous version, the classificator is composed of a set of competitive neural networks, each one used to recognize just one gesture; the main difference lies in the ability to customize the morphology of the net, choosing the number of inputs, layers, neurons per layer and type of activation functions (refer to menu entry Neural net -> Neural Network).

To use the application properly, first define the gestures you want to recognize (Settings -> Manage gestures -> Create gesture), then create the training set (Settings -> Manage gestures -> Create training set) and finally create the test set (Settings -> Manage gestures -> Create test set).

In this version, the gesture sets creation can be performed manually or automatically, moreover it is possible to choose different 'seeds' during automatic creation.
Once gesture sets are created, it is possible to start the training phase (Neural net -> Training and, at the end of that, try out the recognizing capabilities of the system by hand (NeuralNet -> Verify) or by using the performance monitor (NeuralNet -> Performances).
Another news is the possibility to set a confidence threshold for each neural network, which could improve recognizing abilities of the system.

Gestures, gesture sets and neural networks can be saved and loaded as XML files in the APP_PATH\Gestures\ folder; the application uses the following convention (GESTURE is a placeholder for the gesture name):

  1. APP_PATH\Gestures\GESTURE.xml: File used to store the path of the gesture, training set, test set and neural network files
  2. APP_PATH\Gestures\GESTURE\: Folder containing gesture, training set, test set and neural network XML files
  3. APP_PATH\Gestures\GESTURE_G.xml: File containing gesture points
  4. APP_PATH\Gestures\GESTURE_S.xml: File containing gestures used as training set
  5. APP_PATH\Gestures\GESTURE_VS.xml: File containing gestures used as test set
  6. APP_PATH\Gestures\GESTURE_NN.xml: File containing neural network data

About the Neural Network

In the previous version of this article, I wrote that it could be interesting to test out the performances of a classificator based on Autoassociators, compared to a classificator based on Multilayer Perceptrons; well, the good news is that the current version can deal with this: when the neural network input number is the same as the output number, training and verify phases behave differently, allowing the identification of which classificator performs better.

Conclusions

This article introduced a method to recognize mouse gestures using feedforward artificial neural networks, for educational purposes.

Two libraries were provided, which could be used to implement a custom gesture recognizing system; an application, contained in the solution, shows not only a possible implementation, but even a simple program which can be used to develop and test custom recognizing systems.

History

  • 21st October, 2004: Article posted
  • 14th March, 2008: Article updated (code has been completely rewritten)

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
Software Developer (Senior) Apex s.r.l.
Italy Italy
I got my Computer Science (Engineering) Master's Degree at the Siena University (Italy), but I'm from Rieti (a small town next to Rome).
My hobbies are RPG, MMORGP, programming and 3D graphics.
At the moment I'm employed at Apex s.r.l. (Modena, Italy) as a senior software developer, working for a WPF/WCF project in Rome.

Comments and Discussions

 
QuestionMessage Closed Pin
11-Aug-20 19:56
Member 1491245911-Aug-20 19:56 
SuggestionC# Application to Create and Recognize Mouse Gestures (.NET) Pin
Member 1111708722-Feb-15 21:35
Member 1111708722-Feb-15 21:35 
QuestionApp Pin
Virgr25-Jun-14 5:48
Virgr25-Jun-14 5:48 
Questionxml file Pin
Ask2924-Mar-14 20:22
Ask2924-Mar-14 20:22 
Questionhow calculation is being done when input is taken Pin
Ask2920-Sep-13 23:39
Ask2920-Sep-13 23:39 
AnswerRe: how calculation is being done when input is taken Pin
Daniele Mancini23-Sep-13 3:02
Daniele Mancini23-Sep-13 3:02 
GeneralRe: how calculation is being done when input is taken Pin
Ask2926-Oct-13 7:30
Ask2926-Oct-13 7:30 
QuestionRegarding Mouse Gestures Pin
Wisen Technologies3-Aug-13 3:30
Wisen Technologies3-Aug-13 3:30 
AnswerRe: Regarding Mouse Gestures Pin
Daniele Mancini3-Aug-13 4:20
Daniele Mancini3-Aug-13 4:20 
GeneralMy vote of 5 Pin
Kanasz Robert6-Nov-12 0:00
professionalKanasz Robert6-Nov-12 0:00 
GeneralThanks a lot but got few questions Pin
lzf88888-Sep-10 16:23
lzf88888-Sep-10 16:23 
GeneralRe: Thanks a lot but got few questions Pin
Daniele Mancini12-Sep-10 21:49
Daniele Mancini12-Sep-10 21:49 
GeneralRe: Thanks a lot but got few questions Pin
Am3320116-Oct-11 20:55
Am3320116-Oct-11 20:55 
GeneralRe: Thanks a lot but got few questions Pin
Daniele Mancini6-Oct-11 21:01
Daniele Mancini6-Oct-11 21:01 
GeneralMessage Closed Pin
6-Oct-11 21:04
Am3320116-Oct-11 21:04 
GeneralRe: Thanks a lot but got few questions Pin
Daniele Mancini6-Oct-11 22:33
Daniele Mancini6-Oct-11 22:33 
GeneralRe: Thanks a lot but got few questions Pin
Am3320117-Oct-11 0:53
Am3320117-Oct-11 0:53 
GeneralRe: Thanks a lot but got few questions Pin
Daniele Mancini7-Oct-11 2:16
Daniele Mancini7-Oct-11 2:16 
GeneralMessage Closed Pin
10-Oct-11 6:39
Am33201110-Oct-11 6:39 
GeneralRe: Thanks a lot but got few questions Pin
Daniele Mancini10-Oct-11 11:37
Daniele Mancini10-Oct-11 11:37 
GeneralRe: Thanks a lot but got few questions Pin
Daniele Mancini10-Oct-11 23:49
Daniele Mancini10-Oct-11 23:49 
Questionfeature minimization Pin
Nick Vuono16-Apr-09 6:17
Nick Vuono16-Apr-09 6:17 
AnswerRe: feature minimization Pin
Daniele Mancini16-Apr-09 7:50
Daniele Mancini16-Apr-09 7:50 
GeneralRe: feature minimization Pin
Nick Vuono16-Apr-09 9:11
Nick Vuono16-Apr-09 9:11 
GeneralRe: feature minimization Pin
Daniele Mancini16-Apr-09 9:23
Daniele Mancini16-Apr-09 9:23 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.