Click here to Skip to main content
Click here to Skip to main content

C# Application to Create and Recognize Mouse Gestures (.NET)

By , 17 Mar 2008
Rate this:
Please Sign up or sign in to vote.
Sample Image - CMGBlade.jpg

Introduction

It was about time to update the content of this article.
The original article was written in 2004, and the code provided was updated one year later, even if it was not available to download from this page. That said, this software is able to recognize mouse gestures, and was originally inspired by this article.

Even though I took inspiration from the article posted by Konstantin Boukreev, this application is quite different (different programming language, gesture management and neural network at least), even though it uses a similar graphic interface (I found it amusing!) and the same logic behind the computation of gesture features.

This update contains a lot of changes (to be honest, the code has been completely rewritten), and provides an easier integration and a more flexible and extensible implementation of the neural network, moreover the solution file is for Visual Studio 2008.
More information will be provided below.

NOTE: If you are not familiar with artificial neural networks (ANN), please have a look at this page, since explaining such subject in detail goes beyond the scope of this article.

Details

The complete solution is divided into four projects:

  1. NeuralNetworks: A library allowing neural networks creation and manipulation
  2. MouseGestures: A library allowing mouse gestures definition
  3. GestureRecognizer: An executable, providing a possible implementation and integration of previous libraries
  4. Test: A console application used for testing purposes

Regarding the first project, it contains a feed forward neural network implementation (composed of layers of neurons), some typical activation functions to be assigned to the neurons, an implementation of the Backpropagation algorithm and the PerformanceMonitor class, used to compute statistics about the recognizing performances of the system.
The network implementation assumes that input features are already normalized (in other words there is no input layer dedicated to such a task).

The second library contains the implementation of the Gesture class, which is mainly a sorted collection of bidimensional points, and the GestureSet class (a collection of gestures, used to store training, test and validation sets used by the neural network), both providing load/save capabilities (files are saved in XML format).

Other than the above classes, there are some helper functions used to manipulate the geometry of the gestures.

The main difference between this version and the previous one is the ability to extract a variable number of features given the same number of gesture points; such a task is achieved by adding or removing points from the gesture, trying to minimize the deformation of the gesture (refer to the ExtractFeatures(int count) function in the Gesture.cs file).

The executable project is probably the most interesting, since it gives an immediate visual feedback to the user.

Just like the previous version, the classificator is composed of a set of competitive neural networks, each one used to recognize just one gesture; the main difference lies in the ability to customize the morphology of the net, choosing the number of inputs, layers, neurons per layer and type of activation functions (refer to menu entry Neural net -> Neural Network).

To use the application properly, first define the gestures you want to recognize (Settings -> Manage gestures -> Create gesture), then create the training set (Settings -> Manage gestures -> Create training set) and finally create the test set (Settings -> Manage gestures -> Create test set).

In this version, the gesture sets creation can be performed manually or automatically, moreover it is possible to choose different 'seeds' during automatic creation.
Once gesture sets are created, it is possible to start the training phase (Neural net -> Training and, at the end of that, try out the recognizing capabilities of the system by hand (NeuralNet -> Verify) or by using the performance monitor (NeuralNet -> Performances).
Another news is the possibility to set a confidence threshold for each neural network, which could improve recognizing abilities of the system.

Gestures, gesture sets and neural networks can be saved and loaded as XML files in the APP_PATH\Gestures\ folder; the application uses the following convention (GESTURE is a placeholder for the gesture name):

  1. APP_PATH\Gestures\GESTURE.xml: File used to store the path of the gesture, training set, test set and neural network files
  2. APP_PATH\Gestures\GESTURE\: Folder containing gesture, training set, test set and neural network XML files
  3. APP_PATH\Gestures\GESTURE_G.xml: File containing gesture points
  4. APP_PATH\Gestures\GESTURE_S.xml: File containing gestures used as training set
  5. APP_PATH\Gestures\GESTURE_VS.xml: File containing gestures used as test set
  6. APP_PATH\Gestures\GESTURE_NN.xml: File containing neural network data

About the Neural Network

In the previous version of this article, I wrote that it could be interesting to test out the performances of a classificator based on Autoassociators, compared to a classificator based on Multilayer Perceptrons; well, the good news is that the current version can deal with this: when the neural network input number is the same as the output number, training and verify phases behave differently, allowing the identification of which classificator performs better.

Conclusions

This article introduced a method to recognize mouse gestures using feedforward artificial neural networks, for educational purposes.

Two libraries were provided, which could be used to implement a custom gesture recognizing system; an application, contained in the solution, shows not only a possible implementation, but even a simple program which can be used to develop and test custom recognizing systems.

History

  • 21st October, 2004: Article posted
  • 14th March, 2008: Article updated (code has been completely rewritten)

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

About the Author

Daniele Mancini
Software Developer (Senior) Apex s.r.l.
Italy Italy
I got my Computer Science (Engineering) Master's Degree at the Siena University (Italy), but I'm from Rieti (a small town next to Rome).
My hobbies are RPG, MMORGP, programming and 3D graphics.
At the moment I'm employed at Apex s.r.l. (Modena, Italy) as a senior software developer, working for a WPF/WCF project in Rome.
Follow on   Twitter

Comments and Discussions

 
Questionxml file Pinmemberjai bhim24-Mar-14 20:22 
Questionhow calculation is being done when input is taken Pinmemberjai bhim20-Sep-13 23:39 
AnswerRe: how calculation is being done when input is taken PinmemberDaniele Mancini23-Sep-13 3:02 
GeneralRe: how calculation is being done when input is taken Pinmemberjai bhim26-Oct-13 7:30 
QuestionRegarding Mouse Gestures PingroupWisen Technologies3-Aug-13 3:30 
AnswerRe: Regarding Mouse Gestures PinmemberDaniele Mancini3-Aug-13 4:20 
GeneralMy vote of 5 PinmvpKanasz Robert6-Nov-12 0:00 
GeneralThanks a lot but got few questions Pinmemberlzf88888-Sep-10 16:23 
GeneralRe: Thanks a lot but got few questions PinmemberDaniele Mancini12-Sep-10 21:49 
GeneralRe: Thanks a lot but got few questions Pinmemberblibot19826-Oct-11 20:55 
GeneralRe: Thanks a lot but got few questions PinmemberDaniele Mancini6-Oct-11 21:01 
GeneralMessage Removed Pinmemberblibot19826-Oct-11 21:04 
Message Removed
GeneralRe: Thanks a lot but got few questions PinmemberDaniele Mancini6-Oct-11 22:33 
GeneralRe: Thanks a lot but got few questions Pinmemberblibot19827-Oct-11 0:53 
GeneralRe: Thanks a lot but got few questions PinmemberDaniele Mancini7-Oct-11 2:16 
GeneralMessage Removed Pinmemberblibot198210-Oct-11 6:39 
GeneralRe: Thanks a lot but got few questions PinmemberDaniele Mancini10-Oct-11 11:37 
GeneralRe: Thanks a lot but got few questions PinmemberDaniele Mancini10-Oct-11 23:49 
Questionfeature minimization PinmemberNick Vuono16-Apr-09 6:17 
AnswerRe: feature minimization PinmemberDaniele Mancini16-Apr-09 7:50 
GeneralRe: feature minimization PinmemberNick Vuono16-Apr-09 9:11 
GeneralRe: feature minimization PinmemberDaniele Mancini16-Apr-09 9:23 
Questionxml problem PinmemberOberst_BKnight13-Mar-09 12:26 
AnswerRe: xml problem PinmemberDaniele Mancini13-Mar-09 13:43 
GeneralGreat Article PinmemberDreams...18-Nov-08 22:19 
GeneralRe: Great Article PinmemberDaniele Mancini18-Nov-08 22:37 
QuestionCross-Thread-Calls Problem Pinmemberunkreativ3-Jul-07 9:05 
AnswerRe: Cross-Thread-Calls Problem PinmemberDaniele Mancini3-Jul-07 11:48 
GeneralRe: Cross-Thread-Calls Problem PinmemberLucradev12-Feb-08 1:10 
GeneralRe: Cross-Thread-Calls Problem PinmemberDaniele Mancini11-Mar-08 14:16 
QuestionI have a question [modified] PinmemberSeiga Alisue23-Jan-07 4:33 
AnswerRe: I have a question PinmemberBlade Wise23-Jan-07 16:16 
GeneralvbAccelerator PinmemberSerge Lobko-Lobanovsky29-Dec-04 22:51 
GeneralRe: vbAccelerator PinmemberBlade Wise30-Dec-04 0:17 
QuestionUsefullness? Pinmemberjewelsmith30-Oct-04 2:59 
AnswerRe: Usefullness? PinmemberBlade Wise30-Oct-04 3:23 
AnswerRe: Usefullness? PinmemberLabrat00231-Oct-04 12:05 
Generalenglish comments Pinmemberdnh30-Oct-04 2:29 
GeneralRe: english comments PinmemberBlade Wise30-Oct-04 2:31 
GeneralRe: english comments Pinmemberdnh30-Oct-04 8:04 
GeneralRe: english comments PinmemberBlade Wise3-Nov-04 10:20 
GeneralRe: english comments Pinmemberdnh3-Nov-04 10:38 
GeneralRe: english comments PinmemberBlade Wise12-Nov-04 17:05 
GeneralRe: english comments PinmemberBlade Wise27-Nov-04 4:12 
GeneralRe: english comments Pinmemberdnh29-Nov-04 1:51 
GeneralRe: english comments PinmemberBlade Wise29-Nov-04 2:19 
GeneralRe: english comments PinmemberBlade Wise26-Dec-04 14:52 
GeneralGold! PinmemberDot Netter27-Oct-04 15:06 
GeneralRe: Gold! PinmemberBlade Wise27-Oct-04 16:17 
GeneralNice job PinmemberPatrik Svensson22-Oct-04 2:25 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web01 | 2.8.140415.2 | Last Updated 18 Mar 2008
Article Copyright 2004 by Daniele Mancini
Everything else Copyright © CodeProject, 1999-2014
Terms of Use
Layout: fixed | fluid