Click here to Skip to main content
11,705,763 members (58,175 online)
Click here to Skip to main content

C# Application to Create and Recognize Mouse Gestures (.NET)

, 17 Mar 2008 CPOL 155.8K 7K 142
Rate this:
Please Sign up or sign in to vote.
This program can create and recognize mouse gestures.
Sample Image - CMGBlade.jpg

Introduction

It was about time to update the content of this article.
The original article was written in 2004, and the code provided was updated one year later, even if it was not available to download from this page. That said, this software is able to recognize mouse gestures, and was originally inspired by this article.

Even though I took inspiration from the article posted by Konstantin Boukreev, this application is quite different (different programming language, gesture management and neural network at least), even though it uses a similar graphic interface (I found it amusing!) and the same logic behind the computation of gesture features.

This update contains a lot of changes (to be honest, the code has been completely rewritten), and provides an easier integration and a more flexible and extensible implementation of the neural network, moreover the solution file is for Visual Studio 2008.
More information will be provided below.

NOTE: If you are not familiar with artificial neural networks (ANN), please have a look at this page, since explaining such subject in detail goes beyond the scope of this article.

Details

The complete solution is divided into four projects:

  1. NeuralNetworks: A library allowing neural networks creation and manipulation
  2. MouseGestures: A library allowing mouse gestures definition
  3. GestureRecognizer: An executable, providing a possible implementation and integration of previous libraries
  4. Test: A console application used for testing purposes

Regarding the first project, it contains a feed forward neural network implementation (composed of layers of neurons), some typical activation functions to be assigned to the neurons, an implementation of the Backpropagation algorithm and the PerformanceMonitor class, used to compute statistics about the recognizing performances of the system.
The network implementation assumes that input features are already normalized (in other words there is no input layer dedicated to such a task).

The second library contains the implementation of the Gesture class, which is mainly a sorted collection of bidimensional points, and the GestureSet class (a collection of gestures, used to store training, test and validation sets used by the neural network), both providing load/save capabilities (files are saved in XML format).

Other than the above classes, there are some helper functions used to manipulate the geometry of the gestures.

The main difference between this version and the previous one is the ability to extract a variable number of features given the same number of gesture points; such a task is achieved by adding or removing points from the gesture, trying to minimize the deformation of the gesture (refer to the ExtractFeatures(int count) function in the Gesture.cs file).

The executable project is probably the most interesting, since it gives an immediate visual feedback to the user.

Just like the previous version, the classificator is composed of a set of competitive neural networks, each one used to recognize just one gesture; the main difference lies in the ability to customize the morphology of the net, choosing the number of inputs, layers, neurons per layer and type of activation functions (refer to menu entry Neural net -> Neural Network).

To use the application properly, first define the gestures you want to recognize (Settings -> Manage gestures -> Create gesture), then create the training set (Settings -> Manage gestures -> Create training set) and finally create the test set (Settings -> Manage gestures -> Create test set).

In this version, the gesture sets creation can be performed manually or automatically, moreover it is possible to choose different 'seeds' during automatic creation.
Once gesture sets are created, it is possible to start the training phase (Neural net -> Training and, at the end of that, try out the recognizing capabilities of the system by hand (NeuralNet -> Verify) or by using the performance monitor (NeuralNet -> Performances).
Another news is the possibility to set a confidence threshold for each neural network, which could improve recognizing abilities of the system.

Gestures, gesture sets and neural networks can be saved and loaded as XML files in the APP_PATH\Gestures\ folder; the application uses the following convention (GESTURE is a placeholder for the gesture name):

  1. APP_PATH\Gestures\GESTURE.xml: File used to store the path of the gesture, training set, test set and neural network files
  2. APP_PATH\Gestures\GESTURE\: Folder containing gesture, training set, test set and neural network XML files
  3. APP_PATH\Gestures\GESTURE_G.xml: File containing gesture points
  4. APP_PATH\Gestures\GESTURE_S.xml: File containing gestures used as training set
  5. APP_PATH\Gestures\GESTURE_VS.xml: File containing gestures used as test set
  6. APP_PATH\Gestures\GESTURE_NN.xml: File containing neural network data

About the Neural Network

In the previous version of this article, I wrote that it could be interesting to test out the performances of a classificator based on Autoassociators, compared to a classificator based on Multilayer Perceptrons; well, the good news is that the current version can deal with this: when the neural network input number is the same as the output number, training and verify phases behave differently, allowing the identification of which classificator performs better.

Conclusions

This article introduced a method to recognize mouse gestures using feedforward artificial neural networks, for educational purposes.

Two libraries were provided, which could be used to implement a custom gesture recognizing system; an application, contained in the solution, shows not only a possible implementation, but even a simple program which can be used to develop and test custom recognizing systems.

History

  • 21st October, 2004: Article posted
  • 14th March, 2008: Article updated (code has been completely rewritten)

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Daniele Mancini
Software Developer (Senior) Apex s.r.l.
Italy Italy
I got my Computer Science (Engineering) Master's Degree at the Siena University (Italy), but I'm from Rieti (a small town next to Rome).
My hobbies are RPG, MMORGP, programming and 3D graphics.
At the moment I'm employed at Apex s.r.l. (Modena, Italy) as a senior software developer, working for a WPF/WCF project in Rome.

You may also be interested in...

Comments and Discussions

 
SuggestionC# Application to Create and Recognize Mouse Gestures (.NET) Pin
Member 1111708722-Feb-15 21:35
memberMember 1111708722-Feb-15 21:35 
QuestionApp Pin
Virgr25-Jun-14 5:48
memberVirgr25-Jun-14 5:48 
Questionxml file Pin
jai bhim24-Mar-14 20:22
memberjai bhim24-Mar-14 20:22 
Questionhow calculation is being done when input is taken Pin
jai bhim20-Sep-13 23:39
memberjai bhim20-Sep-13 23:39 
AnswerRe: how calculation is being done when input is taken Pin
Daniele Mancini23-Sep-13 3:02
memberDaniele Mancini23-Sep-13 3:02 
GeneralRe: how calculation is being done when input is taken Pin
jai bhim26-Oct-13 7:30
memberjai bhim26-Oct-13 7:30 
QuestionRegarding Mouse Gestures Pin
Wisen Technologies3-Aug-13 3:30
groupWisen Technologies3-Aug-13 3:30 
AnswerRe: Regarding Mouse Gestures Pin
Daniele Mancini3-Aug-13 4:20
memberDaniele Mancini3-Aug-13 4:20 
GeneralMy vote of 5 Pin
Kanasz Robert6-Nov-12 0:00
mvpKanasz Robert6-Nov-12 0:00 
GeneralThanks a lot but got few questions Pin
lzf88888-Sep-10 16:23
memberlzf88888-Sep-10 16:23 
GeneralRe: Thanks a lot but got few questions Pin
Daniele Mancini12-Sep-10 21:49
memberDaniele Mancini12-Sep-10 21:49 
GeneralRe: Thanks a lot but got few questions Pin
blibot19826-Oct-11 20:55
memberblibot19826-Oct-11 20:55 
GeneralRe: Thanks a lot but got few questions Pin
Daniele Mancini6-Oct-11 21:01
memberDaniele Mancini6-Oct-11 21:01 
GeneralMessage Removed Pin
blibot19826-Oct-11 21:04
memberblibot19826-Oct-11 21:04 
GeneralRe: Thanks a lot but got few questions Pin
Daniele Mancini6-Oct-11 22:33
memberDaniele Mancini6-Oct-11 22:33 
GeneralRe: Thanks a lot but got few questions Pin
blibot19827-Oct-11 0:53
memberblibot19827-Oct-11 0:53 
GeneralRe: Thanks a lot but got few questions Pin
Daniele Mancini7-Oct-11 2:16
memberDaniele Mancini7-Oct-11 2:16 
GeneralMessage Removed Pin
blibot198210-Oct-11 6:39
memberblibot198210-Oct-11 6:39 
GeneralRe: Thanks a lot but got few questions Pin
Daniele Mancini10-Oct-11 11:37
memberDaniele Mancini10-Oct-11 11:37 
GeneralRe: Thanks a lot but got few questions Pin
Daniele Mancini10-Oct-11 23:49
memberDaniele Mancini10-Oct-11 23:49 
Questionfeature minimization Pin
Nick Vuono16-Apr-09 6:17
memberNick Vuono16-Apr-09 6:17 
AnswerRe: feature minimization Pin
Daniele Mancini16-Apr-09 7:50
memberDaniele Mancini16-Apr-09 7:50 
GeneralRe: feature minimization Pin
Nick Vuono16-Apr-09 9:11
memberNick Vuono16-Apr-09 9:11 
GeneralRe: feature minimization Pin
Daniele Mancini16-Apr-09 9:23
memberDaniele Mancini16-Apr-09 9:23 
Questionxml problem Pin
Oberst_BKnight13-Mar-09 12:26
memberOberst_BKnight13-Mar-09 12:26 
AnswerRe: xml problem Pin
Daniele Mancini13-Mar-09 13:43
memberDaniele Mancini13-Mar-09 13:43 
GeneralGreat Article Pin
Dreams...18-Nov-08 22:19
memberDreams...18-Nov-08 22:19 
GeneralRe: Great Article Pin
Daniele Mancini18-Nov-08 22:37
memberDaniele Mancini18-Nov-08 22:37 
QuestionCross-Thread-Calls Problem Pin
unkreativ3-Jul-07 9:05
memberunkreativ3-Jul-07 9:05 
AnswerRe: Cross-Thread-Calls Problem Pin
Daniele Mancini3-Jul-07 11:48
memberDaniele Mancini3-Jul-07 11:48 
GeneralRe: Cross-Thread-Calls Problem Pin
Lucradev12-Feb-08 1:10
memberLucradev12-Feb-08 1:10 
GeneralRe: Cross-Thread-Calls Problem Pin
Daniele Mancini11-Mar-08 14:16
memberDaniele Mancini11-Mar-08 14:16 
I have updated the article and the code (providing VS2008 solution files) and submitted everything to CodeProject. I hope they replace everything soon.
QuestionI have a question [modified] Pin
Seiga Alisue23-Jan-07 4:33
memberSeiga Alisue23-Jan-07 4:33 
AnswerRe: I have a question Pin
Blade Wise23-Jan-07 16:16
memberBlade Wise23-Jan-07 16:16 
GeneralvbAccelerator Pin
Serge Lobko-Lobanovsky29-Dec-04 22:51
memberSerge Lobko-Lobanovsky29-Dec-04 22:51 
GeneralRe: vbAccelerator Pin
Blade Wise30-Dec-04 0:17
memberBlade Wise30-Dec-04 0:17 
QuestionUsefullness? Pin
jewelsmith30-Oct-04 2:59
memberjewelsmith30-Oct-04 2:59 
AnswerRe: Usefullness? Pin
Blade Wise30-Oct-04 3:23
memberBlade Wise30-Oct-04 3:23 
AnswerRe: Usefullness? Pin
Labrat00231-Oct-04 12:05
memberLabrat00231-Oct-04 12:05 
Generalenglish comments Pin
dnh30-Oct-04 2:29
memberdnh30-Oct-04 2:29 
GeneralRe: english comments Pin
Blade Wise30-Oct-04 2:31
memberBlade Wise30-Oct-04 2:31 
GeneralRe: english comments Pin
dnh30-Oct-04 8:04
memberdnh30-Oct-04 8:04 
GeneralRe: english comments Pin
Blade Wise3-Nov-04 10:20
memberBlade Wise3-Nov-04 10:20 
GeneralRe: english comments Pin
dnh3-Nov-04 10:38
memberdnh3-Nov-04 10:38 
GeneralRe: english comments Pin
Blade Wise12-Nov-04 17:05
memberBlade Wise12-Nov-04 17:05 
GeneralRe: english comments Pin
Blade Wise27-Nov-04 4:12
memberBlade Wise27-Nov-04 4:12 
GeneralRe: english comments Pin
dnh29-Nov-04 1:51
memberdnh29-Nov-04 1:51 
GeneralRe: english comments Pin
Blade Wise29-Nov-04 2:19
memberBlade Wise29-Nov-04 2:19 
GeneralRe: english comments Pin
Blade Wise26-Dec-04 14:52
memberBlade Wise26-Dec-04 14:52 
GeneralGold! Pin
Dot Netter27-Oct-04 15:06
memberDot Netter27-Oct-04 15:06 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web04 | 2.8.150819.1 | Last Updated 18 Mar 2008
Article Copyright 2004 by Daniele Mancini
Everything else Copyright © CodeProject, 1999-2015
Layout: fixed | fluid