12,506,574 members (53,419 online)
alternative version

291.5K views
186 bookmarked
Posted

# Mouse gestures recognition

, 22 Nov 2001
 Rate this:
Feedforward multilayer neural network and mouse gesture recognition

## Introduction

Recently I installed Opera 5 and was impressed on a Gesture UI. Moreover several weeks ago I noticed a discussion on CodeProject's Lounge about it. To all appearances people want to know about it too :). IMHO, the neural network most suitable for this purpose. As I a little know neural network I tried to implement such feature themselves.

What is Neural Network ? Hm it's not easy to say. A rephrased definition Zurada, J.M.:

"Neural network software is a software which can acquire, store, and utilize experiential knowledge."

I think I can point any person concerned to theory directly to several neural network sites. Here is small list of web resources about Neural networks:

## Implementation

Let's return to mouse gestures. After some research I have chosen a multilayer perceptron and standard back-propagation algorithm for training. The main problem was in the representation of an input data for neural network. The best result I found was in the transformation of a mouse path into a vector of cosines and sines.

For example:

```path   {170:82 172:83 175:85 177:86 ...}
transformed into
vector {0.45 0.55 0.45 0.71 ... 0.89 0.83 0.89 0.71 ...}
```

#### Recognition algorithm.

1. record a mouse path
2. smooth a path to a base points
3. transform points to angles' vector
4. compute sines and cosines
5. pass values (cosines and sines) to network's inputs
6. apply softmax function on an output network vector
7. find and verify a winner

#### Neural network architecture.

• input layers : 32 sinapses
• hidded layer : 32 neurons
• output layer : 29 axons (one for each gesture)
• fully connected layers
• transfer function : log-sigmoid
• incremental training algorithm, standard back-propagation method
• momentum, variable learning rate (slowly reduced)
• input noise

## Application

### Training

Before testing the recognition ability you must train the network (or you can load an file image of trained net). You can customize the parameters of the training process, namely: maximum number of cycles, a momentum value, a learning rate, a minimum value of mean square error (in other words "target error"). The training process will stop after achieving either of the conditions: maximum number of cycles or target error. During the training process you can keep an eye on a error's graph, a current gesture (with noise) and 2D network presentation.

### Testing

As soon as you have a trained net, you can test it. Select the patterns (or test all of them), a speed value and a noise level. Besides, you can familiarize oneself with ideal presentation of gestures via setting minimal noise and minimal speed.

### Recognition

For recognition of mouse gestures you must press right mouse button during moving a mouse. For example for recognition "left" gesture, press right mouse button and move a mouse to the left. If a neural network can recognize the gesture, then you will see the name, probability and ideal presentation of winner. Because of freeware nature of GestureApp the mouse path must have at least 16 points :(. Sorry I didn't implemented a "stretch a path" feature so far.

Note: the direction is very important.

The network is trained to recognize the gestures but not 2D images. Hence, you can draw the "circle" gesture a thousand different ways, but the only valid way is: press mouse button and move a mouse to the right and down and so on. Once more: it's gesture, not 2D image.

### Compatibility

Compatible with Win2k, WinXP, Win98, WinMe. Unfortunately doesn't work on WinNT because of the need for the AlphaBlend API.

## Acknowledgement

Special Thanks:
My wife Julia for her nice artwork ;)

And thanks to:
Pedro Pombeiro for Selection slider control

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

## About the Author

 Web Developer Russian Federation
I am freelance programmer. About 3 years of experience in C++ and I would rather use ATL, STL, WTL but not MFC . Main backgrounds are Win32 API, COM and Networking. Now I am interested about AI (Neural Network, Fuzzy Logic and GA). Currently based in Vladivostok, Russia.

## You may also be interested in...

 Pro Pro

## Comments and Discussions

 FirstPrev Next
 Re: Interpolating < 16 points. Code included Apuhjee19-Sep-08 5:45 Apuhjee 19-Sep-08 5:45
 Compiling with .NET quorm26-Aug-03 13:35 quorm 26-Aug-03 13:35
 Re: Compiling with .NET John M. Drescher26-Aug-03 14:01 John M. Drescher 26-Aug-03 14:01
 Re: Compiling with .NET quorm26-Aug-03 14:41 quorm 26-Aug-03 14:41
 Re: Compiling with .NET John M. Drescher26-Aug-03 14:50 John M. Drescher 26-Aug-03 14:50
 Not really know how to use your app! Chua Wen Ching1-Jul-03 7:27 Chua Wen Ching 1-Jul-03 7:27
 Re: Not really know how to use your app! matlin12326-Jul-03 8:51 matlin123 26-Jul-03 8:51
 Need help in image recognition alexpop19-Jun-03 3:36 alexpop 19-Jun-03 3:36
 Need help in speech recognition Mahesh Bhargava30-May-06 19:36 Mahesh Bhargava 30-May-06 19:36
 Need help in image recognition alexpop19-Jun-03 3:29 alexpop 19-Jun-03 3:29
 A great try, however ... xxxyyyzzz22-Oct-02 18:37 xxxyyyzzz 22-Oct-02 18:37
 what is this proggy good for ? Anonymous19-Jul-02 5:39 Anonymous 19-Jul-02 5:39
 Really Cool Tom Mason21-Mar-02 5:12 Tom Mason 21-Mar-02 5:12
 You may consider a little improvements... igor19604-Jan-02 21:49 igor1960 4-Jan-02 21:49
 Re: You may consider a little improvements... Konstantin Boukreev13-Jan-02 21:58 Konstantin Boukreev 13-Jan-02 21:58
 Re: You may consider a little improvements... Anonymous25-Sep-02 4:47 Anonymous 25-Sep-02 4:47
 Congratulations!!! Konstantin Vasserman18-Dec-01 2:25 Konstantin Vasserman 18-Dec-01 2:25
 Re: Congratulations!!! Konstantin Boukreev18-Dec-01 2:33 Konstantin Boukreev 18-Dec-01 2:33
 Re: Congratulations!!! Konstantin Vasserman18-Dec-01 2:50 Konstantin Vasserman 18-Dec-01 2:50
 Re: Congratulations!!! Konstantin Boukreev18-Dec-01 3:10 Konstantin Boukreev 18-Dec-01 3:10
 Re: Congratulations!!! Konstantin Vasserman18-Dec-01 3:30 Konstantin Vasserman 18-Dec-01 3:30
 Re: Congratulations!!! Konstantin Boukreev18-Dec-01 3:40 Konstantin Boukreev 18-Dec-01 3:40
 Re: Congratulations!!! Konstantin Boukreev18-Dec-01 3:17 Konstantin Boukreev 18-Dec-01 3:17
 Too good Robert Cao17-Dec-01 20:19 Robert Cao 17-Dec-01 20:19
 porting to Windows NT Marc Parisien17-Dec-01 4:23 Marc Parisien 17-Dec-01 4:23
 Re: porting to Windows NT Konstantin Boukreev18-Dec-01 2:38 Konstantin Boukreev 18-Dec-01 2:38
 GestureApp.exe - Unable To Locate ... Erik26-Nov-01 19:17 Erik 26-Nov-01 19:17
 Re: GestureApp.exe - Unable To Locate ... Konstantin Boukreev26-Nov-01 19:26 Konstantin Boukreev 26-Nov-01 19:26
 Re: GestureApp.exe - Unable To Locate ... Erik26-Nov-01 19:33 Erik 26-Nov-01 19:33
 Re: GestureApp.exe - Unable To Locate ... Konstantin Boukreev26-Nov-01 19:38 Konstantin Boukreev 26-Nov-01 19:38
 AlphaBlend Tim Smith24-Nov-01 4:29 Tim Smith 24-Nov-01 4:29
 Re: AlphaBlend Konstantin Boukreev24-Nov-01 16:32 Konstantin Boukreev 24-Nov-01 16:32
 Re: AlphaBlend Nish [BusterBoy]25-Nov-01 5:35 Nish [BusterBoy] 25-Nov-01 5:35
 Re: AlphaBlend Christian Graus9-Dec-01 16:53 Christian Graus 9-Dec-01 16:53
 Interesting Article Norm Almond23-Nov-01 23:41 Norm Almond 23-Nov-01 23:41
 Last Visit: 31-Dec-99 18:00     Last Update: 27-Sep-16 22:03 Refresh 1

General    News    Suggestion    Question    Bug    Answer    Joke    Praise    Rant    Admin

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.