# Mouse gestures recognition

By , 22 Nov 2001

## Introduction

Recently I installed Opera 5 and was impressed on a Gesture UI. Moreover several weeks ago I noticed a discussion on CodeProject's Lounge about it. To all appearances people want to know about it too :). IMHO, the neural network most suitable for this purpose. As I a little know neural network I tried to implement such feature themselves.

What is Neural Network ? Hm it's not easy to say. A rephrased definition Zurada, J.M.:

"Neural network software is a software which can acquire, store, and utilize experiential knowledge."

I think I can point any person concerned to theory directly to several neural network sites. Here is small list of web resources about Neural networks:

## Implementation

Let's return to mouse gestures. After some research I have chosen a multilayer perceptron and standard back-propagation algorithm for training. The main problem was in the representation of an input data for neural network. The best result I found was in the transformation of a mouse path into a vector of cosines and sines.

For example:

```path   {170:82 172:83 175:85 177:86 ...}
transformed into
vector {0.45 0.55 0.45 0.71 ... 0.89 0.83 0.89 0.71 ...}
```

#### Recognition algorithm.

1. record a mouse path
2. smooth a path to a base points
3. transform points to angles' vector
4. compute sines and cosines
5. pass values (cosines and sines) to network's inputs
6. apply softmax function on an output network vector
7. find and verify a winner

#### Neural network architecture.

• input layers : 32 sinapses
• hidded layer : 32 neurons
• output layer : 29 axons (one for each gesture)
• fully connected layers
• transfer function : log-sigmoid
• incremental training algorithm, standard back-propagation method
• momentum, variable learning rate (slowly reduced)
• input noise

## Application

### Training

Before testing the recognition ability you must train the network (or you can load an file image of trained net). You can customize the parameters of the training process, namely: maximum number of cycles, a momentum value, a learning rate, a minimum value of mean square error (in other words "target error"). The training process will stop after achieving either of the conditions: maximum number of cycles or target error. During the training process you can keep an eye on a error's graph, a current gesture (with noise) and 2D network presentation.

### Testing

As soon as you have a trained net, you can test it. Select the patterns (or test all of them), a speed value and a noise level. Besides, you can familiarize oneself with ideal presentation of gestures via setting minimal noise and minimal speed.

### Recognition

For recognition of mouse gestures you must press right mouse button during moving a mouse. For example for recognition "left" gesture, press right mouse button and move a mouse to the left. If a neural network can recognize the gesture, then you will see the name, probability and ideal presentation of winner. Because of freeware nature of GestureApp the mouse path must have at least 16 points :(. Sorry I didn't implemented a "stretch a path" feature so far.

Note: the direction is very important.

The network is trained to recognize the gestures but not 2D images. Hence, you can draw the "circle" gesture a thousand different ways, but the only valid way is: press mouse button and move a mouse to the right and down and so on. Once more: it's gesture, not 2D image.

### Compatibility

Compatible with Win2k, WinXP, Win98, WinMe. Unfortunately doesn't work on WinNT because of the need for the AlphaBlend API.

## Acknowledgement

Special Thanks:
My wife Julia for her nice artwork ;)

And thanks to:
Pedro Pombeiro for Selection slider control

A list of licenses authors might use can be found here

 Konstantin Boukreev Web Developer Russian Federation Member
I am freelance programmer. About 3 years of experience in C++ and I would rather use ATL, STL, WTL but not MFC . Main backgrounds are Win32 API, COM and Networking. Now I am interested about AI (Neural Network, Fuzzy Logic and GA). Currently based in Vladivostok, Russia.

Votes of 3 or less require a comment

 Search this forum Profile popups    Spacing RelaxedCompactTight   Noise Very HighHighMediumLowVery Low   Layout Open AllThread ViewNo JavascriptPreview   Per page 102550
 First Prev Next
 MLP? blibot1982 5 Aug '12 - 11:32
 My vote of 2 CliffordRoche 25 Feb '11 - 7:53
 Still very nice but needs a few changes baylorw 17 Nov '08 - 15:21
 i wasn't sure how well this would work but i tried it out and it's just darn neat. But it's now many years since this was posted and newer versions of Microsoft C++ won't compile this for all sorts of reasons. Here's what i remember changing to get it to work   Numerous errors due to pow. All arguments must be typecast to floating point It's used to do calculate the distance between the current and last point (using the Pythagorean theorem). Pow was being a pain so in the end, i wrote my own distance function:   ```double Board::getStraightLineDistance(POINT start, POINT end) { //--- Calculate how far the point has moved //--- Distance = straight line (Euclidean) distance long changeInX = start.x - end.x; long changeInY = start.y - end.y;   //--- Good old Pythagorean theorem: a^2 + b^2 = c^2 double distance = sqrt((double)(changeInX*changeInX) + (double)(changeInY*changeInY)); return distance; }```   Call it (in TransformPath) like: `d = getStraightLineDistance(*p, *i);` Note that i renamed a lot of the variables, so technically my code says distance=, not d=.   sqrt also complains for the same reason as pow. You have to explicitly convert all the parameters to a floating point number. Here's an example: ```m_cosines[n] = float(pt2.y / sqrt((double)(pt2.x * pt2.x) + (double)(pt2.y * pt2.y))); m_sinuses[n] = (float)sqrt(1. - (double)(m_cosines[n] * m_cosines[n]));```   Not sure how this ever compiled: `static vector_to_path(RECT& rc, vector_real_t& vec, path_t& path);` It has no return type. Even though it doesn't return anything, C++ (although not C) requires a return type. So i changed it to this: `static void vector_to_path(RECT& rc, vector_real_t& vec, path_t& path);`   i think every change i made was in Board with a single exception. SelectionSliderCtrl's definition is wrong. The first template parameter (type) needs to be written in it's own template fashion. Here's the right way to write it: ```template class SelectionSliderCtrlT : public CWindowImpl, T> ```   i think that's everything it took to get this to compile under VS2005. i don't think 2008 requires any additional changes but i haven't tested it yet   -baylor Sign In·View Thread·Permalink
 digitizer or pen tablet processing HOYAM 11 Jul '07 - 10:50
 congratulation (или поздравлаю и жилаю вам успихи все все..)   really very good working and i like it.   i have a tablet (genuis g-pen 340 3X4) connected to my computer through usb interface. i like to make a progarm for recognition pen movement in the tablet (i think it like your program). but i dont know how to start writting this program. how can i read these data through usb. can you help me with your advice and some steps to go in ride. how can i made use of your program(mouse gestures recognition)   thank you Sign In·View Thread·Permalink
 ge processing I also need document about image processing and code Member #3879553 1 Mar '07 - 15:54
 Integrate the project in a console opengl application? HELP! ninkata 28 Feb '07 - 23:46
 How guys! I am very impressed about the accuracy of this application! Congratulations. I would like to use it (or better - use a small subset of it) for the recognition of a couple of mouse gestures in a console opengl application. how should I integrate this project in my application and pass to it a set of points from my gui frontend? (I have especcialy a problem that your project is a MFC application and I havent done much with it, so I dont know how to port it to a console one)   I will be very very greatful if u give me a hint   Nina Sign In·View Thread·Permalink
 Mean square least to fit a circle Michae_geffen 7 Jan '07 - 10:00
 Is someone can help me to find an algorithm to calculate the best circle by given set of points {Xi,Yi};) Sign In·View Thread·Permalink
 need algorithm for VB6. anyone could help? PLEASE... Smart Canix 25 Jun '04 - 3:46
 Continous Monitoring adsjk 29 Dec '03 - 11:13
 Great program!   I am trying to extend it so that it contiously monitors mouse movements, ie you don't have to click to tell it you're about to start a gesture, instead it would continously monitor the mouse and if it ever performs a full circle (for example), it will pick it up.   Any one have any ideas how to go about this?   Cheers   Adam Sign In·View Thread·Permalink
 Interpolating < 16 points. Code included adamtegen 11 Nov '03 - 11:09
 In board.cpp ... bool Board::TransfromPath() { typedef path_t::iterator iterator;   m_path2.assign(m_path.begin(), m_path.end()); if ( 2 > m_path.size()) { m_path2.clear(); m_mode = recognizing_mode; return false; } else if (NUMBER_OF_ANCHOR_POINTS > m_path.size()) { // todo : stretch path or warning message; // m_path.clear();   //AJT interpolate to make the appropriate number of points; std::vector tempVector; for (std::list::iterator it = m_path.begin(); it != m_path.end(); it++) { tempVector.push_back(*it); } m_path2.clear(); long oldPointCount = m_path.size(); for (unsigned int iNewPoint = 0; iNewPoint < NUMBER_OF_ANCHOR_POINTS; iNewPoint++) { float newIndex = (float)(oldPointCount-1) * (float)iNewPoint/(float) (NUMBER_OF_ANCHOR_POINTS-1); int iPoint1 = (int)floor(newIndex); int iPoint2 = iPoint1 + 1;   float diff = (float)iPoint2 - newIndex;   float newX = 0.0f; float newY = 0.0f; newX = diff * tempVector[iPoint1].x; newY = diff * tempVector[iPoint1].y;   if (!IS_CLOSE(diff,1.0f,0.01)) { newX += ( 1.0f - diff) * tempVector[iPoint2].x; newY += ( 1.0f - diff) * tempVector[iPoint2].y; }   POINT newPt; newPt.x = (long)newX; newPt.y = (long)newY; m_path2.push_back(newPt); } //m_path2.push_back(tempVector[oldPointCount-1]) } else if (NUMBER_OF_ANCHOR_POINTS < m_path.size()) { ... Sign In·View Thread·Permalink
 Re: Interpolating < 16 points. Code included Apuhjee 19 Sep '08 - 5:45
 What compiler are you using that lets you use std::vector like this? Doesn't compile on VC 6, .NET 2005, or .NET 2008 for me...   I'd love to get this working, because as-is the swift right-to-left movements I'm used to with Opera are not getting accepted.   Regards ~ jp Sign In·View Thread·Permalink
 Compiling with .NET quorm 26 Aug '03 - 13:35
 Hi,   I'm having some trouble compiling this with Visual Studio .NET. I get the following errors:   Board.cpp(603): error C2589: '(' : illegal token on right side of '::' Board.cpp(603): error C2059: syntax error : '::'   The line containing the problem is:   m_max_error = std::_MAX(m_max_error, current_error);   If anyone has any suggestions for compiling this, please let me know. I've searched and haven't found much information.   Thanks,   Jordan Sign In·View Thread·Permalink
 Re: Compiling with .NET John M. Drescher 26 Aug '03 - 14:01
 Well I guess std::_MAX is not defined in vc7. There should be a max function somewhere in vc7 or define it yourself...   ```template T max(T a, T b) { return a > b ? a : b ; } ```   John Sign In·View Thread·Permalink
 Re: Compiling with .NET quorm 26 Aug '03 - 14:41
 Re: Compiling with .NET John M. Drescher 26 Aug '03 - 14:50
 I looked into the vc6 code. _MAX is a define that makes use of a inline template version of max similar to the one I posted above...   John Sign In·View Thread·Permalink
 Not really know how to use your app! Chua Wen Ching 1 Jul '03 - 7:27
 Hi there.   I am impressed with gestures and neural networks.   But i had no idea how to use it!   This was what i did:   1) Neural Net -> Train... (I had no idea what is it training)   2) I can see a red graph dropping fast!   3) Then, when it is done, should i choose Neural Net -> Test or Neural Net -> Recognize?   4) I assume you can test your gestures in Test! But when i test it, nothing special about it?   5) I can see the points running on the application... not sure what is the purpose for?   6) Comes back to Recognize, i drag some gestures with the right hand mouse, hmm.. nothing special happening!   Argh, friendly speaking, it is kind of confusing using your apps. Maybe you can give me some further guidelines on how your app works?   Is neural networks that necessary for mouse gestures? Is it because you allowed the end user to create their own gestures?   Any help?   Thanks.   Regards, Chua Wen Ching :p Sign In·View Thread·Permalink
 Re: Not really know how to use your app! matlin123 26 Jul '03 - 8:51
 Need help in image recognition alexpop 19 Jun '03 - 3:36
 Hi,   First of all I liked your project. Very good job. And thanks for a source code and explanation.   Now, I need help in image recognition. I have to write a program that recieves a shape/pattern/image of a control of some dialog box for example and recognizes it. I now that this is connected to Neural Network and Pattern Recognition. If you have some links or books or some idea about it please write me.   Thank you,   Alex Sign In·View Thread·Permalink
 Need help in speech recognition Mahesh Bhargava 30 May '06 - 19:36
 Hello Friend's I am working on speech recognition and i am very new in this field, If anyone who working on this type of project can help me. i am facing a problem that which neural network algo i an going to use for recognition. If anybody have some code or links please send me.     Mahesh Bhargava   mb_bhargava@yahoo.com Sign In·View Thread·Permalink
 Need help in image recognition alexpop 19 Jun '03 - 3:29
 Hi,   First of all I liked your project. Very good job, and thanks for a source code and explanation.   Now, I need help in image recognition. I have to write a program that recives a pattern/shape/image of a control of a Windows dialog box for example and would recognize it. I know that this is connected to Neural Network and Pattern Recognition. If you have some links or books or some idea about this please write me.   Thank you,   Alex Sign In·View Thread·Permalink
 A great try, however ... xxxyyyzzz 22 Oct '02 - 18:37
 what is this proggy good for ? Anonymous 19 Jul '02 - 5:39
 Really Cool Tom Mason 21 Mar '02 - 5:12
 You may consider a little improvements... igor1960 4 Jan '02 - 21:49
 To make your recognition direction of stroke independent you may consider small preprocessing before generating your input layer data... I mean => simply normalize your signal. To do that I would recommend scaling (if necessary???) input points and then rotating them into some normal position. The simpliest would be very simple calculation of Axes of Inertia => then rotating all set of points to move that axes horisontally => point rearanging, so array will start with lowest X... Will work => it did work 7 years ago when I was involved with Target recognition using backpropagation NN...   You are great, Kostia => again Molodec... Sign In·View Thread·Permalink
 Re: You may consider a little improvements... Konstantin Boukreev 13 Jan '02 - 21:58
 Hello Igor, Thanks for you suggestions. I already thought about it too but I put off implementation because of some complexity. I agree it will improve the usability of this code. Thanks again Sign In·View Thread·Permalink
 Re: You may consider a little improvements... Anonymous 25 Sep '02 - 4:47
 Congratulations!!! Konstantin Vasserman 18 Dec '01 - 2:25
 Привет Костя!   Поздравляю с победой в конкурсе на лучшую статью!   Спасибо за хороший пример и за то что ты возродил во мне желание снова начать заниматься искуственным итнеллектом и подобными вещами. Sign In·View Thread·Permalink
 Re: Congratulations!!! Konstantin Boukreev 18 Dec '01 - 2:33
 Привет тезка Спасибо за поздравление.   > Спасибо за хороший пример   Ну хорошее дело сделать не грех .   > снова начать заниматься искуственным итнеллектом   Тогда держи вопрос. Ты знаешь какой нибудь хороший пример использования зведы Гросберга для сжатия ? А то я только пока видел одни академические разработоки и не грамма исходников. Sign In·View Thread·Permalink
 Re: Congratulations!!! Konstantin Vasserman 18 Dec '01 - 2:50
 Боюсь что мне прийдется тебя разочаровать. Я не прикосался к этой тематике года наверное с 89-го... Поэтому, мои знания в этой области очень быстро стремятся к нулю...   Только когда я наткнулся на эту твою статью несколько дней назад и пошел по твоим ссылкам читать книги и т.д. - только тогда у меня в голове начали кое-какие аспекты всплывать. А так я нахожусь в полнейшей прострации на данный момент...   Я как раз надеялся, что ты можешь какого-нибудь хорошего чтива или хороших сайтов по этой тематике порекомендовать (помимо упомянутых в твоей статье)... Sign In·View Thread·Permalink
 Re: Congratulations!!! Konstantin Boukreev 18 Dec '01 - 3:10
 > Боюсь что мне прийдется тебя разочаровать.   Нет проблем   > Я как раз надеялся, что ты можешь какого-нибудь хорошего чтива > или хороших сайтов по этой тематике порекомендовать   С удовольствием.   Начнем с книг на русском. "Искусственные нейронные сети" Круглов и Борисов. 2001.   Скоро должна выйти "Основные концепции нейронных сетей" Каллан. на www.books.ru уже можно заказать http://books.ru/shop/books/21475   На http://neurnews.iu4.bmstu.ru/book/ есть подборка всеких разных материалов.   В инете можно найтьи статьи С.Короткий. Достаточно интересно. Можно скачать с http://stud.math.rsu.ru/actuar/dima/neiro5.ru.html.   Также если хочешь то могу выслать учебное пособие "И.Заенцев Нейронные сети:основные модели" в pdf (900 k)   Вот еще на русском, конспект лекций http://nuweb.jinr.ru/~nazaren/unc/nn_ru.html Sign In·View Thread·Permalink
 Re: Congratulations!!! Konstantin Vasserman 18 Dec '01 - 3:30
 Если не трудно, то пошли пожалуйста pdf на мой адрес [removed].   Огромное спасибо за рекомендации. Мне тут хватит на ближайшие месяцы наверное...   Кстати, ты вообще можешь мне писать по адресу (на верху) - я еще не знаю чем, но может чем нибудь я смогу быть тебе полезен.   Еще раз спасибо. Sign In·View Thread·Permalink
 Re: Congratulations!!! Konstantin Boukreev 18 Dec '01 - 3:40
 Re: Congratulations!!! Konstantin Boukreev 18 Dec '01 - 3:17
 Too good Robert Cao 17 Dec '01 - 20:19
 porting to Windows NT Marc Parisien 17 Dec '01 - 4:23
 Hi All!   Simple steps to port this code to NT:   1 - use gdiu.h and gdiu.cpp from http://www.wdj.com/articles/2001/0109/0109b/0109b.htm?topic=articles (don't forget aknowledgements : - ) Thanks Tim!)   2 - change calls to AlphaBlend() to AlphaBlendU(), the last parameter in AlphaBlendU will be: bf.SourceConstantAlpha (instead of the struct bf).   3 - in the file Main.cpp change the following line of code: ofn.lStructSize = sizeof OPENFILENAME; to ofn.lStructSize = OPENFILENAME_SIZE_VERSION_400;   et voila Sign In·View Thread·Permalink
 Re: porting to Windows NT Konstantin Boukreev 18 Dec '01 - 2:38
 GestureApp.exe - Unable To Locate ... Erik 26 Nov '01 - 19:17
 Hi,   I Like your articel, and would really try to test your application. However, I receive an empty Message-Box (Alert-Icon to the left), with no text in it, and the title mentioned in the subject. What's wrong? Also happens when I compile the projet myself (VC 6.0 with SP 5, NT 4.0 with SP 6)   Thank you Sign In·View Thread·Permalink
 Re: GestureApp.exe - Unable To Locate ... Konstantin Boukreev 26 Nov '01 - 19:26
 Re: GestureApp.exe - Unable To Locate ... Erik 26 Nov '01 - 19:33
 Hi,   thanks for the quick reply. Will it run under 2000? Anyway, I have 98 at home, so I am looking forward to try it at home later that afternoon.   Thanks,   Erik Sign In·View Thread·Permalink
 Re: GestureApp.exe - Unable To Locate ... Konstantin Boukreev 26 Nov '01 - 19:38
 AlphaBlend Tim Smith 24 Nov '01 - 4:29
 Some guy named Christian Graus wrote an AlphaBlend for 95 that you could use.   http://www.wdj.com/articles/2001/0109/0109b/0109b.htm?topic=articles   Tim Smith Descartes Systems Sciences, Inc. Sign In·View Thread·Permalink
 Re: AlphaBlend Konstantin Boukreev 24 Nov '01 - 16:32
 Re: AlphaBlend Nish [BusterBoy] 25 Nov '01 - 5:35
 Tim Smith wrote: Some guy named Christian Graus   Ouch!!!! And double ouch!!!!!!!!   Nish     Sonork ID 100.9786 voidmain www.busterboy.org Nish is a BIG fan of Goran Ivanisevic Sign In·View Thread·Permalink
 Re: AlphaBlend Christian Graus 9 Dec '01 - 16:53
 Nah - that guy is a clown. He's got no idea what he's talking about.     Christian   After all, there's nothing wrong with an elite as long as I'm allowed to be part of it!! - Mike Burston Oct 23, 2001   Sonork ID 100.10002:MeanManOzI live in Bob's HungOut now Sign In·View Thread·Permalink
 Interesting Article Norm Almond 23 Nov '01 - 23:41