Introduction
This article presents a class, handling sending of input (mouse and keyboard) to any running Windows Control by its handle.
Background
Some time ago, I was making a program which would win a penalty shootout soccer flash game for me. My task in the application was repeatedly to do these things:
- Grab flash window content
- Do some image processing and find out what to do
- Finally, take some action (send mouse input)
I found some great articles concerning taking of screenshot like this one.
This library helped me with image processing a lot.
On the other hand, I spent a lot of time trying to find out how to send mouse input to flash window correctly... and when I got how to do it, I decided to make this library.
Get the Handle
In order to use the TakeOver class, you need to obtain the handle (int number) of the window you want to control. It's possible to use spy++ tool from Visual Studio and convert HEX value to DEC using Windows calculator. The other possibility is to implement your own Window Picker (like in the demo project). Further details about this problem are out of scope for this article.
Using the Code
The first thing you need to do to take control over the window is to create a TakeOver class instance.
Remo.TakeOver tO = new Remo.TakeOver(targetWindowHandle);
Sending of input messages to a window is quite self-explanatory. The class provides these methods for sending messages:
public void SendLeftButtonDown(int x,int y);
public void SendLeftButtonUp(int x,int y);
public void SendLeftButtonDblClick(int x,int y);
public void SendRightButtonDown(int x,int y);
public void SendRightButtonUp(int x,int y);
public void SendRightButtonDblClick(int x,int y);
public void SendMouseMove(int x,int y);
public void SendKeyDown(int key);
public void SendKeyUp(int key);
public void SendChar(char c);
public void SendString(string s);
Sending input usually requires a target window to be focused to work properly, however it's not always so. The demo application can be tested here. Normally, you will have the application running in the background with controlled window focused, so there should be no problem with sending input messages. Since keyboard messages are not very reliable (I do not know how to encode lParam of SendMessage correctly, any hints?), it is recommended for keyboard input to use the SendKeys class once the window is focused via the SetFocus() method. Sendkeys is part of the standard System.Windows.Forms namespace. To focus target window TakeOver class provides a method:
public void SetFocus();
History
- 4th July, 2006: Initial post
| You must Sign In to use this message board. |
|
|
 |
|
|
Hi Michal,
My name's Nguyen, a Ho Chi Minh University of Natural Science's student
I've read your article and I think this is really a interested article. I am trying to write a small application to support for playing game.^_^
But I have some troubles when trying to test your demo
When I play some flash games and press a key to control my character.Your demo application can not catch the keyboard event,but when I try another.Your demo is OK
I read your code to understand the cause. You catch the KEYDOWN and KEYUP event of PictureBox1 but doesn't do anything
So, how can your application can work properly with my game?
this is link of the game I used to test:
http://www.trochoiviet.com/games/module-pnFlashGames-display-id-202.phtml[^]
I am waiting for your reply
thank you very much!
Nguyen
|
| Sign In·View Thread·PermaLink | |
|
|
|
|
|
|
It seems that the takeover application does not forward mouse messages properly. For example, if you play this swf file in your browser C:\WINDOWS\Help\Tours\mmTour\segment1.swf and then use the takeover app to play it in its window and then move the mouse over the active area of the flash movie in the takeover app, the active areas of the flash movie do not work. You can see the difference by moving the mouse over the active area in the browser.
Can you suggest some ways to fix this issue?
|
| Sign In·View Thread·PermaLink | 1.33/5 |
|
|
|
|
 | games 
-m-rt--n- | 3:45 28 Oct '06 |
|
|
|
|
|
Thank you for writing this article, very interesting ...
But now that you have 'Take(en)Over' the game, how do go about controlling it programmatically, if it is a 3rd party flash game then you will not have any knowledge of it's internals and equally how will your program know where the balls is (or know the position of any other element of game, especially if this it's position is Changing)
OB
|
| Sign In·View Thread·PermaLink | |
|
|
|
|
|
|
Well this is the most funny part of this job. You got to find it out by image processing. Static items are quite easy to find by using some color pattern from it. Dynamic items are more tricky and requires more processing skill. E.g. to find the ball witch is changins it's size, rotation, and position.. i used filters from Image processing lab. I found out constrains in which ball must be at beggining. Then grabbed whole image without ball on starting position (e.g. flying after kick). When needed to find out ball position i used Difference filter first on current grab and image without ball. Then i used Threshold Filter and got Black and White image with ball white and all the rest black. Then it was easy to find ball center from that. Hopefully it's comprehensible enough 
Mmm...
|
| Sign In·View Thread·PermaLink | 3.00/5 |
|
|
|
|
|
|
|
|
|
You should use Sendkeys class for it (System.Windows.Forms):
SendKeys.SendWait(^l);
It sends keys to foreground window.. if you wan't to set foreground window
programmaticaly use SetFocus(); method of TakeOver class.
|
| Sign In·View Thread·PermaLink | 3.00/5 |
|
|
|
|
|
|
