Click here to Skip to main content
Rate this: bad
good
Please Sign up or sign in to vote.
Lately I've been playing around with automation testing based on image processing.
 
The problem I am facing - I am trying to make a program that screenshots a webpage loaded into WebBrowser Control to an image, finds some images based on image processing and simulates user clicks on them, all this should be working while minimised.
The image recognition part is working perfectly.
The screenshot part was taken from . It bugs when the page has flash object. If I click on it manually, the screenshot is all white, and if I click outside of the flash object manually it's ok.
The clicking part rarely works, I couldn't find what causes it to glitch. I'm posting code, couldn't remember where I got it from.
class Interactor
    {
        public static WebBrowser WebBrowser;
 
        [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = false)]
        static extern IntPtr SendMessage(IntPtr hWnd, uint Msg, IntPtr wParam, IntPtr lParam);
 
        [DllImport("user32.dll", SetLastError = true)]
        static extern IntPtr GetWindow(IntPtr hWnd, uint uCmd);
 
        [DllImport("user32.dll", CharSet = CharSet.Auto)]
        static extern int GetClassName(IntPtr hWnd, StringBuilder lpClassName, int nMaxCount);
 

        public static void ClickAt(int x, int y)
        {
            x += 15;
            y += 232;
            IntPtr handle = WebBrowser.Handle;
            StringBuilder className = new StringBuilder(100);
            while (className.ToString() != "Internet Explorer_Server") // your mileage may vary with this classname
            {
                handle = GetWindow(handle, 5); // 5 == child
                GetClassName(handle, className, className.Capacity);
            }
            IntPtr lParam = (IntPtr)((y << 16) | x); // X and Y coordinates of the click
            IntPtr wParam = IntPtr.Zero; // change this if you want to simulate Ctrl-Click and such
            const uint downCode = 0x201; // these codes are for single left clicks
            const uint upCode = 0x202;
            SendMessage(handle, downCode, wParam, lParam); // mousedown
            SendMessage(handle, upCode, wParam, lParam); // mouseup
        }
 
    }
 
I'd prefer solutions using C# and WinForms, but other solutions are wellcome too.
Posted 21-Apr-13 18:18pm
Edited 21-Apr-13 18:19pm
v2
Comments
Sergey Alexandrovich Kryukov at 22-Apr-13 2:00am
   
I removed my first answer as not adequate enough. This problem needs more thinking than that...
New answer is ready.
—SA

1 solution

Rate this: bad
good
Please Sign up or sign in to vote.

Solution 2

Sending Windows messages does not solve the problem in all cases. You could use a very different approach: simulation of mouse input on a low level. This can be done using Windows API SendInput:
http://msdn.microsoft.com/en-us/library/windows/desktop/ms646310%28v=vs.85%29.aspx[^].
 
As you prefer using .NET (and I would agree with your), you can use P/Invoke. And you don't need to do it by yourself: this is already done for you: http://www.pinvoke.net/default.aspx/user32.sendinput[^].
 
This is a perfect tool for things like playing keyboard/mouse "macros", 100% reliable, as it perfectly simulate not the Windows messaging, but original low-level input, exactly as it would come from device drivers. (In fact, a device drivers used to use the predecessor of the mentioned function.)
 
—SA
  Permalink  
Comments
NKlinkachev at 8-May-13 15:50pm
   
Accepting this solution. I had to run everything in a Virtual Machine to have the computer running the tests usable during testing and the it was the only way to get around applications that pull cursor position where sending only click messages doesn't work. Capturing not active window or minimized window is possible though I couldn't make it 100% not glitchy. All of this however is additional requirements and this answer answers my question.
Sergey Alexandrovich Kryukov at 8-May-13 15:53pm
   
Great. Good luck, call again.
—SA

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

  Print Answers RSS
0 OriginalGriff 539
1 Maciej Los 300
2 DamithSL 233
3 Sergey Alexandrovich Kryukov 209
4 BillWoodruff 200
0 OriginalGriff 7,168
1 Sergey Alexandrovich Kryukov 6,377
2 DamithSL 5,461
3 Manas Bhardwaj 4,876
4 Maciej Los 4,450


Advertise | Privacy | Mobile
Web04 | 2.8.1411023.1 | Last Updated 22 Apr 2013
Copyright © CodeProject, 1999-2014
All Rights Reserved. Terms of Service
Layout: fixed | fluid

CodeProject, 503-250 Ferrand Drive Toronto Ontario, M3C 3G8 Canada +1 416-849-8900 x 100