Click here to Skip to main content
15,881,248 members
Please Sign up or sign in to vote.
2.00/5 (1 vote)
Lately I've been playing around with automation testing based on image processing.

The problem I am facing - I am trying to make a program that screenshots a webpage loaded into WebBrowser Control to an image, finds some images based on image processing and simulates user clicks on them, all this should be working while minimised.
The image recognition part is working perfectly.
The screenshot part was taken from . It bugs when the page has flash object. If I click on it manually, the screenshot is all white, and if I click outside of the flash object manually it's ok.
The clicking part rarely works, I couldn't find what causes it to glitch. I'm posting code, couldn't remember where I got it from.
C#
class Interactor
    {
        public static WebBrowser WebBrowser;

        [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = false)]
        static extern IntPtr SendMessage(IntPtr hWnd, uint Msg, IntPtr wParam, IntPtr lParam);

        [DllImport("user32.dll", SetLastError = true)]
        static extern IntPtr GetWindow(IntPtr hWnd, uint uCmd);

        [DllImport("user32.dll", CharSet = CharSet.Auto)]
        static extern int GetClassName(IntPtr hWnd, StringBuilder lpClassName, int nMaxCount);


        public static void ClickAt(int x, int y)
        {
            x += 15;
            y += 232;
            IntPtr handle = WebBrowser.Handle;
            StringBuilder className = new StringBuilder(100);
            while (className.ToString() != "Internet Explorer_Server") // your mileage may vary with this classname
            {
                handle = GetWindow(handle, 5); // 5 == child
                GetClassName(handle, className, className.Capacity);
            }
            IntPtr lParam = (IntPtr)((y << 16) | x); // X and Y coordinates of the click
            IntPtr wParam = IntPtr.Zero; // change this if you want to simulate Ctrl-Click and such
            const uint downCode = 0x201; // these codes are for single left clicks
            const uint upCode = 0x202;
            SendMessage(handle, downCode, wParam, lParam); // mousedown
            SendMessage(handle, upCode, wParam, lParam); // mouseup
        }

    }


I'd prefer solutions using C# and WinForms, but other solutions are wellcome too.
Posted
Updated 21-Apr-13 17:19pm
v2
Comments
Sergey Alexandrovich Kryukov 22-Apr-13 2:00am    
I removed my first answer as not adequate enough. This problem needs more thinking than that...
New answer is ready.
—SA

1 solution

Sending Windows messages does not solve the problem in all cases. You could use a very different approach: simulation of mouse input on a low level. This can be done using Windows API SendInput:
http://msdn.microsoft.com/en-us/library/windows/desktop/ms646310%28v=vs.85%29.aspx[^].

As you prefer using .NET (and I would agree with your), you can use P/Invoke. And you don't need to do it by yourself: this is already done for you: http://www.pinvoke.net/default.aspx/user32.sendinput[^].

This is a perfect tool for things like playing keyboard/mouse "macros", 100% reliable, as it perfectly simulate not the Windows messaging, but original low-level input, exactly as it would come from device drivers. (In fact, a device drivers used to use the predecessor of the mentioned function.)

—SA
 
Share this answer
 
Comments
NKlinkachev 8-May-13 15:50pm    
Accepting this solution. I had to run everything in a Virtual Machine to have the computer running the tests usable during testing and the it was the only way to get around applications that pull cursor position where sending only click messages doesn't work. Capturing not active window or minimized window is possible though I couldn't make it 100% not glitchy. All of this however is additional requirements and this answer answers my question.
Sergey Alexandrovich Kryukov 8-May-13 15:53pm    
Great. Good luck, call again.
—SA

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900