Click here to Skip to main content
15,891,951 members

WebBroser control - capture to image and simulate mouse clicks

NKlinkachev asked:

Open original thread
Lately I've been playing around with automation testing based on image processing.

The problem I am facing - I am trying to make a program that screenshots a webpage loaded into WebBrowser Control to an image, finds some images based on image processing and simulates user clicks on them, all this should be working while minimised.
The image recognition part is working perfectly.
The screenshot part was taken from . It bugs when the page has flash object. If I click on it manually, the screenshot is all white, and if I click outside of the flash object manually it's ok.
The clicking part rarely works, I couldn't find what causes it to glitch. I'm posting code, couldn't remember where I got it from.
C#
class Interactor
    {
        public static WebBrowser WebBrowser;

        [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = false)]
        static extern IntPtr SendMessage(IntPtr hWnd, uint Msg, IntPtr wParam, IntPtr lParam);

        [DllImport("user32.dll", SetLastError = true)]
        static extern IntPtr GetWindow(IntPtr hWnd, uint uCmd);

        [DllImport("user32.dll", CharSet = CharSet.Auto)]
        static extern int GetClassName(IntPtr hWnd, StringBuilder lpClassName, int nMaxCount);


        public static void ClickAt(int x, int y)
        {
            x += 15;
            y += 232;
            IntPtr handle = WebBrowser.Handle;
            StringBuilder className = new StringBuilder(100);
            while (className.ToString() != "Internet Explorer_Server") // your mileage may vary with this classname
            {
                handle = GetWindow(handle, 5); // 5 == child
                GetClassName(handle, className, className.Capacity);
            }
            IntPtr lParam = (IntPtr)((y << 16) | x); // X and Y coordinates of the click
            IntPtr wParam = IntPtr.Zero; // change this if you want to simulate Ctrl-Click and such
            const uint downCode = 0x201; // these codes are for single left clicks
            const uint upCode = 0x202;
            SendMessage(handle, downCode, wParam, lParam); // mousedown
            SendMessage(handle, upCode, wParam, lParam); // mouseup
        }

    }


I'd prefer solutions using C# and WinForms, but other solutions are wellcome too.
Tags: C#, WebBrowserControl, Mouse, Automation

Plain Text
ASM
ASP
ASP.NET
BASIC
BAT
C#
C++
COBOL
CoffeeScript
CSS
Dart
dbase
F#
FORTRAN
HTML
Java
Javascript
Kotlin
Lua
MIDL
MSIL
ObjectiveC
Pascal
PERL
PHP
PowerShell
Python
Razor
Ruby
Scala
Shell
SLN
SQL
Swift
T4
Terminal
TypeScript
VB
VBScript
XML
YAML

Preview



When answering a question please:
  1. Read the question carefully.
  2. Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
  3. If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
  4. Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.
Let's work to help developers, not make them feel stupid.
Please note that all posts will be submitted under the http://www.codeproject.com/info/cpol10.aspx.



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900