Click here to Skip to main content
15,892,059 members
Please Sign up or sign in to vote.
2.00/5 (2 votes)
See more:
I want to make a macro that will move the mouse, click, use keys. Click somewhere that has a group Pixel colors. Like all the shades of red in a rose on a grassy landscape. Purpose, be good for games, and maybe photos*h*op. I could get more creative but those are the basic things.

Which language should I use? What libraries would you recommend?

Exactly what I would want this macro to do.
1. Switch to a window or program that's opened based off its name or whatever to identify it.
2. Click somewhere once based off a group of colors.
3. Sleep
4. Hit a key on the keyboard
5. Repeat
6. And if it DOESN'T find the group of colors to click, it should hit a key on the keyboard, I'm guessing I could figure that out myself with some googling. Doing ifs, elses, and all dat.

This is one of the reasons i like programming because you can do crazy things, even if you can't make this perfect, just point me somewhere so i can learn, or use something to help me create this macro.

Found this code snippet on google, it supposively does everything I want. How would I get this to work in VB express 2010? With a form application, and a button I couldn't get it to work. Is this code incorrect for my needs, show me alternatives?

Where i found the code: http://thebotnet.com/programming/1810-vb-net-creating-rs-bot/

Functions:

FindControl 
ClickControl 
SendKey 
SendChar 
SendKeyString 
SendString 
GetScreenshot (Won't work when window is minimized) 
GetPixelColor (Won't work when window is minimized) 
GetControlSize 
PixelSearch 
ImageSearch (Added this since I thought it might be useful) (Won't work when window is minimized) 


The code that supposed to use those functions.

VB
Imports WindowsControls

'First you must find the window's handle by using Control.FindWindow
Dim Handle As IntPtr = Controls.FindWindow(WebBrowser1.Handle, "ControlNameHere")

'Create a new Controls object
Dim MyControl As New Controls(Handle)

'Start automating stuff here (this code doesn't do anything really)
MyControl.ClickControl(123, 456)
System.Threading.Thread.Sleep(1000)
Dim MyColor As Color = MyControl.GetPixelColor(12, 34)
Dim ClickPoint As Point = MyControl.PixelSearch(MyColor)(0)
MyControl.ClickControl(ClickPoint)

Dim ClickPoint2 As Point = MyControl.ImageSearch(My.Resources.ExamplePic)(0)
MyControl.ClickControl(ClickPoint2)


The class were I'm guessing some of the functions were made?

VB
Imports System
Imports System.Runtime.InteropServices
Imports Microsoft.Win32
Imports System.Windows.Forms
Imports System.Drawing
Imports System.Collections

Namespace WindowsControls
    Class Control
        Public Sub New(ByVal Handle As IntPtr)
            hWnd = Handle
        End Sub

        Private hWnd As IntPtr = IntPtr.Zero
        Public Property ControlHandle() As IntPtr
            Get
                Return hWnd
            End Get
            Set(ByVal value As IntPtr)
                hWnd = value
            End Set
        End Property

#Region "Interop"
         Const WM_KEYDOWN As UInteger = &H100
        Const WM_KEYUP As UInteger = &H101
        Const WM_PAINT As Int32 = &HF
        Const WM_LBUTTONDOWN As Int32 = &H201
        Const WM_LBUTTONUP As Int32 = &H202

        <DllImport("User32.dll", SetLastError:=True)> _
        Private Shared Function PrintWindow(ByVal hwnd As IntPtr, ByVal hDC As IntPtr, ByVal nFlags As UInteger) As Boolean
        End Function

        <DllImport("user32.dll", SetLastError:=True)> _
        Private Shared Function PostMessage(ByVal hWnd As IntPtr, ByVal Msg As UInteger, ByVal wParam As IntPtr, ByVal lParam As IntPtr) As <MarshalAs(UnmanagedType.Bool)> Boolean
        End Function

        <DllImport("user32.dll", CharSet:=CharSet.Auto, SetLastError:=False)> _
        Private Shared Function SendMessage(ByVal hWnd As IntPtr, ByVal Msg As Int32, ByVal wParam As IntPtr, ByVal lParam As IntPtr) As IntPtr
        End Function

        <DllImport("user32.dll", SetLastError:=True)> _
        Private Shared Function FindWindowEx(ByVal parentHandle As IntPtr, ByVal childAfter As IntPtr, ByVal className As String, ByVal windowTitle As IntPtr) As IntPtr
        End Function

        <DllImport("user32.dll")> _
        Private Shared Function GetWindowRect(ByVal hWnd As IntPtr, ByRef lpRect As RECT) As <MarshalAs(UnmanagedType.Bool)> Boolean
        End Function
        <StructLayout(LayoutKind.Sequential)> _
        Private Structure RECT
            Public Left As Integer
            Public Top As Integer
            Public Right As Integer
            Public Bottom As Integer
        End Structure

#End Region

        ''' <summary>
        ''' Gets the control handle
        ''' </summary>
        ''' <param name="parentHandle">The handle of the parent window</param>
        ''' <param name="controlName">The name of the control</param>
        ''' <returns>The control handle</returns>
        Public Shared Function FindControl(ByVal parentHandle As IntPtr, ByVal controlName As String) As IntPtr
            Return FindWindowEx(parentHandle, IntPtr.Zero, controlName, IntPtr.Zero)
        End Function

        ''' <summary>
        ''' Clicks control at specified coordinates
        ''' </summary>
        ''' <param name="hWnd">The window handle</param>
        ''' <param name="x">The x-coordinate</param>
        ''' <param name="y">The y-coordinate</param>
        Public Sub ClickControl(ByVal x As Integer, ByVal y As Integer)
            Dim coords As Integer = (y << 16) + x
            Dim lParam As New IntPtr(coords)
            PostMessage(hWnd, WM_LBUTTONDOWN, IntPtr.Zero, lParam)
            PostMessage(hWnd, WM_LBUTTONUP, IntPtr.Zero, lParam)
        End Sub

        ''' <summary>
        ''' Clicks the control at a specified point
        ''' </summary>
        ''' <param name="p">The point to click</param>
        Public Sub ClickControl(ByVal p As Point)
            ClickControl(p.X, p.Y)
        End Sub

        ''' <summary>
        ''' Sends a specified key to the control
        ''' </summary>
        ''' <param name="k">The key to send</param>
        Public Sub SendKey(ByVal k As Keys)
            PostMessage(hWnd, WM_KEYDOWN, New IntPtr(CInt(k)), IntPtr.Zero)
            PostMessage(hWnd, WM_KEYUP, New IntPtr(CInt(k)), IntPtr.Zero)
        End Sub

        ''' <summary>
        ''' Sends a character to the control
        ''' </summary>
        ''' <param name="c">The character to send</param>
        Public Sub SendChar(ByVal c As Char)
            Const WM_CHAR As UInteger = &H102
            PostMessage(hWnd, WM_CHAR, New IntPtr(Convert.ToInt32(c)), IntPtr.Zero)
        End Sub

        ''' <summary>
        ''' Sends a string to the control as keys
        ''' </summary>
        ''' <param name="s">The string to send</param>
        Public Sub SendKeyString(ByVal s As String)
            s = s.ToLower()
            For Each c As Char In s
                SendKey(DirectCast((Convert.ToInt32(c) - 32), Keys))
            Next
        End Sub

        ''' <summary>
        ''' Sends a string to the specified control
        ''' </summary>
        ''' <param name="s">The string to send</param>
        Private Sub SendString(ByVal s As String)
            For Each c As Char In s
                SendChar(c)
            Next
        End Sub

        ''' <summary>
        ''' Takes a screenshot of the control
        ''' </summary>
        ''' <returns>A bitmap of the control</returns>
        Public Function GetScreenshot() As Bitmap
            Dim controlSize As Size = GetControlSize()
            Dim bmp As New System.Drawing.Bitmap(controlSize.Width, controlSize.Height)
            Dim g As System.Drawing.Graphics = System.Drawing.Graphics.FromImage(bmp)
            Dim dc As IntPtr = g.GetHdc()

            SendMessage(hWnd, WM_PAINT, IntPtr.Zero, IntPtr.Zero)
            PrintWindow(hWnd, dc, 0)

            g.ReleaseHdc()
            g.Dispose()
            Return bmp
        End Function

        ''' <summary>
        ''' Gets the color of a selected pixel
        ''' </summary>
        ''' <param name="x">The x-coordinate</param>
        ''' <param name="y">The y-coordinate</param>
        ''' <returns>The color of the pixel</returns>
        Public Function GetPixelColor(ByVal x As Integer, ByVal y As Integer) As Color
            Return GetScreenshot().GetPixel(x, y)
        End Function

        ''' <summary>
        ''' Gets the pixel color of a specified point
        ''' </summary>
        ''' <param name="p">The point of the pixel</param>
        ''' <returns>The color of the pixel</returns>
        Public Function GetPixelColor(ByVal p As Point) As Color
            Return GetPixelColor(p.X, p.Y)
        End Function

        ''' <summary>
        ''' Gets the size of the control
        ''' </summary>
        ''' <returns>The size of the control</returns>
        Public Function GetControlSize() As Size
            Dim rect As RECT
            GetWindowRect(hWnd, rect)
            Return New Size(rect.Right - rect.Left, rect.Bottom - rect.Top)
        End Function

        ''' <summary>
        ''' Gets the points on the control with the specified color
        ''' </summary>
        ''' <param name="c">The color to search for</param>
        ''' <returns>The points of the pixels with the specified color</returns>
        Public Function PixelSearch(ByVal c As Color) As Point()
            Dim list As New System.Collections.ArrayList()
            Dim controlSize As Size = GetControlSize()
            Dim controlImage As Bitmap = GetScreenshot()
            For y As Integer = 0 To controlSize.Height - 1
                For x As Integer = 0 To controlSize.Width - 1
                    If controlImage.GetPixel(x, y) = c Then
                        list.Add(New Point(x, y))
                    End If
                Next
            Next
            Return DirectCast(list.ToArray(GetType(Point)), Point())
        End Function

        ''' <summary>
        ''' Searches for an image
        ''' </summary>
        ''' <param name="b">The image to search for in the control</param>
        ''' <returns>The points at the upper left corner of the image</returns>
        Public Function ImageSearch(ByVal b As Bitmap) As Point()
            Dim controlSize As Size = GetControlSize()
            Dim controlImage As Bitmap = GetScreenshot()
            Dim list As New ArrayList()
            For y As Integer = 0 To controlSize.Height - 1
                For x As Integer = 0 To controlSize.Width - 1
                    If controlImage.GetPixel(x, y) = b.GetPixel(0, 0) Then
                        Dim validPoint As Boolean = True
                        For by As Integer = 0 To b.Height - 1
                            For bx As Integer = 0 To b.Width - 1
                                If x + bx >= controlImage.Width OrElse y + by >= controlImage.Height Then
                                    validPoint = False
                                    Exit For
                                ElseIf controlImage.GetPixel(x + bx, y + by) <> b.GetPixel(bx, by) Then
                                    validPoint = False
                                    Exit For
                                End If
                            Next
                            If Not validPoint Then
                                Exit For
                            End If
                        Next
                        If validPoint Then
                            list.Add(New Point(x, y))
                        End If
                    End If
                Next
            Next
            Return DirectCast(list.ToArray(GetType(Point)), Point())
        End Function

        ''' <summary>
        ''' Searches for an image
        ''' </summary>
        ''' <param name="i">The image to search for in the control</param>
        ''' <returns>The points at the upper left corner of the image</returns>
        Public Function ImageSearch(ByVal i As Image) As Point()
            Return ImageSearch(DirectCast(i, Bitmap))
        End Function
    End Class
End Namespace


Too much p/invoke does that mean param invoking? I agree it doesn't look like VB.net code, lots of XML thrown in there.

I only know, HTML, CSS.

I've looked at other languages, and read up on them. I know that code has containers, and ways to start/end it. I know that imports go at the top. I know functions can have inputs, and outputs. I know that things are called, or linked to a lot in order to do whatever. Like some functions you have to link to them, also i think if you use namespaces or if you use imports you don't have to link to them sometimes or as much. There's tons of ways to structure code, if statements. I know that a class is where all the hard stuff goes, and you link to them sometimes just to use a function in it. Libraries you link to them to use functions, or whatever. Etc etc. I read up a lot on coding but i forget most of it. cout << printf writeDocument, bah i forget.

Also I don't know a lot of the vocabularly associated with coding. Like if you say your putting a sub base in a class object overflowing the namespace. I'm gonna O_O, just be aware I'd have to google everything your saying =_=

Thank you for looking at this... And to those that looked at my last question on this website
Posted
Updated 21-Dec-11 7:24am
v2

Back off from this project, because it's obvious (and I'm not trying to be mean) that you don't know anything about programming. "Knowing html and CSS" does NOT mean you have any kind of grasp on what it takes to write an application. The best thing you can do is to take a BUNCH of programming classes.
 
Share this answer
 
Comments
Tristan32 21-Dec-11 14:04pm    
Aw cmon, it doesn't sound too hard. If I imported the right stuff, and used the functions, and knew what to put where.

Like: Imports aaaaaaaaa

Code to start everythin aefioahwaoghaghaigwe faw
namespaces or w/e. Just have to look up tutorials on that and what to import and what namespaces to use, and what that does to everything.

var bunchofcolors1 = FindColorPixelLocationsOnScreen(Red)

then just use the functions
like ClickMouse{Coordinates(09,09)}
System.thread.whatever.Sleep(100)
if bunchofcolors1 is true it has found red pixels { Click(Red) }
else { Keydown(enter);
Keyup(enter); }

Explain?


etc etc, why should i back off from this it doesn't sound that hard?
#realJSOP 21-Dec-11 14:29pm    
If it "isn't that hard", why did you feel compelled to ask how to do it? For what it's worth, you don't have to accept my advice. I've only been doing this stuff for 30 years, so it's entirely possible that I'm just blowin' smoke up your ass.
Sergey Alexandrovich Kryukov 21-Dec-11 14:29pm    
I voted 5. Perhaps you have read the OP's code and I didn't, that's why you did not bother giving the technical advice and I did. Indeed, OP's code does not look promising; and the project would be rather difficult. But who knows how much a person can grow. (You might say "Look who is talking!" and you would be right. I often answer pretty much the way you just did.)

Cheers,
--SA
Tristan32 21-Dec-11 14:41pm    
It doesn't sound that hard because if there was already some genius that made the stuff then all I'd have to do is link to it, and use it. If i were to do everything from scratch sure it might be hard. Just cuz i don't like your opinion, doesn't mean I think your blowin smoke up my ass, or that I don't hear you out. Sorry for insulting you if I have in any way.
Sergey Alexandrovich Kryukov 21-Dec-11 14:48pm    
I answered it in a separate question. Again, you did not explain it all in full. In all cases, you will need to find a lot of stuff by yourself.
--SA
The answer is very simple, but the implementation is not very easy. This is doable, and practically only one way. I know 100% of the road map.

As I understand, you need to record and play macro over any applications in the system (as you mentioned games and Photoshop, which is not the only player even on its own market). You can use .NET, any language, C# preferred, but you also need a piece of some native code. I would use C++ or Delphi, but you can use any fully-fledged programming system producing Windows native code.

You need two kinds of action. First, you need to record all input events on raw level, and nothing else. The way to do it is using Windows Hooks. See http://msdn.microsoft.com/en-us/library/windows/desktop/ms632589%28v=vs.85%29.aspx[^].

Here is the problem: you can install a hook using P/Invoked Windows API SetWindowsHookEx, see http://msdn.microsoft.com/en-us/library/windows/desktop/ms644960%28v=vs.85%29.aspx#installing_releasing[^].

This would work fine, but only to hook up events of your process. To install a system-global hook, you need to call this function in a native DLL, according to Microsoft documentation. You can use this DLL in your .NET application and establish some communication with its native code which would call your .NET callback on each of the events. You should understand that the callback will be called in different threads, so you should use proper thread synchronization. This way, you will be able to record the event information needed for you macro record.

Second part of it is playing the recorded macro. You should use P/Invoked API function SendInput, and nothing else. See http://msdn.microsoft.com/en-us/library/windows/desktop/ms646310%28v=vs.85%29.aspx[^], http://pinvoke.net/default.aspx/user32.sendinput[^] (P/Invoke is already written for you).

To learn more of P/Invoke:
http://en.wikipedia.org/wiki/P/invoke[^],
http://msdn.microsoft.com/en-us/library/Aa712982[^] (don't worry, manual of 2003 is still good enough).

See also this CodeProject article: Essential P/Invoke[^].

That's basically it — get to work!

Good luck,
—SA
 
Share this answer
 
v4
Comments
Tristan32 21-Dec-11 14:07pm    
I haven't even read it yet lol nice timing. Thank you a lot for this, cuz googling Vb.net libraries didn't get me anywhere. Love your full answer this time did i get it because I made a better post? =D
Sergey Alexandrovich Kryukov 21-Dec-11 14:18pm    
Certainly. Hope it will work out for you, but you will need to do a good bunch of hard work. In particular, debugging of hooked code is not easy, as you cannot do it directly.
--SA
Tristan32 21-Dec-11 14:14pm    
Kk i read it, would this also record which pixel colors you clicked on? I've had tons of experience trying macro programs. Autohotkey, etc. None that I know of can click somewhere that has a specific color or group of colors.
Also if i had a game, a 3d game, it has a Tree in it in a grassy field, the tree has an image file. Could i link the image to my macro, tell it to click on it, and get it to click on it in the game with the grassy field. Like would that even work with 3d. If not just use the bunch of colors thing. This is me rush typing <<. Anyway thanks for giving me a starting point.
Sergey Alexandrovich Kryukov 21-Dec-11 14:21pm    
You will get exact coordinate of the mouse. Is that not enough? About the further stuff: process are isolated. You will get the events you set up you hook for. So, you will only get system-wide data, nothing specific to a particular application. From this standpoint, there is no such thing as "image". Well, perhaps we can discuss the feasibility if you explain your idea and the ultimate goal better.
--SA
Tristan32 21-Dec-11 14:30pm    
So your saying if I maximized a game on my computer, there's no code I can use that can detect a pixel color from that game, and click on it?
I really want that because I already know a macro program that records and plays and does mouse clicks, and etc.

Its not enuff because on the game if I rotated the camera, the spot I would want to click on might change. Like If i wanted to click on the tree and i rotated camera in the grassy field it might be on the right instead of its original location on the left.
Tristan32 wrote:

So your saying if I maximized a game on my computer, there's no code I can use that can detect a pixel color from that game, and click on it?
This is just one of the relatively difficult problems you would need to solve.

You would need to P/Invoke or use directly in your native code the following Windows API:

GetWindowDC(HWND) (user32), http://msdn.microsoft.com/en-us/library/dd144947%28v=vs.85%29.aspx[^]
ReleaseDC(HWND, HDC) (user32), http://msdn.microsoft.com/en-us/library/dd162920%28v=vs.85%29.aspx[^]
GetPixel(HDC, int, int) (gdi32), http://msdn.microsoft.com/en-us/library/dd144909%28v=vs.85%29.aspx[^].

Unfortunately, no one will do it all for you. You should be able to dig out a good deal of Windows API to use — mostly or all by yourself.

—SA
 
Share this answer
 
v2
Comments
Tristan32 21-Dec-11 14:56pm    
Its fine I'll probably come back here if I can't solve a problem by myself and after researching a lot. I'll figure it out. I think HWND was used in that class. Cool you gave me the libraries =D. I guess i couldve looked at the class file, and other file for them but whatever. I'll try my hardest. Thank you again.
Sergey Alexandrovich Kryukov 21-Dec-11 15:26pm    
I already gave you the "libraries" -- this is Window API ("user32.dll", "gdi32.dll", etc.). All you need is referenced. You can get all HWNDs from desktop HWND. Just get to MSDN and see all other Window-related APIs, see what a hook gives you and untie all data all the way down to the pixel. Keep in mind: addressing of individual pixel is slow.
--SA
December 11th, last year I posted that. Wowzers.
I've come so far with this, but still haven't gone down to far in it.

The solution I found is called: SCAR Divi - http://scar-divi.com

It can do everything a macro can do, and uses a Pascal like compiler. You can wrap libraries from Delphi for it, which I've done.

But it can't do the "recording" for macros. But that is fine with me now...

Now as for making my own macro application. I've figured out so far:

1. How to select windows, move em around, and get their handles, HDCs, and all that stuff.
2. Make em active, and get a screenshot of them using BitBlt.
3. Etc, used SendInput a lot and I am very happy with it.
4. TCanvas, Tons of graphics stuff I learned. Dialogs, a lot of basic functionality. TWebBrowser, etc...

So I think I covered a lot of the "semi old" ? stuff. I am very happy with Delphi, it requires less study to use. Less wordage, and things. And it is very fast (comparable to C++/C)?

Now as for recording for the macro. I can understand why you would use the Global hook. But as for using it for anything else, I just have no idea about the hook stuff.
You can get cursor position fairly easy, but to get what is typed I could see why you would use hooks. I think that might be fairly hard to deal with. Because of all the key combinations and such.

Thanks guys, and happy 11th?! lols

Also p.s. Nice this thread or topic has 600 views. Pretty good for a year
 
Share this answer
 
v3

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900