Click here to Skip to main content
Click here to Skip to main content

Emulate Human Mouse Input with Bezier Curves and Gaussian Distributions

, 14 Apr 2014 CPOL
Rate this:
Please Sign up or sign in to vote.
Emulate Human Mouse Input with Bezier Curves and Gaussian Distributions

Introduction

If one day you find yourself trying to write an application that pretends to be a human (i.e. moves the mouse and clicks buttons in a believable manner), this is one way to solve the problem.

I can think of very few (if any), legitimate reasons to use this code but perhaps Big Brother is watching you and instead you'd rather be down the pub?

The Problem

In .NET, it's very easy to send the mouse cursor to a position on screen:

 System.Windows.Forms.Cursor.Position = new System.Drawing.Point(x, y); 

We can even move it in straight lines:

for (int count = 1; count < 100; count++)
System.Windows.Forms.Cursor.Position 
            = new System.Drawing.Point(count, count); 

It's also relatively easy to call the Windows API to send mouse click events using the signature:

[DllImport("user32.dll")]
public static extern void mouse_event(int dwFlags, int dx, int dy, int dwData, int dwExtraInfo); 

But the average human wrist doesn't trace out straight lines and each line it does trace will be subtly different. Add to that a human is imperfect and doesn't always land the pointer accurately over the target first time. So if we're to automate our mundane clickety-click tasks and deceive Big Brother at the same time, then we have two separate problems to solve:

  1. Generating a curved mouse movement
  2. Distributing semi-accurate mouse clicks across a target

A Solution

  1. Rather than a simple circular arc, we can reasonably argue that mouse movements are somewhat better represented by a Bezier curve. Here is a nice graphical representation of Bezier curves and for our purposes we will use a simple 2D quadratic Bezier spline. This ensures that the curvature of the arc varies smoothly along its length.

    The excellent article Bezier Curves Made Simple by Tolga Birdal presents the algorithm we will use to generate intermediate points between the mouse start position and the intended target.

  2. To mimic inaccuracy in selecting the target, let's use a random normal distribution where the centre of the target co-ordinates are located at the top of the bell curve (i.e. the mean value).

The problem here is that the BCL Random class will generate a uniform distribution across the target. Not very realistic (), so let's transform it using the Box-Muller method as illustrated on stackoverflow and shown below:

Random gaussian = new Random();
bool haveNextNextGaussian = false;
double nextNextGaussian;  

public double NextGaussian()
        {
                double v1, v2, s;
                do
                {
                    v1 = 2 * gaussian.NextDouble() - 1;   
                    v2 = 2 * gaussian.NextDouble() - 1;
   
                    s = v1 * v1 + v2 * v2;
                } while (s >= 1 || s == 0);
                double multiplier = Math.Sqrt(-2 * Math.Log(s) / s);
                nextNextGaussian = v2 * multiplier;
                
                return v1 * multiplier;
        } 

So let's bring it all together in a function that will transform the intended X-Y co-ordinates of a move and click into an approximate list of points that we can then use to drive the mouse.

First off, let's modify the actual target:

public List<Point> MouseMoveAndClick(int x, int y)
{
int pointerAccuracy = 10;  
int targetX = x + Convert.ToInt32(pointerAccuracy * targetDistribution.NextGaussian());
int targetY = y + Convert.ToInt32(pointerAccuracy * targetDistribution.NextGaussian()); 

Then, we have to derive the control point (the co-ordinate that governs the curvature of our spline), which I've arbitrarily chosen to be a vector (of random length) normal from the midpoint between the start and the target:

//declare the original pointer position
int originalX = System.Windows.Forms.Cursor.Position.X;
int originalY = System.Windows.Forms.Cursor.Position.Y;

//find a mid point between original and target position
int midPointX = (x - targetX) / 2;
int midPointY = (y - targetY) / 2;

//Find a co-ordinate normal to the straight line between start and end point, starting at the midpoint and normally distributed
//This is reduced by a factor of 4 to model the arc of a right handed user.
int bezierMidPointX = Convert.ToInt32((midPointX / 4) * (midpointDistribution.NextGaussian()));
int bezierMidPointY = Convert.ToInt32((midPointY / 4) * (midpointDistribution.NextGaussian()));

With our start co-ordinates, our control points and our target points, we can generate the intermediate points:

BezierCurve bc = new BezierCurve();
double[] input = new double[] 
{ originalX, originalY, bezierMidPointX, bezierMidPointY, targetX, targetY };
                
int numberOfDataPoints = 1000;
double[] output = new double[numberOfDataPoints];
                
//co-ords are couplets of doubles hence the / 2
bc.Bezier2D(input, numberOfDataPoints / 2, output);

int pause = 0;

Then looping through the co-ordinates, we can modify the mouse position. For added fun, we can even slow the movement down as our human zeros in on his target:

List<System.Drawing.Point> points = new List<Point>();
for (int count = 1; count != numberOfDataPoints - 1; count += 2)
{
    Point point = new System.Drawing.Point((int)output[count + 1], (int)output[count]);
    points.Add(point);
    System.Windows.Forms.Cursor.Position = point;

    //we can vary when we pause between moving from point to point, but to replicate how a user's action will slow down prior to clicking:
    if ((count % 10) == 0)
        pause = 100 + ((count ^ 5) / (count * 2));

    System.Threading.Thread.Sleep(pause);                    
} 

Finally, send the left click events to the API:

    //Use the Win32 API to send a mouse down/ mouse up events separated by an arbitrary pause
    mouse_event((int)(MouseEventFlags.LEFTDOWN), 0, 0, 0, 0);
    System.Threading.Thread.Sleep(50);
    mouse_event((int)(MouseEventFlags.LEFTUP), 0, 0, 0, 0);

    return points;
} 

Conclusions

Of course, to make serious use of this method, you'll have to programmatically discover the locations of the targets your simulated human will click. Perhaps that will be the subject of a more extensive article, but for now feel free to tinker with the attached example.

History

  • #1 Code and explanation

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

CatchExAs
Technical Lead Alpha Integralis Limited
United Kingdom United Kingdom
CatchExAs aka Nick Wilton runs Alpha Integralis, a software development consultancy to companies in the City of London.
 
Main interests include exploring Algo-Trading, Quant Development and Physics using C# and Java.
 
www.nickwilton.info/Blog
Follow on   LinkedIn

Comments and Discussions

 
Generalgood one PinmemberOmar Gameel Salem15-Apr-14 14:16 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web04 | 2.8.141223.1 | Last Updated 14 Apr 2014
Article Copyright 2014 by CatchExAs
Everything else Copyright © CodeProject, 1999-2014
Layout: fixed | fluid