Leap Motion: Move Cursor

Meshack Musundi

4.91/5 (20 votes)

Feb 21, 2013

CPOL

4 min read

120429

4287

How to use the Leap controller to move the cursor using a finger

Introduction

The Leap is a 3D motion sensing device that detects and tracks hands, fingers and finger-like tools. In this article I'll describe how you can go about using the Leap, and the Leap SDK, to create a .NET console application that enables you to move the cursor using your finger.

Requirements

To run the sample project you require the following,

Leap (developer unit),
Leap SDK,
VS2012

If you don't have a Leap controller you can apply to join the developer program and, hopefully, get a developer unit for free. Joining the developer program will also grant you access to the Leap SDK.

NB: The example project described here was developed using dev unit v.06.5 and version 0.7.3 of the Leap SDK.

Running the Project

To run the sample project you have to add Leap.dll and _LeapCSharp.dll to the project. (These two files are part of the Leap SDK). Ensure that the Copy to Output Directory property of the two files is set to Copy always.

Ensure that you also add a reference to LeapCSharp.NET4.0.dll, which is also part of the Leap SDK, and contains classes that enable you to access the data from the Leap controller.

Leap Coordinate System

Leap uses a right-handed Cartesian coordinate system, with the origin at the center of the controller and the X and Z axes lying in the horizontal plane. The Y axis is vertical.

Moving the Cursor

In order to move the cursor we'll need to call a function in the win32 API. This is done by a method in class MouseCursor.

Public Class MouseCursor

    Private Declare Function SetCursorPos Lib "user32" (x As Integer, y As Integer) As Boolean

    Public Shared Sub MoveCursor(x As Integer, y As Integer)
        SetCursorPos(x, y)
    End Sub

End Class

using System;
using System.Runtime.InteropServices;
using System.Windows;
using System.Windows.Interop;

namespace ConsoleLeapMouse
{
    class MouseCursor
    {
        [DllImport("user32.dll")]
        private static extern bool SetCursorPos(int x, int y);

        public static void MoveCursor(int x, int y)
        {
            SetCursorPos(x, y);
        }

    }
}

MoveCursor() takes two parameters that define the location on the screen where the cursor should move. To get the values for these parameters we'll need to process data coming from the Leap controller.

To get data from the Leap we need to create a Controller object. The Controller object connects the console application with the Leap software and provides hand and finger tracking data through Frame objects.

Imports Leap

Module Module1

    Sub Main()
        Dim cntrl As New Controller
        Dim listener As New LeapListener

        cntrl.AddListener(listener)

        Console.WriteLine("Press Enter to quit...")
        Console.ReadLine()

        cntrl.RemoveListener(listener)
        cntrl.Dispose()
    End Sub

End Module

using System;
using Leap;

namespace ConsoleLeapMouse
{
    class Program
    {
        static void Main(string[] args)
        {
            Controller cntrl = new Controller();
            LeapListener listener = new LeapListener();

            cntrl.AddListener(listener);

            Console.WriteLine("Press Enter to quit...");
            Console.ReadLine();

            cntrl.RemoveListener(listener);
            cntrl.Dispose();       
        }
    }
}

In the code snippet above an object of type LeapListener is added to the Controller object. LeapListener is a subclass of the Listener class which defines a set of callback methods that can be overridden to respond to events dispatched by the Leap.

Imports Leap

Public Class LeapListener
    Inherits Listener

    Public Overrides Sub OnInit(cntrlr As Controller)
        Console.WriteLine("Initialized")
    End Sub

    Public Overrides Sub OnConnect(cntrlr As Controller)
        Console.WriteLine("Connected")
    End Sub

    Public Overrides Sub OnDisconnect(cntrlr As Controller)
        Console.WriteLine("Disconnected")
    End Sub

    Public Overrides Sub OnExit(cntrlr As Controller)
        Console.WriteLine("Exited")
    End Sub

    Private currentTime As Long
    Private previousTime As Long
    Private timeChange As Long

    Public Overrides Sub OnFrame(cntrlr As Controller)
        ' Get the current frame.
        Dim currentFrame As Frame = cntrlr.Frame

        currentTime = currentFrame.Timestamp
        timeChange = currentTime - previousTime

        If (timeChange > 10000) Then
            If (Not currentFrame.Hands.Empty) Then
                ' Get the first finger in the list of fingers.
                Dim finger = currentFrame.Fingers(0)
                ' Gets the closest screen intercepting a ray projecting from the finger.
                Dim screen = cntrlr.CalibratedScreens.ClosestScreenHit(finger)

                If (screen IsNot Nothing And screen.IsValid) Then
                    ' Get the velocity of the finger tip.
                    Dim tipVelocity = finger.TipVelocity.Magnitude

                    ' Use tipVelocity to reduce jitters when attempting
                    ' to hold the cursor steady.
                    If (tipVelocity > 25) Then
                        Dim xScreenIntersect = screen.Intersect(finger, True).x
                        Dim yScreenIntersect = screen.Intersect(finger, True).y

                        If (xScreenIntersect.ToString <> "NaN") Then
                            Dim x = CInt(xScreenIntersect * screen.WidthPixels)
                            Dim y = screen.HeightPixels - CInt(yScreenIntersect * screen.HeightPixels)

                            Console.WriteLine("Screen intersect X: " & xScreenIntersect.ToString)
                            Console.WriteLine("Screen intersect Y: " & yScreenIntersect.ToString)
                            Console.WriteLine("Width pixels: " & screen.WidthPixels.ToString)
                            Console.WriteLine("Height pixels: " & screen.HeightPixels.ToString)

                            Console.WriteLine(vbCrLf)

                            Console.WriteLine("x: " & x.ToString)
                            Console.WriteLine("y: " & y.ToString)

                            Console.WriteLine(vbCrLf)

                            Console.WriteLine("Tip velocity: " & tipVelocity)

                            ' Move the cursor
                            MouseCursor.MoveCursor(x, y)

                            Console.WriteLine(vbCrLf & New String("=", 40) & vbCrLf)
                        End If
                    End If

                End If
            End If

            previousTime = currentTime
        End If

    End Sub

End Class

using System;
using Leap;

namespace ConsoleLeapMouse
{
    class LeapListener: Listener
    {
        public override void OnInit(Controller cntrlr)
        {
            Console.WriteLine("Initialized");
        }

        public override void OnConnect(Controller cntrlr)
        {
            Console.WriteLine("Connected");
        }

        public override void OnDisconnect(Controller cntrlr)
        {
            Console.WriteLine("Disconnected");
        }

        public override void OnExit(Controller cntrlr)
        {
            Console.WriteLine("Exited");
        }

        private long currentTime;
        private long previousTime;
        private long timeChange;

        public override void OnFrame(Controller cntrlr)
        {
            // Get the current frame.
            Frame currentFrame = cntrlr.Frame();

            currentTime = currentFrame.Timestamp;
            timeChange = currentTime - previousTime;
           
            if (timeChange > 10000)
            {
                if (!currentFrame.Hands.Empty)
                {
                    // Get the first finger in the list of fingers
                    Finger finger = currentFrame.Fingers[0];
                    // Get the closest screen intercepting a ray projecting from the finger
                    Screen screen = cntrlr.CalibratedScreens.ClosestScreenHit(finger);

                    if (screen != null && screen.IsValid)
                    {
                        // Get the velocity of the finger tip
                        var tipVelocity = (int)finger.TipVelocity.Magnitude;

                        // Use tipVelocity to reduce jitters when attempting to hold
                        // the cursor steady
                        if (tipVelocity > 25)
                        {
                            var xScreenIntersect = screen.Intersect(finger, true).x;
                            var yScreenIntersect = screen.Intersect(finger, true).y;
                            
                            if (xScreenIntersect.ToString() != "NaN")
                            {
                                var x = (int)(xScreenIntersect * screen.WidthPixels);
                                var y = (int)(screen.HeightPixels - (yScreenIntersect * screen.HeightPixels));

                                Console.WriteLine("Screen intersect X: " + xScreenIntersect.ToString());
                                Console.WriteLine("Screen intersect Y: " + yScreenIntersect.ToString());
                                Console.WriteLine("Width pixels: " + screen.WidthPixels.ToString());
                                Console.WriteLine("Height pixels: " + screen.HeightPixels.ToString());

                                Console.WriteLine("\n");

                                Console.WriteLine("x: " + x.ToString());
                                Console.WriteLine("y: " + y.ToString());

                                Console.WriteLine("\n");

                                Console.WriteLine("Tip velocity: " + tipVelocity.ToString());

                                // Move the cursor
                                MouseCursor.MoveCursor(x, y);

                                Console.WriteLine("\n" + new String('=', 40) + "\n");
                            }

                        }
                    }

                }

                previousTime = currentTime;
            }
        }
    }
}

The most significant method in the code snippet above is the OnFrame() method. The OnFrame() method is called by the Controller object when a new frame of hand and finger tracking data is available. The data is accessed by calling the Controller object's Frame() method. Since the Leap has a really high frame rate, I guess a maximum of about 115fps, attempting to move the cursor at every call of the OnFrame() method will not give the desired results. For this reason I have specified an interval of >10ms between frames for processing of tracking data.

NB: Frame.Timestamp() is the frame capture time in microseconds elapsed since the Leap started.

To get the screen that a finger is pointing to we use the Controller.CalibratedScreens.ClosestScreenHit() method and pass it a Finger object, which in this case is the first Finger object in a list of Finger objects. The specified screen will be the closest screen intercepting a ray projecting from the Finger object. The projected ray emanates from the Finger TipPosition along its Direction vector.

Finger TipPosition and Direction vectors

To get the screen position in pixels, where the finger is pointing at, we multiply the values returned by Screen.WidthPixels() and Screen.HeightPixels() with the normalized coordinates returned by Screen.Intersect(). Normalized coordinates represent the intersection point between the screen and the ray projecting from the Finger object, defined as a percentage of the screen's width and height. Such coordinates would be of the form (0.1, 0.1, 0).

...
Dim xScreenIntersect = screen.Intersect(finger, True).x
Dim yScreenIntersect = screen.Intersect(finger, True).y

If (xScreenIntersect.ToString <> "NaN") Then
    Dim x = CInt(xScreenIntersect * screen.WidthPixels)
    Dim y = screen.HeightPixels - CInt(yScreenIntersect * screen.HeightPixels)
    ...          
End If

...
var xScreenIntersect = screen.Intersect(finger, true).x;
var yScreenIntersect = screen.Intersect(finger, true).y;
                            
if (xScreenIntersect.ToString() != "NaN")
{
    var x = (int)(xScreenIntersect * screen.WidthPixels);
    var y = (int)(screen.HeightPixels - (yScreenIntersect * screen.HeightPixels));
    ...
}

In case you are wondering why I'm setting the value of the y variable by doing screen.HeightPixels - (yScreenIntersect * screen.HeightPixels), it's because my computer's graphics coordinate system places the screen's origin at the bottom-left corner.

NB: If the Finger object is pointing parallel to or away from the screen the components of the vector returned by Screen.Intersect() are all set to NaN (not-a-number).

Conclusion

In this article I have not explained how you can go about implementing mouse clicks. For that you would require custom gestures, plus some win32 api calls. Despite this I hope you found the information in this article useful.

History

21^st Feb 2013: Initial version