Click here to Skip to main content
Click here to Skip to main content

Leap Motion: Move Cursor

, 24 Feb 2013 CPOL
Rate this:
Please Sign up or sign in to vote.
How to use the Leap controller to move the cursor using a finger

 

Introduction

The Leap is a 3D motion sensing device that detects and tracks hands, fingers and finger-like tools. In this article I'll describe how you can go about using the Leap, and the Leap SDK, to create a .NET console application that enables you to move the cursor using your finger.

Requirements

To run the sample project you require the following,

  • Leap (developer unit),
  • Leap SDK,
  • VS2012

If you don't have a Leap controller you can apply to join the developer program and, hopefully, get a developer unit for free. Joining the developer program will also grant you access to the Leap SDK.

NB: The example project described here was developed using dev unit v.06.5 and version 0.7.3 of the Leap SDK.

Running the Project

To run the sample project you have to add Leap.dll and _LeapCSharp.dll to the project. (These two files are part of the Leap SDK). Ensure that the Copy to Output Directory property of the two files is set to Copy always.

 

Ensure that you also add a reference to LeapCSharp.NET4.0.dll, which is also part of the Leap SDK, and contains classes that enable you to access the data from the Leap controller.

Leap Coordinate System

 

Leap uses a right-handed Cartesian coordinate system, with the origin at the center of the controller and the X and Z axes lying in the horizontal plane. The Y axis is vertical.

Moving the Cursor

In order to move the cursor we'll need to call a function in the win32 API. This is done by a method in class MouseCursor.

Public Class MouseCursor

    Private Declare Function SetCursorPos Lib "user32" (x As Integer, y As Integer) As Boolean

    Public Shared Sub MoveCursor(x As Integer, y As Integer)
        SetCursorPos(x, y)
    End Sub

End Class
using System;
using System.Runtime.InteropServices;
using System.Windows;
using System.Windows.Interop;

namespace ConsoleLeapMouse
{
    class MouseCursor
    {
        [DllImport("user32.dll")]
        private static extern bool SetCursorPos(int x, int y);

        public static void MoveCursor(int x, int y)
        {
            SetCursorPos(x, y);
        }

    }
}

MoveCursor() takes two parameters that define the location on the screen where the cursor should move. To get the values for these parameters we'll need to process data coming from the Leap controller.

To get data from the Leap we need to create a Controller object. The Controller object connects the console application with the Leap software and provides hand and finger tracking data through Frame objects.

Imports Leap

Module Module1

    Sub Main()
        Dim cntrl As New Controller
        Dim listener As New LeapListener

        cntrl.AddListener(listener)

        Console.WriteLine("Press Enter to quit...")
        Console.ReadLine()

        cntrl.RemoveListener(listener)
        cntrl.Dispose()
    End Sub

End Module
using System;
using Leap;

namespace ConsoleLeapMouse
{
    class Program
    {
        static void Main(string[] args)
        {
            Controller cntrl = new Controller();
            LeapListener listener = new LeapListener();

            cntrl.AddListener(listener);

            Console.WriteLine("Press Enter to quit...");
            Console.ReadLine();

            cntrl.RemoveListener(listener);
            cntrl.Dispose();       
        }
    }
}

In the code snippet above an object of type LeapListener is added to the Controller object. LeapListener is a subclass of the Listener class which defines a set of callback methods that can be overridden to respond to events dispatched by the Leap.

Imports Leap

Public Class LeapListener
    Inherits Listener

    Public Overrides Sub OnInit(cntrlr As Controller)
        Console.WriteLine("Initialized")
    End Sub

    Public Overrides Sub OnConnect(cntrlr As Controller)
        Console.WriteLine("Connected")
    End Sub

    Public Overrides Sub OnDisconnect(cntrlr As Controller)
        Console.WriteLine("Disconnected")
    End Sub

    Public Overrides Sub OnExit(cntrlr As Controller)
        Console.WriteLine("Exited")
    End Sub

    Private currentTime As Long
    Private previousTime As Long
    Private timeChange As Long

    Public Overrides Sub OnFrame(cntrlr As Controller)
        ' Get the current frame.
        Dim currentFrame As Frame = cntrlr.Frame

        currentTime = currentFrame.Timestamp
        timeChange = currentTime - previousTime

        If (timeChange > 10000) Then
            If (Not currentFrame.Hands.Empty) Then
                ' Get the first finger in the list of fingers.
                Dim finger = currentFrame.Fingers(0)
                ' Gets the closest screen intercepting a ray projecting from the finger.
                Dim screen = cntrlr.CalibratedScreens.ClosestScreenHit(finger)

                If (screen IsNot Nothing And screen.IsValid) Then
                    ' Get the velocity of the finger tip.
                    Dim tipVelocity = finger.TipVelocity.Magnitude

                    ' Use tipVelocity to reduce jitters when attempting
                    ' to hold the cursor steady.
                    If (tipVelocity > 25) Then
                        Dim xScreenIntersect = screen.Intersect(finger, True).x
                        Dim yScreenIntersect = screen.Intersect(finger, True).y

                        If (xScreenIntersect.ToString <> "NaN") Then
                            Dim x = CInt(xScreenIntersect * screen.WidthPixels)
                            Dim y = screen.HeightPixels - CInt(yScreenIntersect * screen.HeightPixels)

                            Console.WriteLine("Screen intersect X: " & xScreenIntersect.ToString)
                            Console.WriteLine("Screen intersect Y: " & yScreenIntersect.ToString)
                            Console.WriteLine("Width pixels: " & screen.WidthPixels.ToString)
                            Console.WriteLine("Height pixels: " & screen.HeightPixels.ToString)

                            Console.WriteLine(vbCrLf)

                            Console.WriteLine("x: " & x.ToString)
                            Console.WriteLine("y: " & y.ToString)

                            Console.WriteLine(vbCrLf)

                            Console.WriteLine("Tip velocity: " & tipVelocity)

                            ' Move the cursor
                            MouseCursor.MoveCursor(x, y)

                            Console.WriteLine(vbCrLf & New String("=", 40) & vbCrLf)
                        End If
                    End If

                End If
            End If

            previousTime = currentTime
        End If

    End Sub

End Class
using System;
using Leap;

namespace ConsoleLeapMouse
{
    class LeapListener: Listener
    {
        public override void OnInit(Controller cntrlr)
        {
            Console.WriteLine("Initialized");
        }

        public override void OnConnect(Controller cntrlr)
        {
            Console.WriteLine("Connected");
        }

        public override void OnDisconnect(Controller cntrlr)
        {
            Console.WriteLine("Disconnected");
        }

        public override void OnExit(Controller cntrlr)
        {
            Console.WriteLine("Exited");
        }

        private long currentTime;
        private long previousTime;
        private long timeChange;

        public override void OnFrame(Controller cntrlr)
        {
            // Get the current frame.
            Frame currentFrame = cntrlr.Frame();

            currentTime = currentFrame.Timestamp;
            timeChange = currentTime - previousTime;
           
            if (timeChange > 10000)
            {
                if (!currentFrame.Hands.Empty)
                {
                    // Get the first finger in the list of fingers
                    Finger finger = currentFrame.Fingers[0];
                    // Get the closest screen intercepting a ray projecting from the finger
                    Screen screen = cntrlr.CalibratedScreens.ClosestScreenHit(finger);

                    if (screen != null && screen.IsValid)
                    {
                        // Get the velocity of the finger tip
                        var tipVelocity = (int)finger.TipVelocity.Magnitude;

                        // Use tipVelocity to reduce jitters when attempting to hold
                        // the cursor steady
                        if (tipVelocity > 25)
                        {
                            var xScreenIntersect = screen.Intersect(finger, true).x;
                            var yScreenIntersect = screen.Intersect(finger, true).y;
                            
                            if (xScreenIntersect.ToString() != "NaN")
                            {
                                var x = (int)(xScreenIntersect * screen.WidthPixels);
                                var y = (int)(screen.HeightPixels - (yScreenIntersect * screen.HeightPixels));

                                Console.WriteLine("Screen intersect X: " + xScreenIntersect.ToString());
                                Console.WriteLine("Screen intersect Y: " + yScreenIntersect.ToString());
                                Console.WriteLine("Width pixels: " + screen.WidthPixels.ToString());
                                Console.WriteLine("Height pixels: " + screen.HeightPixels.ToString());

                                Console.WriteLine("\n");

                                Console.WriteLine("x: " + x.ToString());
                                Console.WriteLine("y: " + y.ToString());

                                Console.WriteLine("\n");

                                Console.WriteLine("Tip velocity: " + tipVelocity.ToString());

                                // Move the cursor
                                MouseCursor.MoveCursor(x, y);

                                Console.WriteLine("\n" + new String('=', 40) + "\n");
                            }

                        }
                    }

                }

                previousTime = currentTime;
            }
        }
    }
}

The most significant method in the code snippet above is the OnFrame() method. The OnFrame() method is called by the Controller object when a new frame of hand and finger tracking data is available. The data is accessed by calling the Controller object's Frame() method. Since the Leap has a really high frame rate, I guess a maximum of about 115fps, attempting to move the cursor at every call of the OnFrame() method will not give the desired results. For this reason I have specified an interval of  >10ms between frames for processing of tracking data.

NB: Frame.Timestamp() is the frame capture time in microseconds elapsed since the Leap started.

To get the screen that a finger is pointing to we use the Controller.CalibratedScreens.ClosestScreenHit() method and pass it a Finger object, which in this case is the first Finger object in a list of Finger objects. The specified screen will be the closest screen intercepting a ray projecting from the Finger object. The projected ray emanates from the Finger TipPosition along its Direction vector.

 

Finger TipPosition and Direction vectors

To get the screen position in pixels, where the finger is pointing at, we multiply the values returned by Screen.WidthPixels() and Screen.HeightPixels() with the normalized coordinates returned by Screen.Intersect(). Normalized coordinates represent the intersection point between the screen and the ray projecting from the Finger object, defined as a percentage of the screen's width and height. Such coordinates would be of the form (0.1, 0.1, 0).

...
Dim xScreenIntersect = screen.Intersect(finger, True).x
Dim yScreenIntersect = screen.Intersect(finger, True).y

If (xScreenIntersect.ToString <> "NaN") Then
    Dim x = CInt(xScreenIntersect * screen.WidthPixels)
    Dim y = screen.HeightPixels - CInt(yScreenIntersect * screen.HeightPixels)
    ...          
End If
...
var xScreenIntersect = screen.Intersect(finger, true).x;
var yScreenIntersect = screen.Intersect(finger, true).y;
                            
if (xScreenIntersect.ToString() != "NaN")
{
    var x = (int)(xScreenIntersect * screen.WidthPixels);
    var y = (int)(screen.HeightPixels - (yScreenIntersect * screen.HeightPixels));
    ...
}

In case you are wondering why I'm setting the value of the y variable by doing screen.HeightPixels - (yScreenIntersect * screen.HeightPixels), it's because my computer's graphics coordinate system places the screen's origin at the bottom-left corner.

NB: If the Finger object is pointing parallel to or away from the screen the components of the vector returned by Screen.Intersect() are all set to NaN (not-a-number).

Conclusion

In this article I have not explained how you can go about implementing mouse clicks. For that you would require custom gestures, plus some win32 api calls. Despite this I hope you found the information in this article useful.

History

  • 21st Feb 2013: Initial version

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Meshack Musundi
Software Developer
Kenya Kenya
Meshack is an avid programmer with a bias towards WPF and VB.NET. He has about 5 years of programming experience initially starting off with Java before shifting to .NET, thanks to the allure of WPF. He has developed several applications, and written several articles about them, which can be viewed here on CodeProject. He currently resides in a small town in Kiambu county, Kenya.
 
Awards;
  • CodeProject MVP 2013
  • CodeProject MVP 2012
  • Best VB.NET article of August 2013
  • Best VB.NET article of February 2013
  • Best VB.NET article of October 2012
  • Best VB.NET article of July 2012
  • Best VB.NET article of February 2012
  • Best VB.NET article of January 2012
  • Best VB.NET article of November 2011
  • Best VB.NET article of June 2011
  • Best VB.NET article of May 2011
  • Best VB.NET article of March 2011
  • Best VB.NET article of February 2011
  • Best VB.NET article of January 2011
  • Best VB.NET article of December 2010
  • Best VB.NET article of November 2010

Comments and Discussions

 
QuestionSDK [modified] PinmemberEGHSanc26-Feb-13 12:12 
AnswerRe: SDK PinmvpMeshack Musundi26-Feb-13 19:10 
GeneralRe: SDK PinmemberEGHSanc27-Feb-13 1:04 
QuestionHardware PinmemberBMoores22-Feb-13 5:47 
AnswerRe: Hardware PinmvpMeshack Musundi22-Feb-13 7:14 
GeneralRe: Hardware PinmemberBMoores22-Feb-13 9:37 
GeneralRe: Hardware Pinmemberkaste8825-Feb-13 3:58 
QuestionI got one of these Leap devices too, but [modified] PinmvpSacha Barber21-Feb-13 8:46 
Am hoping to some way radical stuff with it(such as lightning from the fingers destroying stuff) shame its not full skeleton tracking though.
Sacha Barber
  • Microsoft Visual C# MVP 2008-2012
  • Codeproject MVP 2008-2012
Open Source Projects
Cinch SL/WPF MVVM

Your best friend is you.
I'm my best friend too. We share the same views, and hardly ever argue
 
My Blog : sachabarber.net


modified 21-Feb-13 14:05pm.

GeneralRe: I got one of these Leap devices too, but am hoping to some way radical stuff with it PinmvpMeshack Musundi21-Feb-13 9:04 
GeneralRe: I got one of these Leap devices too, but PinmvpMeshack Musundi21-Feb-13 10:43 
AnswerRe: I got one of these Leap devices too, but PinadminChris Maunder24-Mar-13 19:05 
GeneralMy vote of 5 PinmemberWooters21-Feb-13 7:06 
GeneralRe: My vote of 5 PinmvpMeshack Musundi21-Feb-13 9:02 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web01 | 2.8.1411028.1 | Last Updated 25 Feb 2013
Article Copyright 2013 by Meshack Musundi
Everything else Copyright © CodeProject, 1999-2014
Layout: fixed | fluid