Click here to Skip to main content
15,861,172 members
Articles / Programming Languages / C# 4.0

Leap Motion: Move Cursor

Rate me:
Please Sign up or sign in to vote.
4.91/5 (20 votes)
24 Feb 2013CPOL4 min read 114.9K   4.3K   43   51
How to use the Leap controller to move the cursor using a finger

Image 1 

Introduction

The Leap is a 3D motion sensing device that detects and tracks hands, fingers and finger-like tools. In this article I'll describe how you can go about using the Leap, and the Leap SDK, to create a .NET console application that enables you to move the cursor using your finger.

Requirements

To run the sample project you require the following,

  • Leap (developer unit),
  • Leap SDK,
  • VS2012

If you don't have a Leap controller you can apply to join the developer program and, hopefully, get a developer unit for free. Joining the developer program will also grant you access to the Leap SDK.

NB: The example project described here was developed using dev unit v.06.5 and version 0.7.3 of the Leap SDK.

Running the Project

To run the sample project you have to add Leap.dll and _LeapCSharp.dll to the project. (These two files are part of the Leap SDK). Ensure that the Copy to Output Directory property of the two files is set to Copy always.

Image 2 

Ensure that you also add a reference to LeapCSharp.NET4.0.dll, which is also part of the Leap SDK, and contains classes that enable you to access the data from the Leap controller.

Leap Coordinate System

Image 3 

Leap uses a right-handed Cartesian coordinate system, with the origin at the center of the controller and the X and Z axes lying in the horizontal plane. The Y axis is vertical.

Moving the Cursor

In order to move the cursor we'll need to call a function in the win32 API. This is done by a method in class MouseCursor.

VB.NET
Public Class MouseCursor

    Private Declare Function SetCursorPos Lib "user32" (x As Integer, y As Integer) As Boolean

    Public Shared Sub MoveCursor(x As Integer, y As Integer)
        SetCursorPos(x, y)
    End Sub

End Class
C#
using System;
using System.Runtime.InteropServices;
using System.Windows;
using System.Windows.Interop;

namespace ConsoleLeapMouse
{
    class MouseCursor
    {
        [DllImport("user32.dll")]
        private static extern bool SetCursorPos(int x, int y);

        public static void MoveCursor(int x, int y)
        {
            SetCursorPos(x, y);
        }

    }
}

MoveCursor() takes two parameters that define the location on the screen where the cursor should move. To get the values for these parameters we'll need to process data coming from the Leap controller.

To get data from the Leap we need to create a Controller object. The Controller object connects the console application with the Leap software and provides hand and finger tracking data through Frame objects.

VB.NET
Imports Leap

Module Module1

    Sub Main()
        Dim cntrl As New Controller
        Dim listener As New LeapListener

        cntrl.AddListener(listener)

        Console.WriteLine("Press Enter to quit...")
        Console.ReadLine()

        cntrl.RemoveListener(listener)
        cntrl.Dispose()
    End Sub

End Module
C#
using System;
using Leap;

namespace ConsoleLeapMouse
{
    class Program
    {
        static void Main(string[] args)
        {
            Controller cntrl = new Controller();
            LeapListener listener = new LeapListener();

            cntrl.AddListener(listener);

            Console.WriteLine("Press Enter to quit...");
            Console.ReadLine();

            cntrl.RemoveListener(listener);
            cntrl.Dispose();       
        }
    }
}

In the code snippet above an object of type LeapListener is added to the Controller object. LeapListener is a subclass of the Listener class which defines a set of callback methods that can be overridden to respond to events dispatched by the Leap.

VB.NET
Imports Leap

Public Class LeapListener
    Inherits Listener

    Public Overrides Sub OnInit(cntrlr As Controller)
        Console.WriteLine("Initialized")
    End Sub

    Public Overrides Sub OnConnect(cntrlr As Controller)
        Console.WriteLine("Connected")
    End Sub

    Public Overrides Sub OnDisconnect(cntrlr As Controller)
        Console.WriteLine("Disconnected")
    End Sub

    Public Overrides Sub OnExit(cntrlr As Controller)
        Console.WriteLine("Exited")
    End Sub

    Private currentTime As Long
    Private previousTime As Long
    Private timeChange As Long

    Public Overrides Sub OnFrame(cntrlr As Controller)
        ' Get the current frame.
        Dim currentFrame As Frame = cntrlr.Frame

        currentTime = currentFrame.Timestamp
        timeChange = currentTime - previousTime

        If (timeChange > 10000) Then
            If (Not currentFrame.Hands.Empty) Then
                ' Get the first finger in the list of fingers.
                Dim finger = currentFrame.Fingers(0)
                ' Gets the closest screen intercepting a ray projecting from the finger.
                Dim screen = cntrlr.CalibratedScreens.ClosestScreenHit(finger)

                If (screen IsNot Nothing And screen.IsValid) Then
                    ' Get the velocity of the finger tip.
                    Dim tipVelocity = finger.TipVelocity.Magnitude

                    ' Use tipVelocity to reduce jitters when attempting
                    ' to hold the cursor steady.
                    If (tipVelocity > 25) Then
                        Dim xScreenIntersect = screen.Intersect(finger, True).x
                        Dim yScreenIntersect = screen.Intersect(finger, True).y

                        If (xScreenIntersect.ToString <> "NaN") Then
                            Dim x = CInt(xScreenIntersect * screen.WidthPixels)
                            Dim y = screen.HeightPixels - CInt(yScreenIntersect * screen.HeightPixels)

                            Console.WriteLine("Screen intersect X: " & xScreenIntersect.ToString)
                            Console.WriteLine("Screen intersect Y: " & yScreenIntersect.ToString)
                            Console.WriteLine("Width pixels: " & screen.WidthPixels.ToString)
                            Console.WriteLine("Height pixels: " & screen.HeightPixels.ToString)

                            Console.WriteLine(vbCrLf)

                            Console.WriteLine("x: " & x.ToString)
                            Console.WriteLine("y: " & y.ToString)

                            Console.WriteLine(vbCrLf)

                            Console.WriteLine("Tip velocity: " & tipVelocity)

                            ' Move the cursor
                            MouseCursor.MoveCursor(x, y)

                            Console.WriteLine(vbCrLf & New String("=", 40) & vbCrLf)
                        End If
                    End If

                End If
            End If

            previousTime = currentTime
        End If

    End Sub

End Class
C#
using System;
using Leap;

namespace ConsoleLeapMouse
{
    class LeapListener: Listener
    {
        public override void OnInit(Controller cntrlr)
        {
            Console.WriteLine("Initialized");
        }

        public override void OnConnect(Controller cntrlr)
        {
            Console.WriteLine("Connected");
        }

        public override void OnDisconnect(Controller cntrlr)
        {
            Console.WriteLine("Disconnected");
        }

        public override void OnExit(Controller cntrlr)
        {
            Console.WriteLine("Exited");
        }

        private long currentTime;
        private long previousTime;
        private long timeChange;

        public override void OnFrame(Controller cntrlr)
        {
            // Get the current frame.
            Frame currentFrame = cntrlr.Frame();

            currentTime = currentFrame.Timestamp;
            timeChange = currentTime - previousTime;
           
            if (timeChange > 10000)
            {
                if (!currentFrame.Hands.Empty)
                {
                    // Get the first finger in the list of fingers
                    Finger finger = currentFrame.Fingers[0];
                    // Get the closest screen intercepting a ray projecting from the finger
                    Screen screen = cntrlr.CalibratedScreens.ClosestScreenHit(finger);

                    if (screen != null && screen.IsValid)
                    {
                        // Get the velocity of the finger tip
                        var tipVelocity = (int)finger.TipVelocity.Magnitude;

                        // Use tipVelocity to reduce jitters when attempting to hold
                        // the cursor steady
                        if (tipVelocity > 25)
                        {
                            var xScreenIntersect = screen.Intersect(finger, true).x;
                            var yScreenIntersect = screen.Intersect(finger, true).y;
                            
                            if (xScreenIntersect.ToString() != "NaN")
                            {
                                var x = (int)(xScreenIntersect * screen.WidthPixels);
                                var y = (int)(screen.HeightPixels - (yScreenIntersect * screen.HeightPixels));

                                Console.WriteLine("Screen intersect X: " + xScreenIntersect.ToString());
                                Console.WriteLine("Screen intersect Y: " + yScreenIntersect.ToString());
                                Console.WriteLine("Width pixels: " + screen.WidthPixels.ToString());
                                Console.WriteLine("Height pixels: " + screen.HeightPixels.ToString());

                                Console.WriteLine("\n");

                                Console.WriteLine("x: " + x.ToString());
                                Console.WriteLine("y: " + y.ToString());

                                Console.WriteLine("\n");

                                Console.WriteLine("Tip velocity: " + tipVelocity.ToString());

                                // Move the cursor
                                MouseCursor.MoveCursor(x, y);

                                Console.WriteLine("\n" + new String('=', 40) + "\n");
                            }

                        }
                    }

                }

                previousTime = currentTime;
            }
        }
    }
}

The most significant method in the code snippet above is the OnFrame() method. The OnFrame() method is called by the Controller object when a new frame of hand and finger tracking data is available. The data is accessed by calling the Controller object's Frame() method. Since the Leap has a really high frame rate, I guess a maximum of about 115fps, attempting to move the cursor at every call of the OnFrame() method will not give the desired results. For this reason I have specified an interval of  >10ms between frames for processing of tracking data.

NB: Frame.Timestamp() is the frame capture time in microseconds elapsed since the Leap started.

To get the screen that a finger is pointing to we use the Controller.CalibratedScreens.ClosestScreenHit() method and pass it a Finger object, which in this case is the first Finger object in a list of Finger objects. The specified screen will be the closest screen intercepting a ray projecting from the Finger object. The projected ray emanates from the Finger TipPosition along its Direction vector.

Image 4 

Finger TipPosition and Direction vectors

To get the screen position in pixels, where the finger is pointing at, we multiply the values returned by Screen.WidthPixels() and Screen.HeightPixels() with the normalized coordinates returned by Screen.Intersect(). Normalized coordinates represent the intersection point between the screen and the ray projecting from the Finger object, defined as a percentage of the screen's width and height. Such coordinates would be of the form (0.1, 0.1, 0).

VB.NET
...
Dim xScreenIntersect = screen.Intersect(finger, True).x
Dim yScreenIntersect = screen.Intersect(finger, True).y

If (xScreenIntersect.ToString <> "NaN") Then
    Dim x = CInt(xScreenIntersect * screen.WidthPixels)
    Dim y = screen.HeightPixels - CInt(yScreenIntersect * screen.HeightPixels)
    ...          
End If
C#
...
var xScreenIntersect = screen.Intersect(finger, true).x;
var yScreenIntersect = screen.Intersect(finger, true).y;
                            
if (xScreenIntersect.ToString() != "NaN")
{
    var x = (int)(xScreenIntersect * screen.WidthPixels);
    var y = (int)(screen.HeightPixels - (yScreenIntersect * screen.HeightPixels));
    ...
}

In case you are wondering why I'm setting the value of the y variable by doing screen.HeightPixels - (yScreenIntersect * screen.HeightPixels), it's because my computer's graphics coordinate system places the screen's origin at the bottom-left corner.

NB: If the Finger object is pointing parallel to or away from the screen the components of the vector returned by Screen.Intersect() are all set to NaN (not-a-number).

Conclusion

In this article I have not explained how you can go about implementing mouse clicks. For that you would require custom gestures, plus some win32 api calls. Despite this I hope you found the information in this article useful.

History

  • 21st Feb 2013: Initial version

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
Software Developer
Kenya Kenya
Experienced C# software developer with a passion for WPF.

Awards,
  • CodeProject MVP 2013
  • CodeProject MVP 2012
  • CodeProject MVP 2021

Comments and Discussions

 
QuestionPlease help me to fix this 2 error... Pin
willjone9221-Aug-17 13:08
willjone9221-Aug-17 13:08 
AnswerRe: Please help me to fix this 2 error... Pin
Ye Bai30-Mar-22 5:55
Ye Bai30-Mar-22 5:55 
QuestionPlease help me to fix this 2 error... Pin
willjone9221-Aug-17 13:05
willjone9221-Aug-17 13:05 
QuestionGot CalibrationScreen Problem Pin
Member 1227382322-Feb-16 7:26
Member 1227382322-Feb-16 7:26 
GeneralMy vote of 5 Pin
D V L15-Nov-15 19:37
professionalD V L15-Nov-15 19:37 
QuestionHi I need help How to use vb.net Pin
royji12329-Dec-14 1:11
royji12329-Dec-14 1:11 
Question2 errors i get when building the example c# project Pin
Member 107944524-May-14 17:42
Member 107944524-May-14 17:42 
AnswerRe: 2 errors i get when building the example c# project Pin
Meshack Musundi4-May-14 21:27
professionalMeshack Musundi4-May-14 21:27 
AnswerRe: 2 errors i get when building the example c# project Pin
sukasenyummm4-Aug-14 18:51
sukasenyummm4-Aug-14 18:51 
GeneralRe: 2 errors i get when building the example c# project Pin
Member 1145571928-Mar-15 21:05
Member 1145571928-Mar-15 21:05 
NewsUpdated Project and DLL's - with specially created .net 4.5.1 leap dll's Pin
Shadowcouncil2-Mar-14 14:52
Shadowcouncil2-Mar-14 14:52 
GeneralRe: Updated Project and DLL's - with specially created .net 4.5.1 leap dll's Pin
Meshack Musundi2-Mar-14 19:09
professionalMeshack Musundi2-Mar-14 19:09 
GeneralRe: Updated Project and DLL's - with specially created .net 4.5.1 leap dll's Pin
Shadowcouncil3-Mar-14 6:17
Shadowcouncil3-Mar-14 6:17 
GeneralRe: Updated Project and DLL's - with specially created .net 4.5.1 leap dll's Pin
Shadowcouncil4-Mar-14 18:31
Shadowcouncil4-Mar-14 18:31 
Questionplot circle points on windows screen Pin
rsabishek18-Sep-13 0:47
rsabishek18-Sep-13 0:47 
GeneralRe: plot circle points on windows screen Pin
Meshack Musundi18-Sep-13 9:15
professionalMeshack Musundi18-Sep-13 9:15 
GeneralMy vote of 5 Pin
Weston Miller14-Sep-13 11:19
Weston Miller14-Sep-13 11:19 
GeneralRe: My vote of 5 Pin
Meshack Musundi15-Sep-13 5:45
professionalMeshack Musundi15-Sep-13 5:45 
QuestionVery elegant and well reasoned. Saved me a day. Pin
Shadowcouncil11-Aug-13 9:25
Shadowcouncil11-Aug-13 9:25 
AnswerRe: Very elegant and well reasoned. Saved me a day. Pin
Meshack Musundi11-Aug-13 22:52
professionalMeshack Musundi11-Aug-13 22:52 
GeneralRe: Very elegant and well reasoned. Saved me a day. Pin
Shadowcouncil2-Mar-14 16:55
Shadowcouncil2-Mar-14 16:55 
QuestionZ axis Pin
CdRsKuLL8-Aug-13 1:22
CdRsKuLL8-Aug-13 1:22 
AnswerRe: Z axis Pin
Meshack Musundi8-Aug-13 2:00
professionalMeshack Musundi8-Aug-13 2:00 
GeneralRe: Z axis Pin
CdRsKuLL8-Aug-13 4:35
CdRsKuLL8-Aug-13 4:35 
QuestionFocus Pin
Member 810746525-Jul-13 13:08
Member 810746525-Jul-13 13:08 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.