Click here to Skip to main content
Click here to Skip to main content

Tagged as

Go to top

Kinect – Calculator – Adjust Skeleton Movements To Mouse

, 22 Jul 2011
Rate this:
Please Sign up or sign in to vote.
CodeProjectIn my previous post Kinect – Create Buttons I’ve showed one approach how to create Kinect Buttons for Windows, over the next posts I’ll show more ways to accomplish that by moving windows Cursor based on Kinect Skeleton Right Hand Position.Why To Create Kinect Button?Why not using windows

In my previous post Kinect – Create Buttons I’ve showed one approach how to create Kinect Buttons for Windows, over the next posts I’ll show more ways to accomplish that by moving windows Cursor based on Kinect Skeleton Right Hand Position.

Why To Create Kinect Button?

Why not using windows cursor and create brilliant hand or head (Smile) movement to simulate Click, so we don’t need to create designated Kinect Buttons.

Answers:

  1. You had to stand at least 1 meter from the computer screen and even so I’m young I can’t see very well from that distance…
  2. Kinect precision is still not perfect and it’s hard to hit a 48x48 button from that distance.
  3. Even if you increase desktop resolution it’s not easy and not practical to work with both hands to perform Click.
  4. But, I’m going to do that anyway! Magnifying glass, Cool Head or Hand movements etc…

In this post I’ll take those Kinect Button(working on Timer) implement on a simple Calculator I’ve built and hook the Windows mouse to the Skeleton movement.

The only major issue with that is to adjust the Skeleton Position to your Screen Resolution, as you know the Skeleton coordinates are expressed in meters and comes from 640x480 screen, so I’ve create new class called Positions that will help me to do it.

The AdjustToScreen method gets the Joint (and based on the current screen resolution) and another gets the Joint and specific screen width and height.

public static class Positions
{

    //For - Resolution640x480
    private const float SkeletonMaxX = 0.6f;

    private const float SkeletonMaxY = 0.4f;
 

    private static float Adjust(int primaryScreenResolution, 
        float maxJointPosition, float jointPosition)
    {

        var value = (((((float)primaryScreenResolution) / maxJointPosition) / 2f) * 
            jointPosition) 
            + (primaryScreenResolution / 2);

            
        if (value > primaryScreenResolution || value < 0f) return 0f;

 
        return value;

    }
 

    /// <span class="code-SummaryComment"><summary>
</span>    /// Get the current Joint position and Adjust the Skeleton 
    /// joint position to the current Screen resolution.

    /// <span class="code-SummaryComment"></summary>
</span>    /// <span class="code-SummaryComment"><param name="joint">Joint to Adjust</param>
</span>
    /// <span class="code-SummaryComment"><returns></returns>
</span>    public static Vector AdjustToScreen(Joint joint)

    {
        var newVector = new Vector

        {
            X = Adjust((int)SystemParameters.PrimaryScreenWidth, 
            SkeletonMaxX, joint.Position.X),

            Y = Adjust((int)SystemParameters.PrimaryScreenHeight, 
            SkeletonMaxY, -joint.Position.Y),
            Z = joint.Position.Z,

            W = joint.Position.W
        };

 
        return newVector;

    }
    /// <span class="code-SummaryComment"><summary>
</span>
    /// Get the current Joint position and Adjust the Skeleton joint
    /// position to a specific Screen Size.
    /// <span class="code-SummaryComment"></summary>
</span>
    /// <span class="code-SummaryComment"><param name="joint">Joint to Adjust</param>
</span>    /// <span class="code-SummaryComment"><param name="screenWidth">Screen Width</param>
</span>
    /// <span class="code-SummaryComment"><param name="screenHeight">Screen Height</param>
</span>    /// <span class="code-SummaryComment"><returns></returns>
</span>
    public static Vector AdjustToScreen(Joint joint, int screenWidth, int screenHeight)
    {

        var newVector = new Vector
        {

            X = Adjust(screenWidth, SkeletonMaxX, joint.Position.X),
            Y = Adjust(screenHeight, SkeletonMaxY, -joint.Position.Y),

            Z = joint.Position.Z,
            W = joint.Position.W

        };
 

        return newVector;
    }

}

Now I’ve create another helper class called – NativeMethods to move the set the Cursor  Position based on Skeleton Joint position.

public static class NativeMethods
{

    public partial class MouseOperations
    {

        [DllImport("user32.dll")]
        public static extern bool SetCursorPos(int x, int y);

 
        [DllImport("user32.dll")]

        public static extern bool GetCursorPos(out Point pt);
    }

}

In the SkeletonFrameReady I’ve added another quality check, also check the Joint TrakingState and not just the Data TrakingState, also make sure the joint position quality (W) is high enough before moving the mouse.

The quality check will prevent that mouse from jumping around.

void SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{

    foreach (SkeletonData data in e.SkeletonFrame.Skeletons)
    {

        //Tracked that defines whether a skeleton is 'tracked' or not. 
        //The untracked skeletons only give their position. 
        if (data.TrackingState != SkeletonTrackingState.Tracked) continue;

 
        //Each joint has a Position property that is defined by a Vector4: (x, y, z, w). 

        //The first three attributes define the position in camera space.
        //The last attribute (w)
        //gives the quality level (between 0 and 1) of the 

        foreach (Joint joint in data.Joints)
        {

            switch (joint.ID)
            {

                case JointID.HandRight:
                    if (joint.Position.W < 0.6f || joint.TrackingState != 
                        JointTrackingState.Tracked) return;// Quality check 

 
                    //Based on Skeleton Right Hand Position adjust the location
                    //to the screen

                    var newPos = Positions.AdjustToScreen(joint);
 

                    if (newPos.X == 0f || newPos.Y == 0f) return;
 

                    NativeMethods.MouseOperations.SetCursorPos(
                        Convert.ToInt32(newPos.X), Convert.ToInt32(newPos.Y));
 

                    break;
            }

        }
    }

}

Make sure you enable TransformSmooth, you don’t want the mouse jumping around. (Kinect–How to Apply Smooths Frame Display Using TransformSmoothParameters)

_kinectNui.SkeletonEngine.TransformSmooth = true;
var parameters = new TransformSmoothParameters

{
    Smoothing = 1.0f,

    Correction = 0.1f,
    Prediction = 0.1f,

    JitterRadius = 0.05f,
    MaxDeviationRadius = 0.05f

};
_kinectNui.SkeletonEngine.SmoothParameters = parameters;

Live Demo

Demo Project

This project includes a Common class with this post main methods, from Moving and mouse and adjusting Kinect Skeleton to desktop resolution.

Enjoy

License

This article, along with any associated source code and files, is licensed under The Microsoft Public License (Ms-PL)

Share

About the Author

Shai Raiten
Architect Sela
Israel Israel
Shai Raiten is VS ALM MVP, currently working for Sela Group as a ALM senior consultant and trainer specializes in Microsoft technologies especially Team System and .NET technology. He is currently consulting in various enterprises in Israel, planning and analysis Load and performance problems using Team System, building Team System customizations and adjusts ALM processes for enterprises. Shai is known as one of the top Team System experts in Israel. He conducts lectures and workshops for developers\QA and enterprises who want to specialize in Team System.
 
My Blog: http://blogs.microsoft.co.il/blogs/shair/
Follow on   Twitter

Comments and Discussions

 
-- There are no messages in this forum --
| Advertise | Privacy | Mobile
Web03 | 2.8.140916.1 | Last Updated 22 Jul 2011
Article Copyright 2011 by Shai Raiten
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid