Click here to Skip to main content
Click here to Skip to main content

Kinect for Windows version 2: Body tracking

, 11 Apr 2014
Rate this:
Please Sign up or sign in to vote.
NOTE: This is preliminary software and/or hardware and APIs are preliminary and subject to change. In my previous blog post, I show you how to display the color, depth and infrared streams of Kinect version 2 by transforming the raw binary data into Windows bitmaps. This time, we’ll dive into

NOTE: This is preliminary software and/or hardware and APIs are preliminary and subject to change.

In my previous blog post, I show you how to display the color, depth and infrared streams of Kinect version 2 by transforming the raw binary data into Windows bitmaps.

This time, we’ll dive into the most essential part of Kinect: Body tracking.

The initial version of Kinect allowed us to track up to 21 body joints. The second version allows up to 25 joints. The new joints include the fists and thumbs! Moreover, due to the enhanced depth sensor, the tracking accuracy has been significantly improved. Experienced users will notice less jittering and much better stability. Once again, I would like to remind you of my video, which demonstrates the new body tracking capabilities:

Watch the video on YouTube

Next, we are going to implement body tracking and display all of the new joints on-screen. We’ll extend the project we created previously. You can download the source code here.

Extending the project

In the previous blog post, we created a project with an <Image> element for displaying the streams. We now need to add a <Canvas> control for drawing the body. Here is the updated XAML code:

<Grid>
    <Image Name="camera" />
    <Canvas Name="canvas" />
</Grid>

We also added a reference to Microsoft.Kinect namespace and initialized the sensor:

// Kinect namespace
using Microsoft.Kinect;

// ...

// Kinect sensor and Kinect stream reader objects
KinectSensor _sensor;
MultiSourceFrameReader _reader;
IList<Body> _bodies;

// Kinect sensor initialization
_sensor = KinectSensor.Default;

if (_sensor != null)
{
    _sensor.Open();
}

We also added a list of bodies, where all of the body/skeleton related data will be saved. If you have developed for Kinect version 1, you notice that the Skeleton class has been replaced by the Body class.

Remember the MultiSourceFrameReader? This class gives us access on every stream, including the body stream! We simply need to let the sensor know that we need body tracking functionality by adding an additional parameter when initializing the reader:

_reader = _sensor.OpenMultiSourceFrameReader(FrameSourceTypes.Color |
                                             FrameSourceTypes.Depth |
                                             FrameSourceTypes.Infrared |
                                             FrameSourceTypes.Body);
_reader.MultiSourceFrameArrived += Reader_MultiSourceFrameArrived;

The Reader_MultiSourceFrameArrived method will be called whenever a new frame is available. Let’s specify what will happen in terms of the body data:

  1. Get a reference to the body frame
  2. Check whether the body frame is null – this is crucial
  3. Initialize the _bodies list
  4. Call the GetAndRefreshBodyData method, so as to copy the body data into the list
  5. Loop through the list of bodies and do awesome stuff!

Always remember to check for null values. Kinect provides you with approximately 30 frames per second – anything could be null or missing! Here is the code so far:

void Reader_MultiSourceFrameArrived(object sender, MultiSourceFrameArrivedEventArgs e)
{
    var reference = e.FrameReference.AcquireFrame();

    // Color
    // ...

    // Depth
    // ...

    // Infrared
    // ...

    // Body
    using (var frame = reference.BodyFrameReference.AcquireFrame())
    {
        if (frame != null)
        {
            _bodies = new Body[frame.BodyFrameSource.BodyCount];

            frame.GetAndRefreshBodyData(_bodies);

            foreach (var body in _bodies)
            {
                if (body != null)
                {
                    // Do something with the body...
                }
            }
        }
    }
}

This is it! We now have access to the bodies Kinect identifies. Next step is to display the skeleton information on-screen. Each body consists of 25 joints. The sensor provides us with the position (X, Y, Z) and the rotation information for each one of them. Moreover, Kinect lets us know whether the joints are tracked, hypothsized or not tracked. It’s a good practice to check whether a body is tracked before performing any critical functions. The following code illustrates how we can access the different body joints:

if (body != null)
{
    if (body.IsTracked)
    {
        Joint head = body.Joints[JointType.Head];
        
        float x = head.Position.X;
        float y = head.Position.Y;
        float z = head.Position.Z;

        // Draw the joints...
    }
}

The supported joints by Kinect 2 are the following:

  • SpineBase
  • SpineMid
  • Neck
  • Head
  • ShoulderLeft
  • ElbowLeft
  • WristLeft
  • HandLeft
  • ShoulderRight
  • ElbowRight
  • WristRight
  • HandRight
  • HipLeft
  • KneeLeft
  • AnkleLeft
  • FootLeft
  • HipRight
  • KneeRight
  • AnkleRight
  • FootRight
  • SpineShoulder
  • HandTipLeft
  • ThumbLeft
  • HandTipRight
  • ThumbRight

Neck and thumbs are new joints added in the second version of Kinect.

Knowing the coordinates of every joint, we can now draw some objects using XAML and C#. However, Kinect provides a distance in millimetres, so we need to map the millimetres to screen pixels. In the attached project, I have made this mapping for you. So, the only method you need to call is the DrawPoint or the DrawLine. Here is the DrawPoint:

public static void DrawPoint(this Canvas canvas, Joint joint)
{
    // 1) Check whether the joint is tracked.
    if (joint.TrackingState == TrackingState.NotTracked) return;

    // 2) Map the real-world coordinates to screen pixels.
    joint = joint.ScaleTo(canvas.ActualWidth, canvas.ActualHeight);

    // 3) Create a WPF ellipse.
    Ellipse ellipse = new Ellipse
    {
        Width = 20,
        Height = 20,
        Fill = new SolidColorBrush(Colors.LightBlue)
    };

    // 4) Position the ellipse according to the joint's coordinates.
    Canvas.SetLeft(ellipse, joint.Position.X - ellipse.Width / 2);
    Canvas.SetTop(ellipse, joint.Position.Y - ellipse.Height / 2);

    // 5) Add the ellipse to the canvas.
    canvas.Children.Add(ellipse);
}

Similarly, you can draw lines using the Line object. Download the sample project and check by yourself.

Here is the end result you saw in the video:

Kinect 2 body stream

Notice that the body joints are not perfectly aligned to the background image. Why? Because the color, infrared and depth sensors are not one above the other, so they have a slightly different point of view. You can use the coordinate mapper of the SDK and align them if necessary.

Download the source code

Body tracking was similarly done in the previous sensor. In the next blog post, we are going to see something totally new: facial expressions and hand states.

PS: New Kinect book - 20% off

This blog post is part of a new book I am publishing a new ebook in a few days. The book is an in-depth developer guide about Kinect, using simple language and step-by-step examples. You'll learn usability tips, performance tricks and best practices for implementing robust Kinect apps. Please meet Kinect Essentials, the essence of my 3 years of teaching, writing and developing for the Kinect platform. Oh, did I mention that you'll get a 20% discount if you simply subscribe now? Hurry up ;-)

Subscribe here for 20% off

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

About the Author

Vangos Pterneas
Product Manager LightBuzz
United Kingdom United Kingdom
I'm a Software Engineer and Entrepreneur, passionate about motion technology and the way it can affect people’s lives.
 
I have been a Kinect enthusiast since the release of the very first unofficial hacks and have already published some innovative commercial Kinect applications. These applications include complex home automation systems, 3D body scanning programs and motion-enabled product browsers for businesses.
 
I worked as a Windows developer and consultant for Microsoft Innovation Center and I'm now running my own company, LightBuzz Software. LightBuzz has been awarded the first place in Microsoft’s worldwide innovation competition, held in New York, for effectively combining Kinect and smartphone functionality.
 
When I am not coding, I love writing books, speaking and blogging about my favorite technological aspects.
Follow on   Twitter   Google+   LinkedIn

Comments and Discussions

 
GeneralMy vote of 5 PinpremiumVolynsky Alex11-Apr-14 9:07 
GeneralRe: My vote of 5 PinpremiumVangos Pterneas11-Apr-14 9:09 
GeneralRe: My vote of 5 PinpremiumVolynsky Alex11-Apr-14 9:10 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web02 | 2.8.140721.1 | Last Updated 11 Apr 2014
Article Copyright 2014 by Vangos Pterneas
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid