Click here to Skip to main content
Click here to Skip to main content

Tagged as

Kinect for Windows version 2: Color, depth and infrared streams

, 21 Feb 2014
Rate this:
Please Sign up or sign in to vote.
NOTE: This is preliminary software and/or hardware and APIs are preliminary and subject to change. A month ago, I was happy to receive a brand-new Kinect for Windows version 2 Developer Preview sensor. You can read my previous blog post about the capabilities of the new device. Kinect v2 now include

NOTE: This is preliminary software and/or hardware and APIs are preliminary and subject to change.

A month ago, I was happy to receive a brand-new Kinect for Windows version 2 Developer Preview sensor. You can read my previous blog post about the capabilities of the new device. Kinect v2 now includes 4 types of input streams:

  • Color
  • Depth
  • Infrared
  • Body
  • Audio

Today I will show you how you can acquire and display each bitmap input in a Windows application. In the next blog post, we’ll talk about body tracking. Here is a quick video I made which demonstrates the different streams provided by the new sensor.

<iframe width="100%" height="400" src="http://www.codeproject.com/www.youtube.com/embed/86ALowFuBkY" frameborder="0" allowfullscreen></iframe>

You can download the source code here. Read on if you to implement this functionality yourself.

Requirements

Creating a new project

Come on, launch Visual Studio and create a new WPF application. Make it target .NET framework 4.5 and give it a proper name. I named it KinectStreams. In the XAML file, add an <Image> element and three <Button> controls. The image’s source will be updated every time we have a new Kinect frame. The buttons simply switch between color, depth and infrared frame types.

<Grid>
    <Image Name="camera" />
    <Grid>
        <StackPanel>
            <Button Content="Color" Click="Color_Click" />
            <Button Content="Depth" Click="Depth_Click" />
            <Button Content="Infrared" Click="Infrared_Click" />
        </StackPanel>
    </Grid>
</Grid>

Initializing the sensor

Let’s now dive into the C# code! The primary namespace which provides us with the Kinect-related functionality remains the same:

using Microsoft.Kinect;

Do not forget to add a reference to the corresponding dll file (you’ll normally find it under C:\Program Files\Microsoft SDKs\Kinect\v2.0-DevPreview1311\Assemblies\Microsoft.Kinect.dll).

You now need two classes for connecting to the sensor and reading its camera streams. The first one is KinectSensor class. The second, and this is new to Kinect v2, is MultiSourceFrameReader class:

KinectSensor _sensor;
MultiSourceFrameReader _reader;

Comparing to the first SDK, Kinect SDK v2 Developer Preview provides an easier way for accessing the active sensor. Here’s how you connect to the sensor and start it up:

_sensor = KinectSensor.Default;

if (_sensor != null)
{
    _sensor.Open();
}

Reading the streams

That’s it! We have now connected to the sensor. Let’s specify which streams we need:

_reader = _sensor.OpenMultiSourceFrameReader(FrameSourceTypes.Color |
                                             FrameSourceTypes.Depth |
                                             FrameSourceTypes.Infrared);
_reader.MultiSourceFrameArrived += Reader_MultiSourceFrameArrived;

The MultiSourceFrameArrived event fires when a frame of the specified types is available. Here’s how we can open the color, depth and infrared frames in a single method.

void Reader_MultiSourceFrameArrived(object sender, MultiSourceFrameArrivedEventArgs e)
{
    // Get a reference to the multi-frame
    var reference = e.FrameReference.AcquireFrame();

    // Open color frame
    using (var frame = reference.ColorFrameReference.AcquireFrame())
    {
        if (frame != null)
        {
            // Do something with the frame...
        }
    }

    // Open depth frame
    using (var frame = reference.DepthFrameReference.AcquireFrame())
    {
        if (frame != null)
        {
            // Do something with the frame...
        }
    }

    // Open infrared frame
    using (var frame = reference.InfraredFrameReference.AcquireFrame())
    {
        if (frame != null)
        {
            // Do something with the frame...
        }
    }
}

We’ll now see how to convert each frame reference into a WPF bitmap and display it to our <Image> element.

Color stream

The raw color images have been increased to 1920×1080 resolution. Kinect only gives us an array of pixel RGB values. We need to convert this array into a proper WPF representation. I will not get into more details about WPF bitmap processing – you can simply copy and paste the following method:

private ImageSource ToBitmap(ColorFrame frame)
{
    int width = frame.FrameDescription.Width;
    int height = frame.FrameDescription.Height;

    byte[] pixels = new byte[width * height * ((PixelFormats.Bgr32.BitsPerPixel + 7) / 8)];

    if (frame.RawColorImageFormat == ColorImageFormat.Bgra)
    {
        frame.CopyRawFrameDataToArray(pixels);
    }
    else
    {
        frame.CopyConvertedFrameDataToArray(pixels, ColorImageFormat.Bgra);
    }

    int stride = width * format.BitsPerPixel / 8;

    return BitmapSource.Create(width, height, 96, 96, format, null, pixels, stride);
}

Depth stream

The depth stream provides us with the depth value of every point in the visible area. The depth value is the distance of the sensor. We usually represent the furthest distances using black color and the nearest distances using white color. However, it’s up to you to pick different colors when visualizing depth data. In this conversion, we need two arrays: one array for storing the depth values and another one for storing the corresponding colored values.

private ImageSource ToBitmap(DepthFrame frame)
{
    int width = frame.FrameDescription.Width;
    int height = frame.FrameDescription.Height;

    ushort minDepth = frame.DepthMinReliableDistance;
    ushort maxDepth = frame.DepthMaxReliableDistance;

    ushort[] depthData = new ushort[width * height];
    byte[] pixelData = new byte[width * height * (PixelFormats.Bgr32.BitsPerPixel + 7) / 8];

    frame.CopyFrameDataToArray(depthData);

    int colorIndex = 0;
    for (int depthIndex = 0; depthIndex < depthData.Length; ++depthIndex)
    {
        ushort depth = depthData[depthIndex];
        byte intensity = (byte)(depth >= minDepth && depth <= maxDepth ? depth : 0);

        pixelData[colorIndex++] = intensity; // Blue
        pixelData[colorIndex++] = intensity; // Green
        pixelData[colorIndex++] = intensity; // Red

        ++colorIndex;
    }

    int stride = width * format.BitsPerPixel / 8;

    return BitmapSource.Create(width, height, 96, 96, format, null, pixelData, stride);
}

Infrared stream

The infrared sensor is the ability to view clearly into the dark! Accessing the infrared stream is a new feature, added to Kinect v2. Similarly to displaying depth data, we again need two arrays for holding the infrared and the depth values. Here you go:

private ImageSource ToBitmap(InfraredFrame frame)
{
    int width = frame.FrameDescription.Width;
    int height = frame.FrameDescription.Height;

    ushort[] infraredData = new ushort[width * height];
    byte[] pixelData = new byte[width * height * (PixelFormats.Bgr32.BitsPerPixel + 7) / 8];

    frame.CopyFrameDataToArray(infraredData);

    int colorIndex = 0;
    for (int infraredIndex = 0; infraredIndex < infraredData.Length; ++infraredIndex)
    {
        ushort ir = infraredData[infraredIndex];
        byte intensity = (byte)(ir >> 8);

        pixelData[colorIndex++] = intensity; // Blue
        pixelData[colorIndex++] = intensity; // Green   
        pixelData[colorIndex++] = intensity; // Red

        ++colorIndex;
    }

    int stride = width * format.BitsPerPixel / 8;

    return BitmapSource.Create(width, height, 96, 96, format, null, pixels, stride);
}

That’s it! You can now display every bitmap stream! The only thing left to do is call the corresponding method and display the frame. This is how to display the color frame, for example:

// Open color frame
using (var frame = reference.ColorFrameReference.AcquireFrame())
{
    if (frame != null)
    {
        camera.Source = ToBitmap(frame);
    }
}

Download the source code and test for yourself. Note that the demo contains additional code for body tracking. We’ll cover body tracking throughout the next blog post.

Big hint: Kinect v2 requires you to start the KinectService.exe program before running any Kinect 2 apps. I always forget this detail, so I open this executable using a single line of C# code:

System.Diagnostics.Process.Start(@"C:\Windows\System32\KinectService.exe");

PS 1: Vitruvius

If you want to automate the above bitmap-conversion processes, consider downloading Vitruvius. Vitruvius is a free & open-source library I built, which provides many utilities for your Kinect applications, including gesture detection, voice recognition and drawing extensions. Give it a try, enjoy and even contribute yourself!

Here is how you can display a color frame using Vitruvius in a single line of C# code:

using (var frame = colorFrameReference.AcquireFrame())
{
    if (frame != null)
    {
        camera.Source = frame.ToBitmap();
    }
}

Similarly, you can display a depth or infrared frame with no pain!

PS: New Kinect book – 20% off

Well, I am publishing a new ebook about Kinect development in a couple months. It is an in-depth guide about Kinect, using simple language and step-by-step examples. You’ll learn usability tips, performance tricks and best practices for implementing robust Kinect apps. Please meet Kinect Essentials, the essence of my 3 years of teaching, writing and developing for the Kinect platform. Oh, did I mention that you’ll get a 20% discount if you simply subscribe now? Hurry up ;-)

Subscribe here for 20% off

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Vangos Pterneas
Product Manager LightBuzz
United Kingdom United Kingdom
Vangos Pterneas is a Microsoft Most Valuable Professional in the Kinect technology. He helps companies from all over the world grow their revenue by creating profitable software products. Vangos is the owner of LightBuzz Software agency and author of two technical books.
Follow on   Twitter   Google+   LinkedIn

Comments and Discussions

 
QuestionKinect 2.o Sensor - Freezing Issue PinmemberMember 170594320-Aug-14 13:09 
AnswerRe: Kinect 2.o Sensor - Freezing Issue PinpremiumVangos Pterneas21-Aug-14 23:52 
QuestionKinect Library PinmemberMember 170594314-Jul-14 4:51 
AnswerRe: Kinect Library PinpremiumVangos Pterneas15-Jul-14 8:21 
QuestionKinect 2.0 Wrapper Event PinmemberMember 17059433-Jun-14 5:23 
AnswerRe: Kinect 2.0 Wrapper Event PinpremiumVangos Pterneas5-Jun-14 11:09 
GeneralRe: Kinect 2.0 Wrapper Event PinmemberMember 17059436-Jun-14 5:16 
GeneralRe: Kinect 2.0 Wrapper Event PinpremiumVangos Pterneas6-Jun-14 8:47 
QuestionColor Frame Resolution [modified] PinmemberMember 1068306620-Mar-14 23:50 
QuestionCapturing and Recording depth and color Simultaneously PinmemberMember 1068306619-Mar-14 5:45 
AnswerRe: Capturing and Recording depth and color Simultaneously PinpremiumVangos Pterneas19-Mar-14 6:01 
GeneralRe: Capturing and Recording depth and color Simultaneously PinmemberMember 1068306619-Mar-14 6:45 
GeneralRe: Capturing and Recording depth and color Simultaneously PinpremiumVangos Pterneas19-Mar-14 12:33 
GeneralRe: Capturing and Recording depth and color Simultaneously PinmemberMember 1068306619-Mar-14 22:58 
QuestionKinect v2 for windows not working Pinmembershrikrishna03221-Feb-14 23:24 
AnswerRe: Kinect v2 for windows not working PinmemberVangos Pterneas22-Feb-14 13:37 
GeneralRe: Kinect v2 for windows not working Pinmembershrikrishna03222-Feb-14 19:58 
GeneralRe: Kinect v2 for windows not working PinmemberVangos Pterneas23-Feb-14 4:45 
GeneralRe: Kinect v2 for windows not working Pinmembershrikrishna0323-Mar-14 23:26 
GeneralRe: Kinect v2 for windows not working PinmemberVangos Pterneas3-Mar-14 23:31 
Questionsoftware requirements.. Pinmembershrikrishna03220-Feb-14 20:50 
AnswerRe: software requirements.. PinmemberVangos Pterneas21-Feb-14 2:40 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web04 | 2.8.140821.2 | Last Updated 21 Feb 2014
Article Copyright 2014 by Vangos Pterneas
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid