Click here to Skip to main content
11,580,572 members (70,181 online)
Click here to Skip to main content

Tagged as

Kinect for Windows version 2: Color, depth and infrared streams

, 21 Feb 2014 CPOL 27K 8
Rate this:
Please Sign up or sign in to vote.
NOTE: This is preliminary software and/or hardware and APIs are preliminary and subject to change. A month ago, I was happy to receive a brand-new Kinect for Windows version 2 Developer Preview sensor. You can read my previous blog post about the capabilities of the new device. Kinect v2 now include

NOTE: This is preliminary software and/or hardware and APIs are preliminary and subject to change.

A month ago, I was happy to receive a brand-new Kinect for Windows version 2 Developer Preview sensor. You can read my previous blog post about the capabilities of the new device. Kinect v2 now includes 4 types of input streams:

  • Color
  • Depth
  • Infrared
  • Body
  • Audio

Today I will show you how you can acquire and display each bitmap input in a Windows application. In the next blog post, we’ll talk about body tracking. Here is a quick video I made which demonstrates the different streams provided by the new sensor.

<iframe width="100%" height="400" src="http://www.codeproject.com/www.youtube.com/embed/86ALowFuBkY" frameborder="0" allowfullscreen></iframe>

You can download the source code here. Read on if you to implement this functionality yourself.

Requirements

Creating a new project

Come on, launch Visual Studio and create a new WPF application. Make it target .NET framework 4.5 and give it a proper name. I named it KinectStreams. In the XAML file, add an <Image> element and three <Button> controls. The image’s source will be updated every time we have a new Kinect frame. The buttons simply switch between color, depth and infrared frame types.

<Grid>
    <Image Name="camera" />
    <Grid>
        <StackPanel>
            <Button Content="Color" Click="Color_Click" />
            <Button Content="Depth" Click="Depth_Click" />
            <Button Content="Infrared" Click="Infrared_Click" />
        </StackPanel>
    </Grid>
</Grid>

Initializing the sensor

Let’s now dive into the C# code! The primary namespace which provides us with the Kinect-related functionality remains the same:

using Microsoft.Kinect;

Do not forget to add a reference to the corresponding dll file (you’ll normally find it under C:\Program Files\Microsoft SDKs\Kinect\v2.0-DevPreview1311\Assemblies\Microsoft.Kinect.dll).

You now need two classes for connecting to the sensor and reading its camera streams. The first one is KinectSensor class. The second, and this is new to Kinect v2, is MultiSourceFrameReader class:

KinectSensor _sensor;
MultiSourceFrameReader _reader;

Comparing to the first SDK, Kinect SDK v2 Developer Preview provides an easier way for accessing the active sensor. Here’s how you connect to the sensor and start it up:

_sensor = KinectSensor.Default;

if (_sensor != null)
{
    _sensor.Open();
}

Reading the streams

That’s it! We have now connected to the sensor. Let’s specify which streams we need:

_reader = _sensor.OpenMultiSourceFrameReader(FrameSourceTypes.Color |
                                             FrameSourceTypes.Depth |
                                             FrameSourceTypes.Infrared);
_reader.MultiSourceFrameArrived += Reader_MultiSourceFrameArrived;

The MultiSourceFrameArrived event fires when a frame of the specified types is available. Here’s how we can open the color, depth and infrared frames in a single method.

void Reader_MultiSourceFrameArrived(object sender, MultiSourceFrameArrivedEventArgs e)
{
    // Get a reference to the multi-frame
    var reference = e.FrameReference.AcquireFrame();

    // Open color frame
    using (var frame = reference.ColorFrameReference.AcquireFrame())
    {
        if (frame != null)
        {
            // Do something with the frame...
        }
    }

    // Open depth frame
    using (var frame = reference.DepthFrameReference.AcquireFrame())
    {
        if (frame != null)
        {
            // Do something with the frame...
        }
    }

    // Open infrared frame
    using (var frame = reference.InfraredFrameReference.AcquireFrame())
    {
        if (frame != null)
        {
            // Do something with the frame...
        }
    }
}

We’ll now see how to convert each frame reference into a WPF bitmap and display it to our <Image> element.

Color stream

The raw color images have been increased to 1920×1080 resolution. Kinect only gives us an array of pixel RGB values. We need to convert this array into a proper WPF representation. I will not get into more details about WPF bitmap processing – you can simply copy and paste the following method:

private ImageSource ToBitmap(ColorFrame frame)
{
    int width = frame.FrameDescription.Width;
    int height = frame.FrameDescription.Height;

    byte[] pixels = new byte[width * height * ((PixelFormats.Bgr32.BitsPerPixel + 7) / 8)];

    if (frame.RawColorImageFormat == ColorImageFormat.Bgra)
    {
        frame.CopyRawFrameDataToArray(pixels);
    }
    else
    {
        frame.CopyConvertedFrameDataToArray(pixels, ColorImageFormat.Bgra);
    }

    int stride = width * format.BitsPerPixel / 8;

    return BitmapSource.Create(width, height, 96, 96, format, null, pixels, stride);
}

Depth stream

The depth stream provides us with the depth value of every point in the visible area. The depth value is the distance of the sensor. We usually represent the furthest distances using black color and the nearest distances using white color. However, it’s up to you to pick different colors when visualizing depth data. In this conversion, we need two arrays: one array for storing the depth values and another one for storing the corresponding colored values.

private ImageSource ToBitmap(DepthFrame frame)
{
    int width = frame.FrameDescription.Width;
    int height = frame.FrameDescription.Height;

    ushort minDepth = frame.DepthMinReliableDistance;
    ushort maxDepth = frame.DepthMaxReliableDistance;

    ushort[] depthData = new ushort[width * height];
    byte[] pixelData = new byte[width * height * (PixelFormats.Bgr32.BitsPerPixel + 7) / 8];

    frame.CopyFrameDataToArray(depthData);

    int colorIndex = 0;
    for (int depthIndex = 0; depthIndex < depthData.Length; ++depthIndex)
    {
        ushort depth = depthData[depthIndex];
        byte intensity = (byte)(depth >= minDepth && depth <= maxDepth ? depth : 0);

        pixelData[colorIndex++] = intensity; // Blue
        pixelData[colorIndex++] = intensity; // Green
        pixelData[colorIndex++] = intensity; // Red

        ++colorIndex;
    }

    int stride = width * format.BitsPerPixel / 8;

    return BitmapSource.Create(width, height, 96, 96, format, null, pixelData, stride);
}

Infrared stream

The infrared sensor is the ability to view clearly into the dark! Accessing the infrared stream is a new feature, added to Kinect v2. Similarly to displaying depth data, we again need two arrays for holding the infrared and the depth values. Here you go:

private ImageSource ToBitmap(InfraredFrame frame)
{
    int width = frame.FrameDescription.Width;
    int height = frame.FrameDescription.Height;

    ushort[] infraredData = new ushort[width * height];
    byte[] pixelData = new byte[width * height * (PixelFormats.Bgr32.BitsPerPixel + 7) / 8];

    frame.CopyFrameDataToArray(infraredData);

    int colorIndex = 0;
    for (int infraredIndex = 0; infraredIndex < infraredData.Length; ++infraredIndex)
    {
        ushort ir = infraredData[infraredIndex];
        byte intensity = (byte)(ir >> 8);

        pixelData[colorIndex++] = intensity; // Blue
        pixelData[colorIndex++] = intensity; // Green   
        pixelData[colorIndex++] = intensity; // Red

        ++colorIndex;
    }

    int stride = width * format.BitsPerPixel / 8;

    return BitmapSource.Create(width, height, 96, 96, format, null, pixels, stride);
}

That’s it! You can now display every bitmap stream! The only thing left to do is call the corresponding method and display the frame. This is how to display the color frame, for example:

// Open color frame
using (var frame = reference.ColorFrameReference.AcquireFrame())
{
    if (frame != null)
    {
        camera.Source = ToBitmap(frame);
    }
}

Download the source code and test for yourself. Note that the demo contains additional code for body tracking. We’ll cover body tracking throughout the next blog post.

Big hint: Kinect v2 requires you to start the KinectService.exe program before running any Kinect 2 apps. I always forget this detail, so I open this executable using a single line of C# code:

System.Diagnostics.Process.Start(@"C:\Windows\System32\KinectService.exe");

PS 1: Vitruvius

If you want to automate the above bitmap-conversion processes, consider downloading Vitruvius. Vitruvius is a free & open-source library I built, which provides many utilities for your Kinect applications, including gesture detection, voice recognition and drawing extensions. Give it a try, enjoy and even contribute yourself!

Here is how you can display a color frame using Vitruvius in a single line of C# code:

using (var frame = colorFrameReference.AcquireFrame())
{
    if (frame != null)
    {
        camera.Source = frame.ToBitmap();
    }
}

Similarly, you can display a depth or infrared frame with no pain!

PS: New Kinect book – 20% off

Well, I am publishing a new ebook about Kinect development in a couple months. It is an in-depth guide about Kinect, using simple language and step-by-step examples. You’ll learn usability tips, performance tricks and best practices for implementing robust Kinect apps. Please meet Kinect Essentials, the essence of my 3 years of teaching, writing and developing for the Kinect platform. Oh, did I mention that you’ll get a 20% discount if you simply subscribe now? Hurry up ;-)

Subscribe here for 20% off

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Vangos Pterneas
Product Manager LightBuzz
United Kingdom United Kingdom
Vangos Pterneas is a Microsoft Most Valuable Professional in the Kinect technology. He helps companies from all over the world grow their revenue by creating profitable software products. Vangos is the owner of LightBuzz Software agency and author of two technical books.

You may also be interested in...

Comments and Discussions

 
GeneralThank you Pin
Michaël Polla1 hr 27mins ago
memberMichaël Polla1 hr 27mins ago 
QuestionHighlighting players in depthframe Pin
Member 1160060712-Apr-15 3:59
memberMember 1160060712-Apr-15 3:59 
QuestionKinect depth image to 3D cloud point Pin
Arcsinos2-Apr-15 6:14
memberArcsinos2-Apr-15 6:14 
Questionkinect version 2 accelerometer Pin
Arcsinos23-Mar-15 4:26
memberArcsinos23-Mar-15 4:26 
AnswerRe: kinect version 2 accelerometer Pin
Vangos Pterneas26-Mar-15 5:22
professionalVangos Pterneas26-Mar-15 5:22 
GeneralRe: kinect version 2 accelerometer Pin
Arcsinos26-Mar-15 9:06
memberArcsinos26-Mar-15 9:06 
GeneralRe: kinect version 2 accelerometer Pin
Vangos Pterneas26-Mar-15 9:51
professionalVangos Pterneas26-Mar-15 9:51 
QuestionHow sensitive is the depth sensor ? Pin
PGT25-Feb-15 0:42
memberPGT25-Feb-15 0:42 
AnswerRe: How sensitive is the depth sensor ? Pin
Vangos Pterneas25-Feb-15 1:23
professionalVangos Pterneas25-Feb-15 1:23 
GeneralRe: How sensitive is the depth sensor ? Pin
PGT25-Feb-15 10:58
memberPGT25-Feb-15 10:58 
GeneralRe: How sensitive is the depth sensor ? Pin
Vangos Pterneas25-Feb-15 12:28
professionalVangos Pterneas25-Feb-15 12:28 
QuestionExtracting Raw frame data Pin
KamranQureshi13-Jan-15 7:35
memberKamranQureshi13-Jan-15 7:35 
AnswerRe: Extracting Raw frame data Pin
Vangos Pterneas13-Jan-15 9:53
professionalVangos Pterneas13-Jan-15 9:53 
QuestionHello.Did you know how to change certain parameters like brightness or Exposure or saturation..etc for this particular version of SDK(version 2.0 preview)? Pin
Member 1103756726-Aug-14 4:35
memberMember 1103756726-Aug-14 4:35 
AnswerRe: Hello.Did you know how to change certain parameters like brightness or Exposure or saturation..etc for this particular version of SDK(version 2.0 preview)? Pin
Vangos Pterneas28-Aug-14 1:13
professionalVangos Pterneas28-Aug-14 1:13 
QuestionPerformance Issues - Changing the resolution on the Kinect Pin
Member 1103751126-Aug-14 4:15
memberMember 1103751126-Aug-14 4:15 
AnswerRe: Performance Issues - Changing the resolution on the Kinect Pin
Vangos Pterneas26-Aug-14 4:25
professionalVangos Pterneas26-Aug-14 4:25 
GeneralRe: Performance Issues - Changing the resolution on the Kinect Pin
Boaz Telem26-Aug-14 4:54
memberBoaz Telem26-Aug-14 4:54 
GeneralRe: Performance Issues - Changing the resolution on the Kinect Pin
Vangos Pterneas28-Aug-14 1:12
professionalVangos Pterneas28-Aug-14 1:12 
QuestionKinect 2.o Sensor - Freezing Issue Pin
Member 170594320-Aug-14 13:09
memberMember 170594320-Aug-14 13:09 
AnswerRe: Kinect 2.o Sensor - Freezing Issue Pin
Vangos Pterneas21-Aug-14 23:52
professionalVangos Pterneas21-Aug-14 23:52 
QuestionKinect Library Pin
Member 170594314-Jul-14 4:51
memberMember 170594314-Jul-14 4:51 
AnswerRe: Kinect Library Pin
Vangos Pterneas15-Jul-14 8:21
memberVangos Pterneas15-Jul-14 8:21 
QuestionKinect 2.0 Wrapper Event Pin
Member 17059433-Jun-14 5:23
memberMember 17059433-Jun-14 5:23 
AnswerRe: Kinect 2.0 Wrapper Event Pin
Vangos Pterneas5-Jun-14 11:09
memberVangos Pterneas5-Jun-14 11:09 
GeneralRe: Kinect 2.0 Wrapper Event Pin
Member 17059436-Jun-14 5:16
memberMember 17059436-Jun-14 5:16 
GeneralRe: Kinect 2.0 Wrapper Event Pin
Vangos Pterneas6-Jun-14 8:47
memberVangos Pterneas6-Jun-14 8:47 
QuestionColor Frame Resolution Pin
Member 1068306620-Mar-14 23:50
memberMember 1068306620-Mar-14 23:50 
QuestionCapturing and Recording depth and color Simultaneously Pin
Member 1068306619-Mar-14 5:45
memberMember 1068306619-Mar-14 5:45 
AnswerRe: Capturing and Recording depth and color Simultaneously Pin
Vangos Pterneas19-Mar-14 6:01
memberVangos Pterneas19-Mar-14 6:01 
GeneralRe: Capturing and Recording depth and color Simultaneously Pin
Member 1068306619-Mar-14 6:45
memberMember 1068306619-Mar-14 6:45 
GeneralRe: Capturing and Recording depth and color Simultaneously Pin
Vangos Pterneas19-Mar-14 12:33
memberVangos Pterneas19-Mar-14 12:33 
GeneralRe: Capturing and Recording depth and color Simultaneously Pin
Member 1068306619-Mar-14 22:58
memberMember 1068306619-Mar-14 22:58 
QuestionKinect v2 for windows not working Pin
shrikrishna03221-Feb-14 23:24
membershrikrishna03221-Feb-14 23:24 
AnswerRe: Kinect v2 for windows not working Pin
Vangos Pterneas22-Feb-14 13:37
memberVangos Pterneas22-Feb-14 13:37 
GeneralRe: Kinect v2 for windows not working Pin
shrikrishna03222-Feb-14 19:58
membershrikrishna03222-Feb-14 19:58 
GeneralRe: Kinect v2 for windows not working Pin
Vangos Pterneas23-Feb-14 4:45
memberVangos Pterneas23-Feb-14 4:45 
GeneralRe: Kinect v2 for windows not working Pin
shrikrishna0323-Mar-14 23:26
membershrikrishna0323-Mar-14 23:26 
GeneralRe: Kinect v2 for windows not working Pin
Vangos Pterneas3-Mar-14 23:31
memberVangos Pterneas3-Mar-14 23:31 
Questionsoftware requirements.. Pin
shrikrishna03220-Feb-14 20:50
membershrikrishna03220-Feb-14 20:50 
AnswerRe: software requirements.. Pin
Vangos Pterneas21-Feb-14 2:40
memberVangos Pterneas21-Feb-14 2:40 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web03 | 2.8.150603.1 | Last Updated 21 Feb 2014
Article Copyright 2014 by Vangos Pterneas
Everything else Copyright © CodeProject, 1999-2015
Layout: fixed | fluid