Get the IR Stream and Control the IR Emitter – Kinect for Windows SDK

In this post, I am going to talk about how to capture the Infrared stream from the Kinect sensor and control the IR emitter using Kinect for Window SDK. This feature is available with the Kinect for Windows SDK v1.6. The major components for a Kinect device consist of a depth sensor, color camera, IR Emitter, and a set of microphones with everything secured inside box. The box is attached to a small motor working as the base that enables the device to be tilted.

The IR emitter and the depth sensor work together to produce the depth data, where each pixel comprises the information about the distance between the sensor and the object within the Kinect view area. The IR is invisible to the human eye as it has a longer wavelength than the highest wavelength that a human eye can see in a spectrum. This IR enables Kinect to see in the dark as well.

Similar to the color, depth and skeleton stream data, Kinect SDK also exposes the APIs for capturing the IR data stream. You can capture images in low light conditions, by reading the infrared stream from the Kinect sensor. This will also help you to calibrate other camera with the Kinect sensor. Let’s have a look how to capture the IR data stream using Kinect for Windows SDK.

Capturing the IR Data Stream

The Kinect sensor returns 16 bits per pixel infrared data with a resolution of 640 x 480 as an color image format, and it supports up to 30 FPS. Following are the couple of images (taken in a complete dark room) that captures from IR stream data.

You cannot read color and infrared streams simultaneously, but you can read depth and infrared data simultaneously. The reason behind this is that an infrared stream is captured as a part of a color image format.

Capturing IR stream data is as simple as capturing a color image stream, as the SDK returns the infrared stream as a part of the color image stream data. The only changes are required on ColorImageFormat and PixelFormats.

First, enable the ColorStream with the image format type of InfraredResolution640x480Fps30 as shown below:

this.sensor.ColorStream.Enable(ColorImageFormat.InfraredResolution640x480Fps30);

Then within ColorFrameReady event handler, set the PixelFormats to Gray16 while creating the bitmap source as assigning it to image control (say named as cameraControl).

this.cameraControl.Source = BitmapSource.Create(imageFrame.Width, imageFrame.Height, 
                            96,96,PixelFormats.Gray16,null, pixelData, imageframe.Width * 2);

You can also use the WritableBitmap to write it as pixels as well.

Controlling the IR Emitter

You can also turn off the IR emitter using the KinectSensor.ForceInfraredEmitterOff property. By default, this property is set to false. To turn the IR light off, set the property to true.

ForceInfraredEmitterOff will only work with Kinect for Windows Sensor. Not Kinect for Xbox Sensor.

You can read a bit details about ForceInfraredEmitterOff here.

Let’s put everything in a single solution and see how to in dark / night/ low light using Kinect. We are going to build a "Night vision camera Using Kinect".

Building a Simple "Kinect Night Vision Camera"

Setup a new Visual Studio project by following the below steps:

1. Start a new instance of Visual Studio.
2. Create a new project by navigating to File | New Project.
3. You will see the New Project dialog box. Choose Visual C# as our development language, select WPF Application Template, and type the name as KinectNightVisionCamera
4. From Solution Explorer, right-click on the References folder and select Add References.
5. Include a reference of the Microsoft.Kinect assembly.

Open the "MainWindow.Xaml" file from the solution explorer and use the below XAML snippet for designing a very basic UI for the camera.

<Grid>
        <Grid.RowDefinitions>
            <RowDefinition Height="40"></RowDefinition>
            <RowDefinition></RowDefinition>
        </Grid.RowDefinitions>
        <StackPanel Background="LightGray">
            <Label Content="Kinect Night Vision Camera" VerticalAlignment="Center" 
             HorizontalAlignment="Left" FontSize="20" Foreground="Black" />
        </StackPanel>
        <GroupBox Grid.Row="1"  Height="231" HorizontalAlignment="Left" 
         Margin="17,16,0,0" Name="groupBox1" VerticalAlignment="Top" Width="280">
            <Image x:Name="cameraControl" Stretch="UniformToFill"/>
        </GroupBox>
        <Button Name="buttonExit" Content="Exit" Grid.Row="1" Height="34" 
         HorizontalAlignment="Left" Margin="307,206,0,0"  
         VerticalAlignment="Top" Width="81" Click="buttonExit_Click" />
        <CheckBox IsChecked="True" Checked="ControlIR" Unchecked="ControlIR" 
         Content="IR ON/OFF" Grid.Row="1" Height="21" HorizontalAlignment="Left" 
         Margin="308,172,0,0" Name="checkBox1" VerticalAlignment="Top" Width="76" />
    </Grid>

The basic design of the application will looks similar to the below image.

Now, open the MainWindow.xaml.cs file and first of all, include the namespaces as:

using Microsoft.Kinect;

Next tasks for building the applications are follows:

1. Get the sensor object and enable the Infrared stream
2. Start the sensor
3. Handle the stream event handler

Here is the complete code block.

KinectSensor sensor;

        /// <summary>
        /// Initializes a new instance of the <see cref="MainWindow" /> class.
        /// </summary>
        public MainWindow()
        {
            InitializeComponent();
            Loaded += new RoutedEventHandler(MainWindow_Loaded);
        }
        
        /// <summary>
        /// Handles the Loaded event of the MainWindow control.
        /// </summary>
        /// <param name="sender">The source of the event.</param>
        /// <param name="e">The <see cref="RoutedEventArgs" /> 
        /// instance containing the event data.</param>
        void MainWindow_Loaded(object sender, RoutedEventArgs e)
        {
            if (KinectSensor.KinectSensors.Count > 0)
            {
                this.sensor = KinectSensor.KinectSensors.Where
                (kinect => kinect.Status == KinectStatus.Connected).FirstOrDefault();
                this.sensor.ColorStream.Enable(ColorImageFormat.InfraredResolution640x480Fps30);
                this.sensor.ColorFrameReady += 
                new EventHandler<ColorImageFrameReadyEventArgs>(sensor_ColorFrameReady);
                this.sensor.Start();
            }
            else
            {
                MessageBox.Show("No Device Connected");
                this.Close();
            }
        }
        
/// <summary>
/// Handles the ColorFrameReady event of the sensor control.
/// </summary>
/// <param name="sender">The source of the event.</param>
/// <param name="e">The <see cref="ColorImageFrameReadyEventArgs" /> 
/// instance containing the event data.</param>
        void sensor_ColorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
        {
            // Get the current image frame from sensor.
            using (ColorImageFrame imageFrame = e.OpenColorImageFrame())
            {
                // Check if there is any frame drop
                if (imageFrame == null)
                {
                    return;
                }

                // get the frame pixel data length
                byte[] pixelData = new byte[imageFrame.PixelDataLength];

                imageFrame.CopyPixelDataTo(pixelData);

// note that stride has been set to imageFrame.Width*2, as its a 16bits/pixel
                cameraControl.Source = BitmapSource.Create(imageFrame.Width,
                    imageFrame.Height,
                    96,
                    96,
                    PixelFormats.Gray16,
                    null,
                    pixelData,
                    imageFrame.Width * 2);
            }
        }

If you run the application, you will find the sensor is returning a gray scale steam with many dots, which is nothing but the IR dots.

Handle the ForceInfraredEmitterOff property by using the following code snippet. The ControlIR() method is called from the checkbox’s checked and unchecked events. (The below code will only work with Kinect for Windows Sensor. If you are using Kinect for xBox Sensor, you will get an InvalidOperationException with the message "The feature is not supported by this version of the hardware".

/// <summary>
/// Controls the IR.
/// </summary>
/// <param name="sender">The sender.</param>
/// <param name="e">The <see cref="RoutedEventArgs" /> 
/// instance containing the event data.</param>
        private void ControlIR(object sender, RoutedEventArgs e)
        {
            CheckBox chkbox = sender as CheckBox;

            if (chkbox.IsChecked == true)
            {
                this.sensor.ForceInfraredEmitterOff = false;
            }
            else if (chkbox.IsChecked == false)
            {
                this.sensor.ForceInfraredEmitterOff = true;
            }
        }

Finally, in the button exit close the application and before closing, make sure you are turning the sensor off if it is running.

/// <summary>
/// Handles the Click event of the buttonExit control.
/// </summary>
/// <param name="sender">The source of the event.</param>
/// <param name="e">The <see cref="RoutedEventArgs" /> 
/// instance containing the event data.</param>
        private void buttonExit_Click(object sender, RoutedEventArgs e)
        {
            if(this.sensor !=null && this.sensor.IsRunning)
            {
                this.sensor.Stop();
            }
            this.Close();
        }

Quick Video – Kinect Night Vision Camera

Download Sample Application With Source Code

Download the "Kinect Night Vision Camera Application" (KinectNightVisionCamera – SDK v1.6.zip) from the location http://abhijan.me/VSPaYD.