Click here to Skip to main content
15,868,058 members
Articles / Desktop Programming / WPF

Kinect v2 Point Cloud using VTK for visualisation

Rate me:
Please Sign up or sign in to vote.
4.78/5 (14 votes)
27 Dec 2014CPOL4 min read 81.4K   7K   24   25
The goal of this article is to extract a point cloud using the Microsoft Kinect v2 sensor, visualize it using the VTK toolkit, save it for printing or further work (e.g. using Meshlab)

Image 1

Introduction, Quick Start

  1. Prerequisites:
    • A Kinect v2 sensor
    • The Microsoft Kinect V2 SDK version 2.0 installed
    • Visual Studio 2012 or higher
  2. Start the program KinectPointCloud.exe
  3. Check the checkbox "Save if depth frame ..."
  4. Click on „Capture“ – see below

    Image 2

  5. Hold still in front of the sensor while scanning.
    Capture will stop when the image quality is high – „Depth OK…“ is over 45% - see below

    Image 3

  6. Click on „Show Point Cloud“ to open the point cloud
  7. View the point cloud with or without color info

    Image 4

  8. Open the point cloud for further editing in external tools like MeshLab (either the .ply file -contains color info - or the .xyz file)

    Image 5

Other Source Code Used

Different open source code is used within the project:

  • Parts of the Microsoft samples contained within the Kinect SDK
  • The VTK library by means of the C# wrapper Activiz:
  • Parts of the code of Vangos Pternas from his Codeproject article:

Coding

Grabbing the Data with Kinect

Extracting a point cloud from the Kinect by using the standard Microsoft sample code gives quite poor results – e.g. up to 30% of the depth frame points miss depth information. Either the depth is zero or there are artefacts on the depth image which occur from low depth precision.

Improve depth quality

The method to improve the depth quality used is Frame Interpolation.
A number of 10 frames are captured, the „interpolated“ image is the average of the depths.

For quality check of the grabbed image, I also calculate an "entropy“ image, which consists of the difference between maximum and minimum depth, i.e., the pixel quality is best if the pixel is zero (all measurements show the same depth), and the pixel quality is worse if the entropy is higher. For instance, if a head is scanned in the range of 0,5 to 1 m, the camera and the scanned target are still, the ambient light is OK (scan with daylight), the depth precision should be between 0 and 5 for > 95% of the pixels. This means a maximum of 5 millimeters depth uncertainty for the points scanned.

Image 6

A depth histogram is calculated:
The x axis shows the depth precision from 0 to 32. All depth precisions above 32 are ignored, since they are less than 1% of the total scanned points.

On the y axis, the number of points are plotted.
Fort the example below, you see that more than 50% of the points have a precision of 1 mm or less and 34 % of the points a depth precision of 2, etc.

Image 7

If you scan an image like in the example below, only about 16% of the points have a good precision, then you better do not use the result. You will have a lot of artefacts, see the entropy image below.

Image 8

Code Sample

The Main Window used is MainWindow.xaml, the class doing all the Kinect work is KinectUC.

Start Capture:

C++
private void buttonConnect_Click(object sender, RoutedEventArgs e)
{
       KinectConnect();
}

Save point cloud:

C++
private void buttonSave_Click(object sender, RoutedEventArgs e)
{
           SaveDepthPoints();
           SaveDepthBitmap();
           SaveDepthPointsInterpolated();
           SaveImageInterpolated();
           SaveColorBitmap();
           SaveColorInfoForDepth_FileAndImage();
 }

Show the Point cloud with:

C++
private void buttonShowPointCloud_Click(object sender, RoutedEventArgs e)
{
          this.OpenSavedDepthData();

          ShowDepthScreenshot();
          FormVTK vtkForm = new FormVTK();
          vtkForm.ShowPointCloud
           (this.DepthMetaData.FrameData, DepthMetaData.XResDefault, DepthMetaData.YResDefault);
}

There are utils for Kinect data handling like:

C++
namespace KinectUtils
{
    public class DepthMetaData : MetaDataBase
    {            
           ...
        public WriteableBitmap DepthBitmap
        {
            get
            {
                if (depthBitmap == null)
                {
                    this.pixels = ImageUtil.ConvertUshortToByte(this.FrameData);
                    depthBitmap = ImageUtil.ByteArrayToWriteableBitmap_Gray
                    (pixels, DepthMetaData.XResDefault, DepthMetaData.YResDefault);
                }
                return depthBitmap;
            }
        }

or VTK handling, like public class ColorUtils.

The code can be best understood if you debug the main usecases: Capture, save data and show in VTK window.

Remarks

The depth statistics is only valid if the target and the scanner do not move.

If large movements occur, the percentage values may not add to 100% total. The reason is that the depth points cut out of the image are different if the sensor or the scan target move.

Conclusion

A point cloud can be scanned with acceptable quality using the Microsoft Kinect v2 camera if one uses the procedures described in this article like image interpolation and saving the point cloud only if the depth precision is high.

Code Usage

The code and all information of this article may be used in any application as long as you cite this article in the acknowledgements.

I am happy for any comments or ideas for code improvements.

Points of Interest

I am interested in stitching point clouds scanned from different angles together and then put them together to get a better point cloud, e.g. using some ICP algorithm. Any tip or even better - C# code which fits with the architecture used here is highly appreciated.

See Also

"Kinect-Point-Cloud-using-OpenGL" - this article shows the UI in an OpenGL Control. For my further work with the Kinect, I'll use the OpenGl control. However, I keep the code of the VTK, for people interested in this, and for later usage of other VTK samples.

History

  • Version 0.9.0.1 of code on 9/30/2014
  • 0.9.0.2 on 12/26/2014
    • Replaced Kinect version 2.0
    • Minor code fix for opening point clouds

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
Engineer
Germany Germany
Other interests: music playing, photography, travelling

Comments and Discussions

 
QuestionGet VTK to work under VS 2012+ Pin
geo_dev19-Aug-15 11:14
geo_dev19-Aug-15 11:14 
AnswerRe: Get VTK to work under VS 2012+ Pin
geo__dev19-Aug-15 11:20
geo__dev19-Aug-15 11:20 
GeneralRe: Get VTK to work under VS 2012+ Pin
Edgar Maass30-Sep-15 8:58
professionalEdgar Maass30-Sep-15 8:58 
GeneralRe: Get VTK to work under VS 2012+ Pin
geo__dev1-Oct-15 8:06
geo__dev1-Oct-15 8:06 
GeneralRe: Get VTK to work under VS 2012+ Pin
Edgar Maass1-Oct-15 8:18
professionalEdgar Maass1-Oct-15 8:18 
GeneralRe: Get VTK to work under VS 2012+ Pin
geo__dev1-Oct-15 10:05
geo__dev1-Oct-15 10:05 
GeneralRe: Get VTK to work under VS 2012+ Pin
Edgar Maass1-Oct-15 11:01
professionalEdgar Maass1-Oct-15 11:01 
GeneralRe: Get VTK to work under VS 2012+ Pin
geo__dev1-Oct-15 11:20
geo__dev1-Oct-15 11:20 
AnswerRe: Get VTK to work under VS 2012+ Pin
Edgar Maass30-Sep-15 8:45
professionalEdgar Maass30-Sep-15 8:45 
GeneralMy vote of 5 Pin
Member 239256316-Jun-15 23:34
Member 239256316-Jun-15 23:34 
Generalaligning point cloud mesh Pin
Member 1134252729-Dec-14 19:50
Member 1134252729-Dec-14 19:50 
GeneralRe: aligning point cloud mesh Pin
Edgar Maass31-Dec-14 1:05
professionalEdgar Maass31-Dec-14 1:05 
QuestionCapturing point cloud of small object toy of size 5 x 5 inches not successful. Pin
Vijay PPP21-Dec-14 14:44
Vijay PPP21-Dec-14 14:44 
AnswerRe: Capturing point cloud of small object toy of size 5 x 5 inches not successful. Pin
Vijay PPP21-Dec-14 15:19
Vijay PPP21-Dec-14 15:19 
GeneralRe: Capturing point cloud of small object toy of size 5 x 5 inches not successful. Pin
Edgar Maass26-Dec-14 6:53
professionalEdgar Maass26-Dec-14 6:53 
GeneralRe: Capturing point cloud of small object toy of size 5 x 5 inches not successful. Pin
Edgar Maass27-Dec-14 3:13
professionalEdgar Maass27-Dec-14 3:13 
Questionkinect xbox Pin
Member 1079890717-Nov-14 9:36
Member 1079890717-Nov-14 9:36 
AnswerRe: kinect xbox Pin
Edgar Maass17-Nov-14 22:20
professionalEdgar Maass17-Nov-14 22:20 
GeneralRe: kinect xbox Pin
Member 1079890717-Nov-14 23:48
Member 1079890717-Nov-14 23:48 
GeneralRe: kinect xbox Pin
Edgar Maass18-Nov-14 0:20
professionalEdgar Maass18-Nov-14 0:20 
-You have to install the Kinect v1 SDK
-read the samples of v1 SDK to know about the capture methods
-replace all the capture methods in my code with the ones of v1 SDK
GeneralRe: kinect xbox Pin
Member 1079890718-Nov-14 0:34
Member 1079890718-Nov-14 0:34 
QuestionImages Pin
PeterWalter2-Oct-14 23:42
professionalPeterWalter2-Oct-14 23:42 
AnswerRe: Images Pin
Edgar Maass3-Oct-14 1:25
professionalEdgar Maass3-Oct-14 1:25 
GeneralRe: Images Pin
Edgar Maass3-Oct-14 7:58
professionalEdgar Maass3-Oct-14 7:58 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.