Multithreaded Kinect Stream Saver Application

Introduction

This article is intended for researchers who wish to develop a markerless motion capture system based on a Microsoft Kinect sensor. This article will focus on hands-on C++ programming tips, while providing sample C++ code indicating how to record and store the Kinect streams (Skeleton, Color and Depth streams) to a specified directory. 

The Kinect Stream Saver application is based on the KinectExplorer-D2D (for C++) sample code and is compatible with SDK 1.7. Modifications to the sample code will allow user to display and store the Kinect stream in real-time at 30 Frames Per Second (FPS). Hence, the recording task would not slow down the processing tasks (data streaming and displaying).

In this article, you can find step-by-step instructions on how to store stream data (color, depth, skeleton) to a dynamic FIFO buffer as the data is received from the Kinect device, and separately write the data from the FIFO buffer to output files as described above. The disk writes are done in a separate
thread to avoid degradation of frame processing speed during data collection.

Please note that all the C++ sample codes explained in this article should be built and run by the Visual Studio.

Prerequisites

As a starting point, this article assumes that you

• are reasonably familiar with C++ libraries, examples, and documentation needed to create applications for Windows,
• can figure out the basics of building and running C++ codes in Visual Studio

Requirements

• Microsoft Visual Studio
• Kinect for Windows SDK
• Source Codes of the application called "Kinect Explorer - D2D C++"

Using the code

Creating a Thread  

One of the main issues with recording the Kinect streams in real time is to avoid degradation of frame processing speed during data collection. The only solution for keeping the frame rate at 30 FPS is to create a new thread to store the stream contents. In this way, there would be one thread to do the processing tasks including initializing the Kinect sensor, annotating data streams and storing the streams to a dynamic buffer; and, another thread to write the Kinect streams to output files in a disk.

In the first step, we can create a thread using a CreateThread function. The new thread must be created within the application main loop. 

The following is a simple example that demonstrates how to create a new thread that executes the defined function, SaveThread. If we are using the Kinect Explorer application, we can create the new thread in the MessageLoop function defined in the "KinectWindow.cpp".  

We also need to Create an event to stop the new thread concordant with creating a new thread.  

WPARAM KinectWindow::MessageLoop()
{
    {...} 
    
    // Create a new thread to save the streams
    e_hSaveThread = CreateThread (NULL, 0, SaveThread, this, 0, NULL );   

    // Create an event to stop the savethread
    e_hStopSaveThread = CreateEventW (nullptr, TRUE, FALSE, nullptr);     
 
    {...}
    
    WaitForSingleObject(e_hSaveThread, INFINITE);
    CloseHandle(e_hSaveThread);
 
    return msg.wParam;
}

void KinectWindow::OnClose(HWND hWnd, WPARAM wParam)
{
    ...
 
    // Stop  thread
    if (INVALID_HANDLE_VALUE != e_hSaveThread)
    {
        SetEvent(e_hStopSaveThread);
    }

 
    ...
}

Here is the code indicating how to define the SaveThread function. The first function returns the handle of the thread and the second function can take care of the storing procedures.

ORD WINAPI KinectWindow::SaveThread(LPVOID lpParam) 
{
    KinectWindow *pthis = (KinectWindow *)lpParam;
    return pthis->SaveThread( );
} 

WORD WINAPI KinectWindow::SaveThread()
{
    bool SaveProcessing = true;
    
        
    while(SaveProcessing)
    {
	// stop event was signalled;
	if ( WAIT_OBJECT_0 == WaitForSingleObject(e_hStopSaveThread,1))
	{
	    SaveProcessing = false;
            break;
	}	
       ...    
     }
 return 0;
}

Once the second thread is created, we can proceed with storing the Kinect streams.

Saving the skeletal stream into a .csv file

Since the application is multithreaded, there should be a FIFO (First in, Fisrt out) buffer for saving the streams. In this way, the Kinect streams will be pushed back to the FIFO buffer by the main thread (ProcessThread function in Kinect Explorer - D2D) and the the second thread (SaveThread function) will remove the last data streams from the FIFO. The removed data streams can be saved into a file as any formats.

In C++, Deque or double-ended queues are sequence containers with dynamic sizes that can be expanded or contracted on both ends (either its front or its back). Here, we will use deque to implement our FIFO buffer for saving the streams.

The following is a simple exmaple that demonstrates how to declare a dynamic FIFO buffer for storing the skeleton streams.

struct    SkeletonStream // Dynamic Buufer to store Skeleton streams
{  
    deque <NUI_SKELETON_DATA> SkeletonJointPosition[NUI_SKELETON_COUNT];
    deque <DWORD>             dwframenumber;
    deque <LARGE_INTEGER>     FrameTime;
};
SkeletonStream              SkeletonBuffer;     

Here, you can see more examples on how to pass NUI_SKELETON_FRAME to a function called BufferSkeletonStream which is responsible for buffering the streams.

The next important step is to ensure that the multithreaded application shares the buffer between without any issue. In other words, the threads needs to be synchronized . A critical section object provides synchronization on using the FIFO buffer between the two threads. In this way, the consuming thread can call EnterCriticalSection function (available in MSDN Kernel32.lib) to request ownership of a critical section. After it is done with the FIFO buffer, it can call the LeaveCrtiticalSection function to release ownership of a critical section. If the critical section object is currently owned by another thread, EnterCriticalSection waits indefinitely for ownership.

Note: You need to add the following code to ProcessSkeleton function defined in "NuiSkeletonStream.cpp".

void NuiSkeletonStream::ProcessSkeleton()
{
    ...
 
    // Set skeleton data to stream viewers
    AssignSkeletonFrameToStreamViewers(&m_skeletonFrame);
 
    // save the Skeleton stream
    NUI_SKELETON_FRAME* nui_skeletonFrame = &m_skeletonFrame;
    BufferSkeletonStream(nui_skeletonFrame);
 
    UpdateTrackedSkeletons();
} 

/// Push back the Skeleton streams into a buffer
void NuiSkeletonStream::BufferSkeletonStream(const NUI_SKELETON_FRAME* pFrame)
{
    nui_skeleton_frame = pFrame;
    
    // Request ownership of the critical section.
    EnterCriticalSection(&CriticalSection_Skeleton);
 
    for (int i = 0 ; i < NUI_SKELETON_COUNT ; i++)
    {
        SkeletonBuffer.SkeletonJointPosition[i].push_back(
           nui_skeleton_frame->SkeletonData[i]);
    }
    SkeletonBuffer.FrameTime.push_back(nui_skeleton_frame->liTimeStamp);
    SkeletonBuffer.dwframenumber.push_back(nui_skeleton_frame->dwFrameNumber);
    
    // Release ownership of the critical section.
    LeaveCriticalSection(&CriticalSection_Skeleton);
}  

Note: Once we define the BufferSkeletonStream function, we need to add the following code to the SaveThread function  having been already defined in "KinectWindow.cpp". The code will store the Skeleton streams including the 20 joint positions into a folder as a .csv file.  

  DWORD WINAPI KinectWindow::SaveThread(LPVOID lpParam) 
{
    KinectWindow *pthis = (KinectWindow *)lpParam;
    return pthis->SaveThread( );
}
 
WORD WINAPI KinectWindow::SaveThread()
{
    bool SaveProcessing = true;
    
        
    while(SaveProcessing)
    {
	// stop event was signalled;
	if ( WAIT_OBJECT_0 == WaitForSingleObject(e_hStopSaveThread,1))
	{
	    SaveProcessing = false;
            break;
	}
           
        // save the skeleton stream    
        SaveSkeletonStream;        
        }
 return 0;
}
/ Save Skeleton streams as .csv files
void KinectWindow:: SaveSkeletonStream
{
    bool EnSave = false;        
 
    NUI_SKELETON_DATA TempSkeletonBuffer[NUI_SKELETON_COUNT];
    DWORD TempframeNumber;
    LARGE_INTEGER TempFrameTime;
 
    // Request ownership of the critical section.
    EnterCriticalSection(&CriticalSection_Skeleton);
 
    if (!SkeletonBuffer.dwframenumber.empty())
    {
        for (int i = 0 ; i < NUI_SKELETON_COUNT ; i++)
        {
            TempSkeletonBuffer[i] = SkeletonBuffer.SkeletonJointPosition[i].front();          
            SkeletonBuffer.SkeletonJointPosition[i].pop_front(); 
            SkeletonBuffer.SkeletonJointPosition[i].shrink_to_fit();
        }
        TempframeNumber = SkeletonBuffer.dwframenumber.front();
        TempFrameTime = SkeletonBuffer.FrameTime.front();
        
        SkeletonBuffer.dwframenumber.pop_front(); 
        SkeletonBuffer.FrameTime.pop_front(); 
        SkeletonBuffer.dwframenumber.shrink_to_fit();
        SkeletonBuffer.FrameTime.shrink_to_fit(); 
        
        EnSave = true;
    }
 
    // Release ownership of the critical section.
    LeaveCriticalSection(&CriticalSection_Skeleton);
 
    if (EnSave)
    {
        if (SkeletonJoint)
        {
           for( int j = 0 ; j < NUI_SKELETON_POSITION_COUNT ; j++ ) {                                           
                 SkeletonJoint << TempSkeletonBuffer[i].SkeletonPositions[j].x<<","<<
                                  TempSkeletonBuffer[i].SkeletonPositions[j].y<<","<<
                                  TempSkeletonBuffer[i].SkeletonPositions[j].z<<","<<
                                  fixed<< 
                                  TempSkeletonBuffer[i].eSkeletonPositionTrackingState[j]<< 
                                  endl;}    
         }                          
         SkeletonTime << fixed <<  TempframeNumber 
                      <<","<< TempFrameTime.QuadPart <<endl;
        
        EnSave = false;
    }
    else if (EnStop_Skel)
    {
        EnStop_Skel = false;
        SkeletonJoint.close();        
        SkeletonTime.close();
        SkeletonBuffer.dwframenumber.clear();
        SkeletonBuffer.FrameTime.clear();
        for( int i = 0 ; i < NUI_SKELETON_COUNT ; i++ )
        {
            SkeletonBuffer.SkeletonJointPosition[i].clear();
        }        
    } 
}

Saving the Color stream into a .bitmap file  

Again, for saving the color streams, we need to declare a dynamic FIFO buffer. The only difference is that the there should be a dynamic vector (or deque) to store the color streams (a vector of 640 by 480) which makes the insertion and deletion image pixels more complicated. A shared_ptr offers the programmers different complexity tradeoffs and should be used accordingly. With using a shared_ptr, there is always a minimum risk of memory leakage as the destructor always make sure that the allocated memory is deleted.

struct ColorStream // Dynamic Buufer to store color streams
{
    deque <shared_ptr<vector <BYTE> > >        ColorImageBuffer;
    deque <DWORD >                    c_width;
    deque <DWORD >                    c_height;    
};
ColorStream                        ColorBuffer;  

Here, you can see more examples on how to pass NuiImageBuffer to a function called BufferColorStream which buffers the color streams.

Note: We need to add the following code to the ProcessColor function defined in NuiColorStream.cpp.

void NuiColorStream::ProcessColor()
{
    HRESULT hr;
    NUI_IMAGE_FRAME imageFrame;
 
    ...
 
    // Make sure we've received valid data
    if (lockedRect.Pitch != 0)
    {
        ...
        
    if (m_pStreamViewer)
        {
            // Set image data to viewer
            m_pStreamViewer->SetImage(&m_imageBuffer);
        }
 
    // save the color stream
    NuiImageBuffer* nui_Color = &m_imageBuffer;
    BufferColorStream(m_pColorStream->nui_Color);
    
    }
 
    // Unlock frame data
    pTexture->UnlockRect(0);
 
ReleaseFrame:
    m_pNuiSensor->NuiImageStreamReleaseFrame(m_hStreamHandle, &imageFrame);
}

void NuiColorStream::BufferColorStream(const NuiImageBuffer* pImageiTimeStamp)
{
    const NuiImageBuffer*    nui_Buffer; // Pointer to Image Buffer
    nui_Buffer = pImage;
    shared_ptr<vector <BYTE>> Buffer(new vector<BYTE>(nui_Buffer->GetBufferSize()));
	copy ( nui_Buffer->GetBuffer(), 
	       nui_Buffer->GetBuffer() + nui_Buffer->GetBufferSize(), 
	       Buffer->begin());	
    
    // Request ownership of the critical section.
    EnterCriticalSection(&CriticalSection_Color);
 
    ColorBuffer.height.push_back(nui_Buffer->GetHeight());
    ColorBuffer.width.push_back(nui_Buffer->GetWidth());    
    ColorBuffer.ImageBuffer.push_back(Buffer);
    
    // Release ownership of the critical section.
    LeaveCriticalSection(&CriticalSection_Color);
}

Note: Once we define the BufferColorStream function, we need to add the following code to the SaveThread function having been already defined in "KinectWindow.cpp". The code will write the Color streams including the Color pixels into a disk as a bitmap format.  

    DWORD WINAPI KinectWindow::SaveThread(LPVOID lpParam) 
{
    KinectWindow *pthis = (KinectWindow *)lpParam;
    return pthis->SaveThread( );
}
 
WORD WINAPI KinectWindow::SaveThread()
{
    bool SaveProcessing = true;
    
        
    while(SaveProcessing)
    {
	// stop event was signalled;
	if ( WAIT_OBJECT_0 == WaitForSingleObject(e_hStopSaveThread,1))
	{
	    SaveProcessing = false;
            break;
	}
    
        // save the skeleton stream    
        SaveSkeletonStream;
 
        // save the color stream
        SaveColorStream;    
        }
 return 0;
}

// Save Color images as Bitmaps
void KinectWindow::SaveColorStream
{
    bool EnSave = false;    
               
    shared_ptr<vector <BYTE>>  TempColorBuffer
			       (new vector<BYTE>(ColorBuffer.ImageBuffer.size()));
    DWORD                TempHeight;
    DWORD                TempWidth;
 
    // Request ownership of the critical section.
    EnterCriticalSection(&CriticalSection_Color);
 
    if (! ColorBuffer.ImageBuffer.empty())
    {            
        TempColorBuffer = ColorBuffer.ImageBuffer.front();
        TempHeight =      ColorBuffer.height.front();
        TempWidth =       ColorBuffer.width.front();
        Pop_FrontColor()
        EnSave = true;
    }        
 
    // Release ownership of the critical section.
    LeaveCriticalSection(&CriticalSection_Color);
 
    if (EnSave)
    {
        WCHAR c_screenshotPath[MAX_PATH];
	StringCchPrintfW(c_screenshotPath,
	                _countof(c_screenshotPath),
	                L"%s\\%ld_Color.bmp",
	                foldername,c_Counter);	
			
	HRESULT hr = SaveBitmapToFile(TempColorBuffer->data(),
				      TempWidth,
				      TempHeight,  
				      32, 
				      c_screenshotPath);            
        c_Counter++;
        EnSave = false;            
    } else if (EnStop_Color) c_Counter = 0;
}

void KinectWindow:: Pop_FrontColor()
{
	ColorBuffer.ImageBuffer.pop_front();			
	ColorBuffer.height.pop_front();			
	ColorBuffer.width.pop_front();				
	ColorBuffer.FrameNumber.pop_front();				
	ColorBuffer.FrameTime.pop_front();

	ColorBuffer.ImageBuffer.shrink_to_fit();			
	ColorBuffer.height.shrink_to_fit();
	ColorBuffer.width.shrink_to_fit();
	ColorBuffer.FrameNumber.shrink_to_fit();
	ColorBuffer.FrameTime.shrink_to_fit();
} 

// Save Bitmap images to a file
HRESULT StreamSaver::SaveBitmapToFile(BYTE* pBitmapBits, 
	                              LONG lWidth, 
				      LONG lHeight, 
				      WORD wBitsPerPixel, 
				      LPCWSTR lpszFilePath)
{ 
    DWORD dwByteCount = lWidth * lHeight * (wBitsPerPixel / 8);
 
    BITMAPINFOHEADER bmpInfoHeader = {0};
 
    bmpInfoHeader.biSize        = sizeof(BITMAPINFOHEADER);  // Size of the header
    bmpInfoHeader.biBitCount    = wBitsPerPixel;             // Bit count
    bmpInfoHeader.biCompression = BI_RGB;                    // Standard RGB
    bmpInfoHeader.biWidth       = lWidth;                    // Width in pixels
    bmpInfoHeader.biHeight      = -lHeight;                  // Height in pixels
    bmpInfoHeader.biPlanes      = 1;                         // Default
    bmpInfoHeader.biSizeImage   = dwByteCount;               // Image size in bytes

    BITMAPFILEHEADER bfh = {0};
 
    bfh.bfType    = 0x4D42;                                           
    bfh.bfOffBits = bmpInfoHeader.biSize + 
		    sizeof(BITMAPFILEHEADER);       // Offset to the start of pixel data 
    bfh.bfSize    = bfh.bfOffBits +  
		    bmpInfoHeader.biSizeImage;      // Size of image + headers

    // Create the file on disk to write to
    HANDLE hFile = CreateFileW(lpszFilePath, 
				GENERIC_WRITE, 
				0, 
				NULL, 
				CREATE_ALWAYS, 
				FILE_ATTRIBUTE_NORMAL, 
				NULL);

    // Return if error opening file
    if (NULL == hFile) 
    {
        return E_ACCESSDENIED;
    }
 
    DWORD dwBytesWritten = 0;
    
    // Write the bitmap file header
    if ( !WriteFile(hFile, &bfh, sizeof(bfh), &dwBytesWritten, NULL) )
    {
        CloseHandle(hFile);
        return E_FAIL;
    }
    
    // Write the bitmap info header
    if ( !WriteFile(hFile, &bmpInfoHeader, sizeof(bmpInfoHeader), &dwBytesWritten, NULL) )
    {
        CloseHandle(hFile);
        return E_FAIL;
    }
    
    // Write the Color Data
    if ( !WriteFile(hFile, pBitmapBits, bmpInfoHeader.biSizeImage, &dwBytesWritten, NULL) )
    {
        CloseHandle(hFile);
        return E_FAIL;
    }   
    
    // Close the file
    CloseHandle(hFile);
  return S_OK;
}

Saving the Depth stream is pretty much the same as Color stream.

Points of Interest

• Synchronization between the threads using critical section object
• 2D dynamic vector

Terms of Use 

If your research benefits from the Kinect Stream Saver application provided by IATSL,
please consider the following Terms of Use:

- please cite the followign paper in any publications about your work:

    * Elham Dolatabadi, Babak Taati, Gemma S. Parra-Dominguez, Alex Mihailidis,
    “A markerless motion tracking approach to understand changes in gait and balance: A Case Study”,
    the Rehabilitation Engineering and Assistive Technology Society of North America (RESNA 2013 Annual   Conference).  Student Scientific Paper Competition Winner

- please acknowledge this application in any publications about your work. Thank you for your support.

    Acknowledgement Text Examples:
    We would like to acknowledge Intelligent Assisstive Technology and Systems Lab for the application that   facilitated thiS research, including the Kinect Stream Saver application developed by Elham Dolatabadi.

SourceCodes:

You can also download the source codes for Kinect Stream Saver Application from the following Url:

http://kinectstreamsaver.codeplex.com/

History   

• Sep. 11, 2013: First version.
• Sep. 12, 2013: Second Version
• Sep. 30, 2013: Third Version
• Oct. 12, 2013: Fourth Version