Kinect Recording Interface

• Download KinectRecordingTestFinal-noexe.zip - 823.7 KB
• Download KinectRecordingTestFinal.zip - 1.4 MB 

Introduction

Since I have started to develop desktop applications using Kinect, I had to test my code by standing-up in front of the sensor. At start it was funny, but as soon as I had to do it tens of times within a day it was starting to be annoying. So, I decided to create an application that could record and playback any moves!

Background 

In this article it is assumed that you have basic knowledge of using Kinect sensor & Kinect SDK (not OpenNI) and you are familiar with C#. The application is written for Kinect Beta 2 SDK driver version 1.0.0.45.

If your are not familiar with Kinect, take a look at a great series of tutorials! 

Prerequisites

• Kinect Sensor
• Kinect Beta 2 SDK
• .Net Framework 4.0
• Visual Studio
• Json.Net (Included in project)  

  Using KinectRecorderInterface 

Its use is intended to be as simple as possible!

Declaring Creating the class as most of the classes.  

 KinectRecorderInterface recorder = new KinectRecorderInterface();

Initializing Open the Kinect sensor through the recorder and select an appropriate ImageResolution (DepthImage can be up 640x480).  Notice: Only one application at a time can use the Kinect!! 

Use To open only the DepthStream :

recorder.InitializeSensor(ImageResolution.Resolution320x240);

Use To open both of DepthStream & VideoStream :

recorder.InitializeSensor(ImageResolution.Resolution320x240, ImageResolution.Resolution640x480);

Whole Initialization code  :

private KinectRecorderInterface recorder;
public void InitializeKinectRecorder()
{
    recorder = new KinectRecorderInterface();
 
    //Initialize Sensor through recorder
    recorder.InitializeSensor(ImageResolution.Resolution640x480, ImageResolution.Resolution640x480);
 
    //Set all the event handlers
    recorder.PlaybackSkeletonDataReady += new KinectRecorderInterface.PlaybackFrameHandler(recorder_PlaybackSkeletonDataReady);
    recorder.SkeletonFrameReady += new KinectRecorderInterface.SkeletonFrameHandler(recorder_SkeletonFrameReady);
    recorder.ImageFrameReady += new KinectRecorderInterface.ImageFrameHandler(recorder_ImageFrameReady);
    recorder.DepthFrameReady += new KinectRecorderInterface.DepthFrameHandler(recorder_DepthFrameReady);
} 

That's all!

Assigning event handlers Now you can create proper event handlers to take data from Kinect. Notice: EventHandlers and the EventArgs are the same as using the Kinect SDK, except of the playback!! 

recorder.SkeletonFrameReady += new KinectRecorderInterface.SkeletonFrameHandler(recorder_SkeletonFrameReady);
recorder.ImageFrameReady += new KinectRecorderInterface.ImageFrameHandler(recorder_ImageFrameReady);
recorder.DepthFrameReady += new KinectRecorderInterface.DepthFrameHandler(recorder_DepthFrameReady);

Playback EventHandler is pretty much the same, but it differentiates of its EventArgs (See later on 'SkeletonData VS SerializableSkeletonData')

recorder.PlaybackSkeletonDataReady += new KinectRecorderInterface.PlaybackFrameHandler(recorder_PlaybackSkeletonDataReady); 

Now, you can retrieve Kinect's data as usually (in realtime) from SkeletonTracking, DepthStream and VideoStream, but there is no recording or playback at the time.

Starting / Stop Recording Anytime you can start StartRecording() method, in order to start capturing only available (Tracked) SkeletonData. Use StopRecording() method, in order to stop capturing and save any stored SkeletonData. Saving data are stored by default in "Recordings"  folder, unless you change it from RecordDirectory. Notice: If recording has no SkeletonData, saving will be aborted.

recorder.RecordDirectory = "./StoreData"; 

recorder.StartRecording();
//Wait as much you want
recorder.StopRecording();

Now, by default, a new folder "Recordings" has been created along with the project's executable; and a file called "recording_00x.dat" has been created within the folder.

# Start / Stop Playback

After you have recorded some moves you are able to playback them any time you want to. Use StartPlayback() to start playback of the last available recording. Also you can use of StartPlayback(recording_xxx.dat), in order to playback a recording of your choice. Notice: Each time the application is starting/recording/saving an updated catalog (List<string>) of the recordings is available through the ExistingRecords attribute, to let you choose any recording.

//Play the last available recording
recorder.StartPlayback();  

//Play a record of your choice
List<string> records = recorder.ExistingRecords;
recorder.StartPlayback(records[0]);

SkeletonData VS SerializableSkeletonData

As soon as I reached the point that I had to store the data, I realized that I couldn't serialize SkeletonData because were not marked as Serializable. For that reason I implemented Json.Net which worked great! But, unfortunately when i tried to deserialize and cast (see below) the stored data, it failed again as SkeletonData does not have any public constructor (its constructor it is set to internal)!! A better solution is welcomed!!!

<Code Not Working>

//This code gave me an error because there is no public constructor for SkeletonData
var json = System.IO.File.ReadAllText(filepath);
var skeletonData = Newtonsoft.Json.JsonConvert.DeserializeObject<List<SkeletonData>>(json);

</Code Not Working>

Finally, I didn't come up with any other solution rather than create a new class almost identical to SkeletonData as shown below.

At the only point that you will get SerializableSkeletonData, will be at the time that you will get SkeletonDataArgs from PlaybackFrameHandler. SkeletonDataArgs holds SerializableSkeletonData.

Using SerializableSkeletonData  It's almost the same process as using SkeletonData!

For example let's say that we have the following code for SkeletonData 

private void recorder_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs skeletonFrameArgs)
{
    //Get the first skeleton from SkeletonFrame
    SkeletonData skeleton = skeletonFrameArgs.SkeletonFrame.Skeletons[0];
 
    if (skeleton != null)
    {
        //Do something with your data
        setEclipsePos(HeadEclipse, skeleton.Joints[JointID.Head]);
        setEclipsePos(LeftHandEclipse, skeleton.Joints[JointID.HandLeft]);
        setEclipsePos(RightHandEclipse, skeleton.Joints[JointID.HandRight]);
    }
}

The same function through Playback using SerializableSkeletonData

private void recorder_PlaybackSkeletonDataReady(object sender, SkeletonDataArgs skeletonDataArgs)
{
    SerializableSkeletonData skeleton = skeletonDataArgs.serializableSkeletonData;
 
    if (skeleton != null)
    {
        //Do something with your data
        setEclipsePos(HeadEclipse, skeleton.Joints[(int)JointID.Head]);
        setEclipsePos(LeftHandEclipse, skeleton.Joints[(int)JointID.HandLeft]);
        setEclipsePos(RightHandEclipse, skeleton.Joints[(int)JointID.HandRight]);
    }
}

Extra Properties  

KinectRecorderInterface has a PlayPause methods while you are in recording or playback mode.

Use PlayPauseRecording() or PlayPausePlayback() each time you want to pause your recording or playback. Use again the same method to continue..!

//Pause or Resume recording
recorder.PlayPauseRecording(); 

//Pause or Resume playback
recorder.PlayPausePlayback();

Use Status to check or see the current status of the recording interface!

public enum KinectRecorderStatus
{
        Idle = 0,
        Initializing = 1,
        Recording = 2,
        Playing = 3,
        Saving = 4,
        Loading = 5,
        NotConnected = 6,
        PlayingPaused = 7,
        RecordingPaused = 8
}
//Print the current status
statusLabel.Text = recorder.Status.ToString();
//Check 'Status'
if (recorder.Status == KinectRecorderStatus.Playing)
{
    //Do something
}

Application has already HotKey identification running on background! You can set, up to 4 HotKeys to handle: 

• Start / Stop Recording : RecordHotKey
• Start / Stop Playback   : PlaybackHotKey
• Pause / Resume Recording : PauseRecordingHotKey
• Pause / Resume Playback   : PausePlaybackHotKey

It is not necessary to enable HotKeys, but you have to assign them a Key value.

//Set 'F9' as HotKey for recording
recorder.RecordHotKey = System.Windows.Forms.Keys.F9;

Application's Class Diagram

Including only the public attributes.