Click here to Skip to main content
Click here to Skip to main content
Technical Blog

Tagged as

Kinect and HTML5 using WebSockets and Canvas

, 10 Dec 2013 CPOL
Rate this:
Please Sign up or sign in to vote.
Kinect and HTML5 using WebSockets and Canvas

Update (Dec 10, 2013): Want to learn more about WebSockets? Consider my new book, Getting started with HTML5 WebSocket Programming.

Kinect defined Natural User Interaction. HTML5 redefined the Web. Currently, there are various tutorials describing how to interact with a Kinect sensor using Windows Forms or WPF for the user interface. But what about using a web interface for handling Kinect data? Trying to combine those two hot, cutting-edge technologies, I came up with a pretty and open-source solution, which I am going to describe in this blog post.

I am going to use the official Kinect SDK for my demo, but the same principles apply to OpenNI SDK, too.

Prerequisites

Results

The project consists of two sub-projects: A server-side application which uses Kinect SDK and a client-side web page displaying the skeleton joints on an HTML5 canvas.

Client application:

Server application:

Tutorial

Here is, step by step, a way to achieve the above functionality:

Step 1: Server application

The server application's job is straightforward: Detect the users' joints, pack the data and send them to the clients using web sockets.

In order to detect the joints' coordinates, we need to add a reference to our preferred Kinect SDK and handle the skeleton events. I recommend Microsoft SDK over OpenNI because it's far less complicated:

static void Nui_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
    List users = new List();

    foreach (var user in e.SkeletonFrame.Skeletons)
    {
        if (user.TrackingState == SkeletonTrackingState.Tracked)
        {
            users.Add(user);
        }
    }

    if (users.Count > 0)
    {
        string json = users.Serialize();

        foreach (var socket in _sockets)
        {
            socket.Send(json);
        }
    }
}

In order to send the data to the subscribed clients, we need to "pack" the joints' coordinates in a way that the clients will be able to understand and process. I decided to encode the coordinates in JSON format, as JSON is a lightweight and easy-to-understand way of transmitting data through the web. You can find the JSON-encoder class in the source code files.

Considering the data transmission, I highly recommend the use of Fleck. Fleck, based on Nugget, is a C# web socket library that does what it says with minimum configuration effort: It broadcasts the desired data to the subscribed clients using lightweight web sockets. Here is how you can initialize a web socket server:

_sockets = new List();

var server = new WebSocketServer("ws://localhost:8181");

server.Start(socket =>
{
    socket.OnOpen = () =>
    {
        Console.WriteLine("Connected to " + socket.ConnectionInfo.ClientIpAddress);
        _sockets.Add(socket);
    };
    socket.OnClose = () =>
    {
        Console.WriteLine("Disconnected from " + socket.ConnectionInfo.ClientIpAddress);
        _sockets.Remove(socket);
    };
    socket.OnMessage = message =>
    {
        Console.WriteLine(message);
    };
});

After collecting, packing and transmitting the data, clients can now consume and process them accordingly.

Step 2: Client application

Time for HTML5 bits! We have the server set up, so let's add the web page clients. HTML5 spec recommends web socket support, currently implemented by Internet Explorer 10, Google Chrome and Mozilla Firefox. Web sockets are great for direct message communication between server and clients.

All we need is some JavaScript for receiving the server data and a canvas for drawing the points. Here is the JavaScript event handler for getting the data and drawing the joints to a <canvas> element:

socket.onmessage = function (evt) {
    status.innerHTML = "Kinect data received.";

    // Get the data in JSON format.
    var jsonObject = eval('(' + evt.data + ')');

    context.clearRect(0, 0, canvas.width, canvas.height);
    context.fillStyle = "#FF0000";
    context.beginPath();

    // Display the skeleton joints.
    for (var i = 0; i < jsonObject.skeletons.length; i++) {
        for (var j = 0; j < jsonObject.skeletons[ i ].joints.length; j++) {
            var joint = jsonObject.skeletons[ i ].joints[ j ];

            // Draw!!!
            context.arc(parseFloat(joint.x), parseFloat(joint.y), 
			10, 0, Math.PI * 2, true);
        }
    }

    context.closePath();
    context.fill();
};

Step 3: Mixin it up!

Now run the server application and then open the HTML file in Internet Explorer 10 or Google Chrome. Stand in front of a Kinect sensor and may the force be with you.

FAQ

  • Do I really need a server application? Yes. You can't run the Kinect SDK directly from your browser. JavaScript cannot access your hardware. The server app is nothing but a Visual Studio Console project!
  • Do I need to pack my data to JSON? No. You can use any format you'd like. JSON is a common format for transmitting data, especially when file size matters. You could use XML or any other custom format, but JSON is lightweight and needs no custom parser. After all, JSON is a subset of JavaScript, so JavaScript can handle it directly.
  • Why not using WCF instead of web sockets? Because it's tougher. Using web sockets, messages can be sent after a server event occurs. And it's a lot easier!

Downloads

You will definitely need to download source code and binaries in order to develop your own HTML5 Kinect applications. Enjoy!

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Vangos Pterneas
Product Manager LightBuzz
United Kingdom United Kingdom
Vangos Pterneas is a Microsoft Most Valuable Professional in the Kinect technology. He helps companies from all over the world grow their revenue by creating profitable software products. Vangos is the owner of LightBuzz Software agency and author of two technical books.
Follow on   Twitter   Google+   LinkedIn

Comments and Discussions

 
Questionmissing microsoft.research.kinect.nui ! Pinmembernaeem aleahmadi8-Jun-12 3:53 
AnswerRe: missing microsoft.research.kinect.nui ! PinmemberVangos Pterneas11-Jun-12 16:10 
GeneralInteresting usage of Fleck PinmemberJason Staten17-Jan-12 21:28 
QuestionWell done PinmemberMember 45654337-Jan-12 7:38 
AnswerRe: Well done PinmemberVangos Pterneas9-Jan-12 3:30 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web01 | 2.8.1411028.1 | Last Updated 10 Dec 2013
Article Copyright 2012 by Vangos Pterneas
Everything else Copyright © CodeProject, 1999-2014
Layout: fixed | fluid