Intel® RealSense™ Code Sample "Sketch"

Intel®Developer Zone offers tools and how-to information for cross-platform app development, platform and technology information, code samples, and peer expertise to help developers innovate and succeed. Join our communities for the Internet of Things, Android*, Intel® RealSense™ Technology and Windows* to download tools, access dev kits, share ideas with like-minded developers, and participate in hackathons, contests, roadshows, and local events.

Abstract

This code sample uses the Intel® RealSense™ SDK for Windows* to create a simple virtual drawing app named Sketch. This Windows desktop app, developed in C#/WPF, demonstrates a number of hand tracking and gesture recognition capabilities available in the Intel RealSense SDK:

• Retrieval of hand joint x-y-z (world) coordinates
• Selectively capturing gestures data
• Getting alert status information (i.e., hand detection, calibration, and boundary status)

(Note: The full functionality of this sample app requires a front-facing 3D camera.)

WATCH the short video overview of Sketch here.

Introduction to Sketch

Sketch is a simple drawing app that allows the user to simulate drawing on a canvas by using gestures and hand motions. Figure 1 shows the Sketch user interface (developed in WPF/XAML).

Three gestures (depicted on the screen with their accompanying action) are enabled for interacting with the virtual canvas:

• Finger Pinch ("Draw") – causes the cursor to become solid and draw a line on the canvas. The position of the cursor on the canvas is controlled by the x and y coordinates of the user’s middle fingertip. The line thickness is controlled by the z-axis of the user’s middle fingertip (i.e., pulling away from the camera causes the line to become thinner, as if decreasing pressure on the pen or brush).

• Spread Fingers ("Hover") – deactivates the pen, making the cursor become an empty circle. Hovering allows the pen to be moved to other sections of the canvas without drawing a line. It also allows the user to select colors from the right-hand palette by simply hovering over them.

• Wave ("Erase") – Wipes the drawing canvas clean, making it ready for a new drawing.

Details

The Sketch app simulates drawing on a canvas when the user makes the "two_fingers_pinch_open" gesture. This gesture was selected because it approximates the hand pose a user would make when holding a pen or brush. Figure 2 demonstrates this gesture.

Pen placement and stroke thickness are determined by tracking the tip of the user’s middle finger, which may seem unintuitive considering the finger-pinch is the enabling gesture for drawing on the canvas. The reason for tracking the middle finger is to avoid occlusions that can occur when the thumb and index finger are pressed together. Tracking the middle finger instead of the index finger or thumb results in better performance.

The Sketch app also demonstrates how to acquire and display hand alert status information (in this case hand detection, calibration, and border exceptions). Providing this type of feedback, in one form or another, assists users in positioning their hands correctly in front of the camera. Although the presentation of this information is simplistic in this sample app, developers are encouraged to provide similar cues to their users to improve the overall user experience.

Check It Out

To experiment with this app and learn more about how it works, please download it here.

About Intel® RealSense™ Technology

To get started and learn more about the Intel RealSense SDK for Windows, go to https://software.intel.com/en-us/realsense/intel-realsense-sdk-for-windows.