Tips and Tricks to use Unity with the Intel RealSense SDK 2014

Intel®Developer Zone offers tools and how-to information for cross-platform app development, platform and technology information, code samples, and peer expertise to help developers innovate and succeed. Join our communities for the Internet of Things, Android*, Intel® RealSense™ Technology and Windows* to download tools, access dev kits, share ideas with like-minded developers, and participate in hackathons, contests, roadshows, and local events.

Important Note on Unity Versions:

• Unity 4.x is 32-bit based. You can use Unity Pro or the free version of Unity as long as you use the signed libraries in the Intel RealSense SDK R2 release (v4.)
• Unity 5 is 64-bit based. You must copy over the 64 bit libraries ( libpxccpp2c.dll and libpxccpp2c.dll.signtaure) to your project.

Documentation that may be helpful.

• Intel provided: The RSSDK Unity Toolkit guide is installed at C:\Program Files (x86)\Intel\RSSDK\doc\PDF\sdktoolkit.pdf
  • Unity_Getting_Started
  • Face Tracking tutorial includes face location, pose, landmark, and expression detection, face recognition, alerts and code samples
  • Hand/Joint/Blob Tracking tutorial covers gesture, hand and joint tracking, blob and contour extraction, data smoothing & code samples
  • Raw Stream Capture tutorial: Catching and rendering color and depth streams with code samples
  • Unofficial - Intel developer Amit M has Unity Toolkit Demo and RS/Unity RFBDepth Alignment videos
• Marty G of Sambiglyon has shared these guides:
  • Setting up the SDK files in Unity and programming objects in the 'Inspector' panel to respond to camera inputs.
  • Converting PXC scripts for use in Unity script files
  • Prototyping RealSense project controls with physical controllers using Unity's built-in "Input Manager".
  • Simple Hand Tracking /Object moving guide.

It should be noted that one can use the Unity DLL without using the Unity Toolkit. If you post questions or issues, please distinguish between the two.

Some Minimum Requirements:

• With Gold R1 ONLY Windows 8.1 is supported. The only camera supported is Creative RealSense F200 USER facing camera, which must be in a true (blue) USB 3.0 direct port (not hubbed).
• Need to use Unity Pro because the free version of Unity doesn't support C++ plugins. There may exist workarounds to this, but they are not endorsed by Intel. Even as part of the contest, Intel is NOT providing any Pro licenses. But note that RSSDK 2014 can use C# and comes packaged with a lot of pre-built "extensions" for interfacing the camera with Unity. (thanks Marty!)
• NOTE: Gold R1 was compiled against Unity Pro 4.1 and was also tested on v4.5 It was not validated against 4.6 Beta.

Now some tips/tricks:

• Full Screen: Make the App go Full Screen in windowed/unwindowed modes, whatever the display resolution.
  • ​Note Intel is suggesting a minimum resolution of 1368x768
  • Before compiling, in Player’s Settings, make sure the "Use Direct3D 11" box is checked.
    • If only D3D9 is available in Player Settings, Uncheck "Default is Native Resolution" or "Default is Full Screen" and set default screen width and height to 1368x768
  • Set it to use ONLY specific aspect ratios: 16:9 and 16:10
  • Result: Based on the "Windowed" Checkbox setting on the "Resolution Selection Screen"
    • If checked, App will start windowed but can be resized/maximized
    • If unchecked, App will always start at full screen.
  • Note you can check resolution // Cap screen resolution, an example
    if ( Screen.currentResolution.width > 1366 || Screen.currentResolution.height > 768 ) Screen.SetResolution(1366, 768, true);

• Performance:

  • The AcquireFrame desired setting in the Unity Update() function should be (false,0) to avoid impacting performance. (In Unity, it is a blocking call which locks the frame rate to RS and is used to wait until any frame data is available)
    if (sm.AcquireFrame (false) != pxceStatus.PXCM_STATUS_NO_ERROR)

  • While 2 hands tracking may be in the same thread, it is suggested you create separate thread for face and voice or any multi-modal use.

• Smoothing in Unity: as in the SDK, 4 types of smoothing are available. Here's SUGGESTIONS for their use in Unity. Top 2 most preferred.
  • Spring: Time based but has a linear effect on smoothed data (Marty suggests smoothing factor=1 with TrackingAction script.
  • Quadratic: Time based smoothing which acts as stabilizer and smoother.
  • Weighted: Allows averaging the data according to any set of weights (Gaussian, Uniform, etc.) . Good for hands
  • Stabilize smoothing: good for UX as it keep existing location unless a large movement is detected
• Voice/Speech in Unity. Unity can do Commands, and Speech to text (can use the C# wrapper sample from framework folder) Speech synthesis may be doable the same way (use sample from C#).
• 3D Segmentation - Beards. Currently long beards may cause very slow detection as the beard absorbs a lot of IR and may separate the head from body, confusing detection. The issue is being worked.
• Face Recognition Registration.
  • In Gold R1, 2D face recognition is only using ID#100.
  • Gold changed from Beta and now waits 10 consecutive frames before confirming recognition (not settable since used to lessen false positives.

• Hand Tracking if out of FOV: Use the "HandLost" rule in the Unity Toolkit. (RSUnityToolkit/Internals/Rules). Checks for number of hands detected (SenseToolkitManager.Instance.HandDataOutput.QueryNumberOfHands())

• If you have issues with UnityEngine.Texture2D PXCMImage.ImageData.ToTexture2D(Int32 plane, Int32 width, Int32 height)),
  use ToTexture2D that specifies a Texture2D as a parameter.

• From Marty G: If you find that your objects are moving jerkily and unpredictably or tearing themselves away from objects that they are attached to as child objects, go into the Inspector panel settings for a particular TrackingAction script and set all 'Virtual World Box' and 'Real World Box' values to zero.

• From Marty G: If you want Unity to run a script inside an object other than MovingObject, edit the script inside TriggerObject to give the name of that object, the name of the script and the function type at the start of that script (Start(), Update(), etc).. For more info see this forum post.

Looking forward to hearing other tips and tricks! Will hopefully update this weekly!