Click here to Skip to main content
15,393,955 members
Articles / Desktop Programming / WPF
Article
Posted 28 Jan 2019

Stats

7.8K views
165 downloads
2 bookmarked

Control NAO Using Kinect

Rate me:
Please Sign up or sign in to vote.
4.83/5 (5 votes)
28 Jan 2019CPOL3 min read
How to Control NAOqi-Based Robots like NAO, Pepper using Kinect

Image 1

Introduction

This article shows how to control NAOqi Robots like (NAO, Pepper, Romeo) using Kinect device using NAO-Kinect.

Background

My dream is to control robots. Part of it is to control robots using Kinect and here is how I made it.

I developed a program before about controlling Servo using Kinect and Arduino in CodeProject. Also Control Arm Robot using Kinect in YouTube link. Now I upgraded it when I got my own NAO robot. I hope you like it and it may be useful in your programs.

What this Project Does

  1. Catches joints position of joints for the person who stands in front of Kinect
  2. Displays it in numbers and graphics on Windows Forms Application
  3. Passes the Kinect Joints values from Windows to NAOqi Robot environment (Python)
  4. Applies the moves on NAOqi-based robot like: (NAO, Pepper or romeo)

Image 2

Prerequisites

To implement this project, you have to have some software and hardware:

Hardware

  1. Kinect (V2 Recommended)
  2. Windows Computer with USB3 Port
  3. NAOqi-based Robot (optional)

Software

  1. Kinect SDK

    From Aldebaran Website login/Register and download:

  2. NAOqi SDK: "pynaoqi-2.1.4.13.win32.exe" (Recommended version)
  3. Choregraphe: "choregraphe-suite-2.1.4.13-win32-setup.exe"(Recommended version)

Using the Code

The code is divided into four parts and rules.

  1. Main
    • Type: Windows Application
    • Rule: User Interface
    • Description: Project which displays Kinect Joints values and Robot data and interacts with user actions
  2. dllKinectCatcher
    • Type: Windows DLL
    • Rule: Kinect - Windows Interface
    • Description: Project which catches Required Events and live data values from Kinect
  3. DllNAO.netV2
    • Type: Windows DLL
    • Rule: NAOqi - Windows Interface
    • Description: Control NAOqi-Based Robots using C#, it is an improvement for the original NAO.NET project listed down
  4. Baku.libqiDotNet
    • Type: Windows DLL
    • Rule: dotnet NAoqi Interface
    • Description: Unofficial .NET wrapper library for "qi Framework", the messaging library created by Aldebaran Robotics

1. Main Project

The main project is a Windows Forms Application. (Introduction form and Control Form)

It is just a simple Windows Forms Application with many textBoxes to hold the joints coordinates (X,Y,Z) for every one. Live data display.

Main Project Points of Interest

The best interesting point about that is how all text boxes update at the same time with no hanging... it uses threads and working background to do that .

Image 3

Sample Code

This example for how to update the hand's (left and right) values Async using BeginInvoke:

C#
//Right Hand
txtKinRHandX.BeginInvoke
  (new Action(() => txtKinRHandX.Text = e.hand.Get(utilities.Side.Right).x.ToString()));
txtKinRHandY.BeginInvoke
  (new Action(() => txtKinRHandY.Text = e.hand.Get(utilities.Side.Right).y.ToString()));
txtKinRHandZ.BeginInvoke
  (new Action(() => txtKinRHandZ.Text = e.hand.Get(utilities.Side.Right).z.ToString()));

//Left Hand
txtKinLHandX.BeginInvoke
  (new Action(() => txtKinLHandX.Text = e.hand.Get(utilities.Side.Left).x.ToString()));
txtKinLHandY.BeginInvoke
  (new Action(() => txtKinLHandY.Text = e.hand.Get(utilities.Side.Left).y.ToString()));
txtKinLHandZ.BeginInvoke
  (new Action(() => txtKinLHandZ.Text = e.hand.Get(utilities.Side.Left).z.ToString()));

2. dllKinect

dllKinect is a Windows DLL project which be allowed to catch the joints data from Kinect and convert it to suitable data to read.

Why DLL Kinect?

Microsoft provides Kinect SDK for control Kinect, but this SDK provides only WPF platform but not Windows Application.

To make it easy to catch data into Windows Forms application, I had to develop this application which means that you can cancel this DLL project if you will develop your application directly using WPF.

Example Code

Note: This code is just an example about how to catch the joints.

C#
public  KinectCatcher.utilities.Position Get(KinectCatcher.utilities.Side hand)
{
    KinectCatcher.utilities.Position position = new utilities.Position();
    switch (hand)
    {
        case KinectCatcher.utilities.Side.Right:
            position = RightHandPosition;
           return position;
        case KinectCatcher.utilities.Side.Left:
           position = LeftHandPosition;
             return position;  
        default:
             return new utilities.Position();
    }
}

3. DllNAO.netV2

Second main part of the solution is NAO.NET Project, which allows you to catch and convert the joints data, from Windows to NAOqi movement data which can be applied on the robot.

In the past, I posted another article about how to execute Python from Windows application and as an example, I can control NAO robot using it, you can check it here: NAO.NET V1. But it cannot fit in this case because it is generic, slow and there are many other problems, so I had to make something which uses another technique like messaging, and developed another version named NAO.NETV2.

Main Code

This is the main code which explains how NAO.NETv2 works. It can control any NAOqi robot using windows, which is amazing and has enough to be controlled.

C#
try
{
    if (!session.IsConnected)
    {
        result.ResultStatus = ACall.structAcallResult.Result.failure;
        result.dateTime = DateTime.Now;
        result.Message = "Not Connected";
        return result;
    }
    var nAOQiService = session.GetService(NAOqiParameters.Service.ToString("F"));
    switch (NAOqiParameters.Service)
    {
        case NAOqiServices.ALMemory:
            break;
        case NAOqiServices.ALTextToSpeech:
        
            result.NAOqiObject = 
                nAOQiService[NAOqiParameters.method].Call(NAOqiParameters.SingleParamValue);
            result.Message = "called ALTextToSpeech Successfully";
            
            break;
        #region Motion Service
        case NAOqiServices.ALMotion:
            
            QiList< QiString> names
                        = QiList.Create(NAOqiParameters.names);
            QiList< QiList<QiDouble>> angleLists = 
                QiList.Create< QiList< QiDouble>>(new QiList<QiDouble>[] 
                { QiList.Create(NAOqiParameters.AngleList[0]) });
            
            for ( int i = 1 ; i < NAOqiParameters.AngleList.Count(); i++ ) 
            {
                angleLists.QiValue.AddElement
                        (QiList.Create(NAOqiParameters.AngleList[i]).QiValue);
            }
            QiList< QiList<QiDouble>> timeLists = 
                QiList.Create< QiList<QiDouble>>(new QiList<QiDouble>[] 
                { QiList.Create(NAOqiParameters.timeLists[0]) });
            for ( int i = 1; i < NAOqiParameters.timeLists.Count() ; i++ )
            {
                timeLists.QiValue.AddElement
                     (QiList.Create(NAOqiParameters.timeLists[i]).QiValue);
            }
           QiBool isAbsolute;
            if (NAOqiParameters.isAbsolute)
            {
                isAbsolute = new QiBool(true);
            }
            else
            {
                isAbsolute = new QiBool(false);
            }
            result.NAOqiObject = nAOQiService["angleInterpolation"].Call
                         (names, angleLists, timeLists, isAbsolute);
            result.Message = "successfully called ALMotioin Service";
            
            break;
        #endregion Motion Service
        default:
            break;
    }
    
    result.dateTime = DateTime.Now;
 
    result.ResultStatus = ACall.structAcallResult.Result.Sucess;
    session.Close();
    session.Destroy();
    return result;
}

History

Of course, I want to have more control and integration of this project, I may update more code and control options later on gitHub as well. For more projects or help, you can contact me by leaving a comment below.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Tareq_Gamal
Team Leader ArabicRobotics.com
Egypt Egypt
Tareq Gamal El-din Mohammed,
---------
Website:
www.ArabicRobotics.com

---------

Graduated from Modern Academy for Computer science and Information Technology. Egypt,
Then flow Microsoft development track Certificates:
MCAD.NET (Microsoft Certified Application Developer)
MCSD.NET (Microsoft Certified Solution Developer)
Microsoft SharePoint Administration, Configuration and Development.

Robotics fields was a Hobby since 2002,
started to develop some applications for "Robosapien", "RoboSapienV2", "RS Media", RoboMe and WowWee Rovio. from WowWee company,

Started working with robots as a professional way at 2014
By using "NAOqi" Robotics from Aldebaran.

By developing some applications and libraries like :
NAO.NET.
https://www.youtube.com/watch?v=oOyy-2XyT-c

OpenCV with NAO Robot:

- NAORobot Vision using OpenCV -TotaRobot P1
https://www.youtube.com/watch?v=MUcj8463x08

- NAO Robot Vision using OpenCV - P2
https://www.youtube.com/watch?v=93k1usaS-QM

NAO Alarm Clock :
https://www.youtube.com/watch?v=djLlMeGLqOU
-----------------------------

also Robotic Arm Project:


Other Projects Developed by Tareq Gamal El-din Mohammed :

Developed and posted some applications in Code Project web site like :

- Control your Distributed Application using Windows and Web Service
http://www.codeproject.com/Articles/101895/Control-your-Distributed-Application-using-Windows


- Quick and dirty directory copy
http://www.codeproject.com/Articles/12745/Quick-and-dirty-directory-copy

- Program Execute Timer(From the Web)
http://www.codeproject.com/Articles/12743/Program-Executer-Timer

Comments and Discussions

 
QuestionKinect Coordinate System to Nao Coordinate System Tranformation Pin
Mrityunjay Jha23-Dec-20 17:55
MemberMrityunjay Jha23-Dec-20 17:55 
AnswerRe: Kinect Coordinate System to Nao Coordinate System Tranformation Pin
Tareq_Gamal23-Dec-20 22:21
MemberTareq_Gamal23-Dec-20 22:21 
GeneralRe: Kinect Coordinate System to Nao Coordinate System Tranformation Pin
Mrityunjay Jha25-Dec-20 20:25
MemberMrityunjay Jha25-Dec-20 20:25 
GeneralRe: Kinect Coordinate System to Nao Coordinate System Tranformation Pin
Tareq_Gamal28-Dec-20 5:03
MemberTareq_Gamal28-Dec-20 5:03 
AnswerRe: Kinect Coordinate System to Nao Coordinate System Tranformation Pin
Tareq_Gamal11-Jan-21 18:36
MemberTareq_Gamal11-Jan-21 18:36 
PraiseGreat Pin
Member 1291127028-Jan-19 21:30
MemberMember 1291127028-Jan-19 21:30 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.