Using Microsoft Kinect with myRIO (update)

[xkingkiller]

This guide will describe how to use Kinect on myRIO through the USB port. This involves setting up libfreenect libraries on the myRIO, and testing it with an example project. This library only contains basic APIs to read raw RGB image and depth image from Kinect. Other functions such as audio, 3D skeleton abstraction are not included.

Instructions:

A. Download the attached 2 zip files.

     The "KinectLib.zip" contains libfreenect library files and the "Kinect Project.zip" contains the example Kinect VIs to use the library to read data from Kinect.

B. Setting up libfreenect Library.

     Requirements:

     In order to complete this part of the tutorial you will need software to SSH into your myRIO. This example uses PuTTY. You will also need to use SFTP to transfer files to the myRIO. This example uses FileZilla.

     Steps:

     1. Connect myRIO through its device USB port to your PC.

     2. Enable SSH. Go into NI MAX and find your myRIO under Remote Systems item in the left pane. Browse to your myRIO. In the System Settings tab of the device, under Startup Settings, there should be several checkboxes. Enable the box to the right of “Enable Secure Shell Server(sshd)” if it is not already. Hit Save.

     3. Connect to the myRIO with SFTP. Username is 'admin' and Password is left blank.

     4. Copy over libfreenect.so.0.1.2, libfreenect_sync.so.0.1.2 and libkinect_wrapper.so in /user/lib folder on your myRIO. And copy liblibusb.so.1.0.8 to /usr/local/lib

     5. Open an SSH connection with the myRIO. User is admin and password is left blank.

    

     6. Create symbol link to these files through the following commands in your PuTTY.   

     =============================================================

     #ln -s /usr/lib/libfreenect.so.0.1.2 /usr/lib/libfreenect.so.0.1

     #ln -s /usr/lib/libfreenect_sync.so.0.1.2 /usr/lib/libfreenect_sync.so.0.1

     #ln -s /usr/local/lib/liblibusb.so.1.0.8 /lib/libusb-1.0.so.0

     =============================================================

     7. Reboot your myRIO and your libfreenct library should be ready to go.

C. Run Example

Software Requirements:

• LabVIEW 2013
• LabVIEW Real-Time 13.0.0 - NI myRIO 13.0
• Kinect Example VIs (attached)

Hardware Setup:

Connect your Kinect to the myRIO USB port and powered it by the wall adapter. Also, you can power Kinect with a external 12V DC. To make sure your Kinect is working properly, again SSH to your myRIO and type the following commands to check whether all Kinect USB devices are correctly mounted. And you should be able to see three USB devices in the list: "Xbox NUI Audio", "Xbox NUI Motor", "Xbox NUI Camera". If you cannot see them, make sure your Kinect is correctly connected.

     ====================================================

     #cat /proc/bus/usb/devices

     ====================================================

Software Overview:

In the included project, there are two top level VIs which can be run after the Kinect is connected, and the libraries are installed on the myRIO.

Kinect Depth Controller has controls for the Depth Resolution and Depth Format, as well as the LED option and Tilt Degree. All depth formats and resolutions should work (NOTE: resolution and format have to be set before the program is run, and don’t have effect while the program is running). Tilt degree controls the motor inside of the Kinect that rotates it. LED option controls the LED on the front of the Kinect.

Depth resolutions: Tiny (160x120), Low(320x240), and Medium(640x480)

After Init Kinect Systems has run, any of the Vis that read depth or video data from the Kinect should work (exceptions for video resolutions noted below). It is important that Close Kinect Systems is called (use stop button, don’t abort); if it’s not executed, on next execution of the VI the myRIO will lose connection immediately. However, running it again after the disconnection will work.

Kinect Video Controller has the same controls as Kinect Depth Controller, but this time the Video Resolution and Video Format controls will take effect (again, these must be set before executing the VI). NOTE: as of right now, none of the video formats work in Low(320x240) resolution, all of the formats work in Medium(640x480) resolution, and only RGB and Bayer formats work in High(1280x1024) resolution.

If you will be doing image processing, or converting the depth to physical units, the projects on Obstacle Avoidance and Color Following with the Kinect have examples of this. Basically, you will want to modify the Read Video or some Read Depth VI so that you get the 2D array of points instead of the picture. From there you can convert it to an IMAQ image and process it with IMAQ functions.

The lower resolutions for both Video and Depth are much faster than higher resolutions. In the other projects, we’ve been getting about 15fps on tiny depth resolution, and ~3fps on medium resolution video.

Kinect VIs & Wrapper library

Credit for the Kinect VIs and wrapper code goes to Vicentiu Neagoe, Mihai Serban, and Adrian Papp.

This is a updated version of the original documentation written by erik.j.o

The attached "Async_wrapper.h" is the c header file of this library.

[probs]

Hi!

This is a very good instruction. I have successfully controlled kinect by following the steps above!

For more flexbility, Is it possible to share the full fuction list of libkinect_wrapper with us? or where can we get it?

Thanks,

Polin Liao

[xkingkiller]

If you see the myRIO->subVIs->library calls of the project I attached, you can see most of the VI-warppers for the APIs. I also attached the header file of this library - Async_wrapper.h now, you can refer to the header file.

[probs]

Hi Xkingkiller,

Thanks for your reply. I'll check it out.

Thanks,

Polin Liao

[PeteFromQuanser]

Hey Xkingkiller,

Thanks a lot for putting this together. I followed your instructions, but I'm running into a problem. When I run #cat /proc/bus/usb/devices I don't see the motor device, and when I run either of the sample VIs I immediately get "Connection to the target (myRIO) has been lost". Here is what I see when I check the installation:

Any help would be very much appreciated.

Thanks,

Peter

[xkingkiller]

Hi Peter,

    Can you try to make sure your Kinect is working properly with your PC host, say using a MicroSoft SDK, and your myRIO is initialized with the "Getting Started Wizard". Normally, the Audio and Camera device will not show up due to disconnection of the 12 external DC Power. Never came across the missing of the Motor device.

[PeteFromQuanser]

I installed the Kinect SDK for Windows and was able to get the Kinect up and running. I was also able to run a couple sample VIs for myRIO with no problem. I reinstalled both the myRIO software, and the Kinect drivers to no avail. I'm not sure what this indicates in terms of debugging, but the light on the front of the sensor is flashing when connected to myRIO, as opposed to solid when connected to my PC. Does it matter that I'm not using the specific Kinect for Windows, but the XBOX 360 Kinect? I will try to test with another Kinect this afternoon to see if it's a model problem (the current one I have is 1473, but we have a 1414 as well).

Any other thoughts are appreciated.

Thanks

[xkingkiller]

It seems to be a known issue with libfreenect for the new Kinect model 1473. I found a discussion about this issues: https://github.com/OpenKinect/libfreenect/issues/316

Can you successfully running the examples on your Kinect 1414?

[WallyFromDilbert]

Ok, so I am in need of some help. I have followed the instructions several times know but I keeping getting an error saying "Connection to the target (myRIO) has been lost". When I run the #cat /proc/bus/usb/devices command I do not see the "Xbox NUI Audio", "Xbox NUI Motor", or "Xbox NUI Camera” ….. this is what I see. I am using a kinect model 1517, for windows.

[WallyFromDilbert]

I do not think there was been an update to the lib fiel for model 1517 to work, you can always try...... but I doubt it will work. I had the same problem (read post above), the code only works for model 1414 to my knowledge.

[djonas92]

I tried it before reading the comments, i get the same error "Connection to the target (myRIO) has been lost" so the didn't update it. That's why i asked.

[racoon90]

I want to use this program on Window.

but .so file is not available.

what shoud i do?

[Jorge_Valerio]

Hi,

I followed the instructions and tested using #cat and apparently everything is Ok, but when I run the example got this message:

__________________________________________

Initializing...

Calculating dependencies...

Checking items for conflicts. This operation could take a while...

Preparing items for download. This operation could take a while...

Deploying myRIO(successfully deployed target settings)

Deploying Kinect_init(SubVI).viKinect_init(SubVI).vi loaded with errors on the target and was closed.

LabVIEW:  Failed to load shared library libkinect_wrapper.*:kinect_set_log_level:C          . Ensure that the library is present on the RT target. Use MAX to install NI software or FTP to transfer custom libraries to the RT target.

LabVIEW:  Failed to load shared library libkinect_wrapper.*:kinect_init:C          . Ensure that the library is present on the RT target. Use MAX to install NI software or FTP to transfer custom libraries to the RT target.

Deployment completed with errors

________________________________________

What did I do wrong?

Thank you in advanced.

[BradM]

https://decibel.ni.com/content/docs/DOC-30232

[M.Arafuka]

Hi,  I am not familiar with Linux and having the same proble as Jorge_Valerio got the error message. It seems that I am doing wrong on the step 4 of copying files. Could you give me step-by-step instructions of how to do this by using FileZilla? Or any useful URL?  Thanks.

[BradM]

I have not used Filezilla, but based on the screenshots, it looks like the older style FTP client interface. I would ssh to the target and check for the library files in the locations listed in step 4 for a better idea of how things are going awry.

Once ssh'd into the target using, e.g., PuTTY

(check to make sure that the libraries have been copied to the correct locations)

ls /usr/lib/libfree*

ls /usr/lib/libkin*

ls /usr/local/lib/*usb*

(check to make sure that the library loading cache knows of these new libraries)

ldconfig -p | grep -E "lib(free|ki)nect"

[approgramer]

I am using Kinesthesia toolkit for kinect to measure angles between joints...It worked fine on computer but when i load these vi to myrio the deploy botton is brocken,,.. Why doesnt it run on myrio.

I did all the above steps and the depth imaging and videio worked, but my project requirs skeletal tracking. how can i get .so libraries for skeletal tracking.

if i import the kinesthesia library( those are .vi files) into myrio using filezilla following the same above steps will those VIs work on myrio.

can i import .vi files into myrio and will it work?

if not is it possible to convert .vi file into .so? if yes how can it be done??

[M.Arafuka]

Hi, Jorge_Valerio,

I have the same troble that you faced.  I could finally do the step4 and see all three devices of Kinect and all three libraries in PuTTY. It seems that I did it correctly. When I run the sample VI, I got the same massage of "libusb-1.0.so.0:cannot open shared object file: No such file or directory" that Jorge_Valerio faced. How did you overcome?

[M.Arafuka]

I could finally copy the files stated in the step4. I used Win SCP instead of FileZilla. Now I can see three libraries below.

libkinect_wrapper.so (libc6) => /usr/lib/libkinect_wrapper.so

libfreenect_sync.so.0.1 (libc6) => /usr/lib/libfreenect_sync.so.0.1

libfreenect.so.0.1 (libc6) => /usr/lib/libfreenect.so.0.1

But when I run the example VIs, I get the error message of "libusb-1.0.so.0: cannot open shared object file; No such file or directory". What should I do? It seems like the same problem that Jorge_Valerio faced.

[approgramer]

thanks bradm

[BradM]

masaman,

You show the listing for three of the four libraries, did you copy over liblibusb.so.1.0.8 and create the noted symlink?

[M.Arafuka]

I could finally use Kinect with myRIO.  I checked the symbol link again. Is it wrong statement in step6? I re-created symbol link below. Then, it worked and I could use the examples.

wrong:   #ln -s /usr/local/lib/liblibusb.so.1.0.8 /lib/libusb-1.0.so.0

correct:  #ln -s /usr/local/lib/liblibusb.so.1.0.8 /lib/libusb-1.0.so.1

But anyway, thank you all who supported me.

Thanks

[Lexicondi]

Hey xkingkiller!

Thank you so much for this libraries. Are there pallettes in labview that I can use in conjuction with this libraries to adapt your code to my needs? It doesn't work with the Kinesthesia toolkit certainly and I don't think it works with the IMAQ Vision toolkit as well.

Basically, I am truing to acquire the maximum pixel from the depth map. I will be glad if you could point me in the direction by which I could do this.

Thanks!

[chokkk]

hello when I try to make the correction still have the same problem. Needless to note that the specification to the library " usr " Might help

[Baarath12]

hi..i have went through the instructions...but im facing a problem since im using the newer model of kinect. Does anyone found a way to solve the problem for connecting the newer model.

[BradM]

Fatik.Hasan,

It seems like you are trying to run a VI intended for running on the target (where you have just setup the noted libraries from KinectLib.zip) on your desktop (from the My Computer context). The VI is looking for libkinect_wrapper.dll, which is likely a result of the VI using a CLFN configured to use libkinect_wrapper.*. Also, the dependency search dialog is a giveaway.

[fatikhasan]

BradM,

I've followed all the required steps.

Verified all the libraries learning from your comments,

and I can also see my Kinect devices detected by myRIO.

but whenever I run a VI the following happens:

it gets stuck here each and every time and when I cancel it, the following error appears:

What did I do wrong? and how I can troubleshoot this issue?

Some help would be greatly appreciated.

[BradM]

Can you check to make sure that liblibusb-1.0.so.0 exists in the ldcache?

ldconfig -p | grep libusb

Also, note that you created the link in the wrong location, which is likely leading to your issue.

From the original instructions:

ln -s /usr/local/lib/liblibusb.so.1.0.8 /lib/libusb-1.0.so.0

[Fatik.Hasan]

I finally got it to work! You've been really helpful. Thanks alot BardM!

[DDHITO]

Hello SID

I had the same problem as you had,could you plz tell me how you solved the problem?

[BradM]

The link creation failed, you need to add a space:

//You have

ln -s/usr/local/...

//You need

ls -s /usr/local/...

[fatikhasan]

After you have transferred the libraries to myRIO, use the following lines of code in PuTTY:

ln -s /usr/lib/libfreenect.so.0.1.2 /usr/lib/libfreenect.so.0.1

ln -s /usr/lib/libfreenect_sync.so.0.1.2 /usr/lib/libfreenect_sync.so.0.1

ln -s /usr/local/lib/liblibusb.so.1.0.8 /lib/libusb-1.0.so.0

Verify if you have copied the libraries in the right location, and if the libraries are cahced, using the methods in the conversation between BradM and I.

For verification of cache:

ldconfig -p | grep -E "lib(free|ki)nect" (3 files should appear when you run this)

ldconfig -p | grep libusb (1 file should appear when you run this)

Do tell me if you get it to work.

[Fatik.Hasan]

Reset the myRIO.

Go to NI MAX, right click on your device under Remote Systems tab and click Format Disc.

Then restart myRIO, use the "Getting Sarted with myRIO" wizard to install software on NI myRIO.

Make sure that the myRIO doesn't disconnect or you don't lose electricity during the process.

Again, copy the libraries to the mentioned locations in myRIO using fileZilla nad run the lines of code that <fatikhasan> posted.

Restart myRIO.

Your kinect should be ready to go.

Also, this doesn't work with LabVIEW 2015. My recommendation is the use 2013.

[DDHITO]

hello hasan

i try many times but still couldn't make it work. I had reseted myRio and check everything

i can see all device working well, but just can't make libusb-1.0-0 work

[DDHITO]

I am stuck in the same place , how could i open the VIs and adjust the CLFNs? And I am using Labview 2014, is that relavent to the problem i got?

[BradM]

DDHITO,

Why are you trying to install libusb through opkg? The library that is needed (which is liblibusb, not sure why it's labeled that way) is in the zip file attached to the project description. Please make sure to follow the instructions carefully, taking note of any error messages or warnings that may happen during running any of the commands.

[BradM]

DDHITO,

That seems correct. What does the output of ldconfig -p | grep usb show?

By the way, if you're using PuTTY, simply selecting the text in the window will copy it to the clipboard, if memory serves me.

[DDHITO]

BradM,

This is what the output of ldconfig -p | grep usb shown

sometimes the clipboard doesn.t work properly, i always select the text and press Ctrl+P.

[DDHITO]

BradM,

This is Why I am trying to install libusb through opkg          https://decibel.ni.com/content/docs/DOC-31239

this is the return of it

[fatikhasan]

DDHITO.

Don't type "#" while typing lines of codes in putty. Directly start from ln -s ...

This is the only problem.

[DDHITO]

Fatikhasan

It works!!!!!!!!!!

Thank you Bro!!!!!!!

[DDHITO]

BradM,

Thank you for your patient, and your time, it works properly!!!

[fatikhasan]

BradM,

The mistake that enthusiats are making in this tutorial is that they are typing "#" while executing the commands in putty. The rest is good. I also did this mistake back while I was interfacing Kinect with myRIO.

Though I learnt alot from you and discovered more about LabVIEW and myRIO. Thanks for all the support you gave me, and all the other members of the community.

DDHITO,

Congratulations! Do tell me what project you're working on with Kinect and myRIO.

[DDHITO]

Hello faithhasan.

Sorry that i missed your comment. I am working on a project called Obstacle Avoidance AGV as my graduation project. I am using MyRio with kinect to contral a four Mecanum wheel AGV.

[L_NG]

Hi I'm new here, I'm using XBOX 360 Kinect (Model 1414), NI myRIO 1900 (18.0), Labview 2018.
in my NI MAX, myRIO did detect the device as Xbox NUI but in USB instead of "camera".



I did the steps as shown in this forum, but mine looks nothing like the one that almost everyone here get...
using Putty

I don't have the " /proc/bus/usb/devices " directory but I do have this directory "/proc/bus/input/devices" . 

I have connected XBOX 360 Kinect to myRIO using USB and also supplied the Kinect with the wall plug, Power up the myRIO with wall plug as well.

The XBOX 360 Kinect has yellow light blinking as well

Can anyone help me with this? ...