How to align the RGB image of the Intel D435 camera with the depth map in LabVIEW?

[zdz]

In Intel's official LabVIEW example for D435 camera, "RealSense_HelloWorld_PointCloud_ext_RGB2.vi" cannot achieve the alignment of RGB image and depth image. I see that people are also struggling with this problem on github. Does anyone know the reason? Very anxious and grateful!

[Bob_Schor]

Gee, it would have helped a lot if you provided the LabVIEW code that doesn't seem to work for you.  Please help us to help you ...  (I'll admit it, I'm too lazy to do a Web search to look up the D435 camera from Intel, find the code, etc. ...).

Bob Schor

[rolfk]

Actually I'm not sure that the LabVIEW VIs would help much. This is an Intel RealSense library in the form of a shared library and they do provide LabVIEW wrappers for that library (a very remarkable feat I have to say and certainly laudable). In terms of what is really done with the image data, the VIs really do not help much without knowing exactly what the underlying functions are really doing. And even if the problem is not in the library itself but a bug in the LabVIEW binding VIs I doubt that anyone not being pretty intimate with the functionality of the underlying functions has a big chance to find out what is the problem.

First step would be to verify that that exact same operation does actually work elsewhere, in one of the non-LabVIEW examples. If that is the case then you would have to make yourself familiar with what the functions actually are doing to understand where the calling of those functions may go wrong. If it fails in that other example too, it's bug report time to the project as I doubt anyone other than project members are likely going to dig into the actual library code to start debugging. It's definitely quite far outside my own expertise and I therefore won't be very helpful for the actual bug hunting.

[zdz · Accepted solution]

I have solved this problem through LabVIEW，There is a problem with Intel's official example for LabVIEW. In fact, it is not fully applicable to D435, and the content is not as rich as other programming languages. Therefore, in terms of RGB-Depth alignment, the example is that the RGB camera and the left camera (binocular camera) are the same camera. The Realsense device is not suitable for D435 and D435i. We need to obtain it by ourselves through coordinate system conversion: First, we need to obtain the internal and external parameters of the camera, which are in rs-sensor-control.exe in the tool folder after SDK installation. Get, convert the pixel coordinates of the left infrared camera into camera coordinates, and then get the world coordinates, the world coordinates have an offset in the X direction, and then get the world coordinates of the RGB camera, then get the RGB camera coordinates, and then get the RGB pixel coordinates , which realizes the pixel conversion from the depth map to the RGB map. To convert the RGB pixel coordinates to the depth map coordinates, a corresponding approximate search is required.

[hk182326]

Hello.

I had the same problem and found this forum.
I opened rs-sensor-control.exe but I am not sure what to look for.
Could you please tell me your solution in detail?

[zdz]

Hope it helps you

[hk182326]

Thank you.

I got the RGB and Depth info to use as well as the pics you sent.
But after this I don't know how to handle this information.
If you don't mind, I would like to know the detailed process after this.

Sorry for my ignorance.

Sincerely.