Machine Vision

cancel
Showing results for 
Search instead for 
Did you mean: 

USB3 Camera Freezing Issue

We have three Basler USB3 cameras connected directly (not through a hub) to the USB3 ports of a high-end workstation (Intel Xeon Gold 6130 2.1GHz, 3.7GHz Turbo, 16C processor, 64GB RAM, Nvidia Quadro RTX 5000, 16 GB GPU). LabVIEW 2019 and Vision acquisition software 2019 along with Vision Development Module 2019 have been installed on the computer. The cameras configuration has been modified in the MAX, primarily to change the resolution to 1920 x 640 and the exposure time to 1000. note that these changes have been made in MAX once during the camera installation and saved to the camera files. These settings are not accessed or modified during the runtime application. The runtime application uses a loop to continuously grab images from the three cameras. This module is identical to the HL Grab.vi example in the LabVIEW examples which shows how to grab continuously from a single camera. In order to grab from three cameras, we just used a for loop to grab from three cameras. Everything seems to work fine until one or more cameras randomly freezes, most likely due to a timeout error, and never recovers from it. This happens randomly in the sense that sometimes it occurs within a few minutes and some other time after a few hours. Note that these cameras are required grab continuously for at least eight hours without any interruption. When the camera freeze issue occurs, the only way to recover from it is to restart the computer and restart the application, which is undesirable. On the NI forum, we found a discussion which seems to be very similar to our issue here: https://forums.ni.com/t5/Machine-Vision/IMAQdx-Timeout-0XBFF6901B-after-running-camera-acquisition-f... . We have tried almost everything possible including changing the USB3 cables, USB3 ports, deleting camera icd files, cleaning the cameras and so on, however, the issue seems to reappear randomly every now and then. Please advise any resolution for this issue. Also, is there a way to handle the exception programmatically with something like the following:

Pseudo Code:

if (frame_acquisition_time > 3000 or frame_corrupted)

{

    acquire a new frame for that specific camera;

    continue;

}

What appears to be causing the issue is one or two corrupted frames in thousands of frames and the above approach might work. Does LabVIEW do something like this already? If not, is there a way to code this into the acquisition module?

Download All
0 Kudos
Message 1 of 4
(3,836 Views)

Can you check device manager and report the host controller that you're using? NI recommends Intel and Renesas USB host controllers, and have seen some quirks with other vendors. One thing to try might be a separate host controller with dedicated bandwidth per port such as this one: https://www.ni.com/en-us/support/model.pcie-8244.html

 

 

 

0 Kudos
Message 2 of 4
(2,772 Views)

Attached is a screenshot of the USB controller in the device manager. 

Note that the LabVIEW application does not change any camera attributes during runtime. The acquisition exposure time was changed to 1000 and the resolution to 1920 x 640 in NI MAX and saved to camera icd file at the time of camera installation, and these are never changed afterwards. But while I set the resolution, pixel format, and exposure time in NI MAX, I did not explicitly set the frame rate. Could that be the reason for the occasional timeout errors noticed in LabVIEW? In this case, do the cameras attempt to run at their highest bandwidth? If so, can the combined bandwidth exceed the USB3 port limit? I think this could happen if the USB3 ports on the Dell 7920 workstation that I am using are not dedicated but shared. Should we be using a PCIe USB3 card, like you suggested, instead to interface the cameras?

0 Kudos
Message 3 of 4
(2,745 Views)

Thanks for the screenshot -- Intel is a reliable host controller. It's possible/likely you're sharing bandwidth, but that's not necessarily a problem as long as the total bandwidth between all the ports exceeds the combined bandwidth of your cameras. To see if it's a bandwidth issue, I would recommend throttling down the bandwidth on all three acquisitions to see if that makes the timeout go away. Basler cameras always implement the standard attribute for doing this: Camera Attributes::Device Control::Device Link Throughput Limit. The max throughput I've seen for a single USB3 camera is 375 MB/s or so, but not all sensors are capable of saturating the bus. Make a note of what the current throughput is (Device Link Current Throughput) and then I would be extra cautious by setting each camera to have a limit of ~125 MB/s, which is about a third of the max bandwidth of a single USB3 port (taking into account the protocol overhead). If the error still occurs when all three cameras do not exceed ~375 MB/s, you can be confident it's not a bandwidth problem.

 

Regarding your questions on attributes, it shouldn't really matter whether you set camera attributes at runtime or if you save them beforehand and don't change them. I am not sure what the defaults are on the cameras you're using, so without explicitly limiting the frame rate I'm not sure if it'd already be limited in the out of the box settings. You can see what those settings are in MAX under Camera Attributes::Acquisition Control::Enable Acquisition Frame Rate/Acquisition Frame Rate. If you're playing around with limiting the bandwidth, I would recommend using Device Link Throughput Limit as described above instead of Acquisition Frame Rate. This is because even if the frame rate is lower, there can be spikes of activity if all cameras are transmitting data at the same time where the instantaneous bandwidth would be higher than you want. FLIR has a nice document about multiple USB3 cameras that I just came across that you might be interested in: https://www.flir.com/globalassets/support/iis/application-notes/tan2012005-usb3-multiple-camera.pdf

 

Hope this helps,

Katie

Software Engineer, Vision R&D

0 Kudos
Message 4 of 4
(2,703 Views)