NI-1744 Smart Cameras go offline using Vision Builder AI and LV2010?

[Kevin_R]

I've run into a perplexing and frustrating problem.

I have a set of 6 NI-1744 smart cameras used to monitor sample motion in an automated system

A central robot moves samples between 6 satellite chambers; there is a camera mounted above the entrance to each of the satellites; the sample carriers are drilled with up to 10 holes encoding their numbers in binary (0-1023). There is also an L-shaped registration mark drilled near the region for the number encoding. Matters are complicated somewhat in that the second generation of carriers has a slightly more complicated registration mark (which includes the simple L of the first generation).

What I would like to accomplish is this:

Most of the time, I just want to be able to see what's under each camera, illuminating from above using a ring light attached to the camera. Call that Inspection A.

I may need to capture an image without the ring light. Call that Inspection B

When a sample transfer is occurring, I need to run a more complicated inspection (Inspection C):

1. A light is turned on underneath the end of the sample transfer arm (the "fork"), backlighting the region where the hole pattern would be if a carrier is in fact present.
2. Acquire an image of the fork with the backlight on.
3. Check to see if the image shows the fork to be empty. If so, return that information and the inspection is complete.
4. If not, look for the (backlit) registration mark and establish a coordinate system for the holes. Look to see which holes are present, calculate the sample number, return the data and the inspection is complete
5. If not, look for the 2nd gen registration mark. If it is found, establish the coordinates as in step 4, calculate the number, return the data and complete.
6. If none of the registration patterns are found, turn on the ring light, capture an image and return it so the operator can intervene and enter the appropriate data.

I have successfully built the inspection, and it appears to work in the Vision Builder configuration interface, or if run from the Vision Builder inspection interface. If I attempt to *use* the inspection, accessing it via the VBAI interface in Labview, one or more of the cameras will hang after one or more inspections and stops responding to LabVIEW. It will take multiple reboots to get it back on line and visible to either VB or Labview.

Originally, I had thought to configure the camera to run inspections continously and select inspection A, B, or C based on a variable I could access using the Labview Shared Variable Engine.  Every time I tried that the camera in question would hang. My current sort-of-successful software uses the VBAI functions in LabVIEW to open Inspection A, B, or C on the camera, run that inspection a single time and return the results and inspection image. That still ends up with one or more cameras hanging, especially if I've added the case to handle the more complicated registration mark.

I think I may be running out of memory in the camera. I have occasionally received an out of memory error message when running the inspection in debug (step) mode in the VB configuration interface, at which point the camera will disconnect from the VB interface. When I look at the system monitor tab for a camera in MAX, I can see that it is showing me 11.5M free/124M total memory, and 72.3Mfree/118M total disk space. If I understand those numbers, that means that less than 10% of the memory is free when the inspection starts!

I'm only communicating with the cameras via the ethernet interface. Are there any software components I don't need to have installed on the 1744 (MAX lists about 12 different things installed!) in that case?

Alternatively, it appears there are VBAI functions available that might let me acquire an image and then process it via a local copy of Vision Builder running on the host PC, rather than in the camera. Is that so, and if so, would it be faster than running the inspection in the camera?

I've attached a zipfile with the inspection that appears to stall the camera, and samples of typical images captured of the empty fork and a sample holder.

[Brad]

Hello Kevin,

I tried running the inspection on my smart camera and ran into memory issues pretty quickly. Let me see if I can do something to help reduce the amount of memory used by the acquisition steps....

Thanks,

Brad

[BullDozier]

Hey Kevin,

I'm also seeing the same memory problems.  It's an issue with our current smart cameras that can run out of memory for certain scenarios involving large inspections.  The reason you have so little memory free at the start of the inspection is because Vision Builder will pre-allocate most of the memory it needs for running the inspection at inspection load time.  This saves time during the inspection with allocating memory.  The bulk of the memory that gets allocated at run-time is taken up by the images being acquired from the smart camera's sensor.  Since you have the SXGA sensor smart camera, you are at a disadvantage here in memory footprint size.

There are a couple of things we can do to reduce your memory footprint that could help you get to a level of memory usage that allows you to run successfully. The easiest thing to do would be to reduce the image size in the acquisition step to display only what you need to view in order to complete the inspection.  This will allow the driver to allocate less memory for your images and save you some memory.  Another thing you can do is to combine repeated steps/states.  From looking at your inspection, it appears that you could eliminate one of the calculate number states since it appears that they both just count the binary number etched into the rod.  If you cannot reduce the size of the image or eliminate steps/states then another option would be to revert back to Vision Builder 2009.  Vision Builder 2009 had a smaller memory footprint overall on the smart camera and that would save you some space without modifying any part of your inspection. 

Let me know if any of those are options for you.  Running out of memory on the Smart Camera can be frustrating.  It will always crash/reboot the smart camera because it's a real-time system.  Anytime you run out of memory on a real-time system, it's viewed as a critical error to the target. 

[Kevin_R]

Evan --

I thought of having both acquire and locate steps then use the same step for reading the binary code as an obvious way to reduce the size of the code, but couldn't figure out a way to conditionally specify the origin for reading the binary code (i.e., if the first registration location works, use that origin and proceed, if the second works, use that origin instead). If there was a variable type for coordinate origins, I didn't recognize it.

Kevin

[Brad]

You could use a Calculator step in VBAI 2010, and the calculator step can produce a point result and a numeric angle which can be used by the Coord Sys step so now your coord sys will be dynamic based on results from Calculator step.

Hope this helps,

Brad

[Kevin_R]

Evan -- thanks, that should help.

I'm also looking at acquiring images that are smaller than the full frame.

One more question -- how many images is the camera hanging on to in memory? I only ever want to work with a just-acquired image. Is there a way to configure that without using external triggering to start image acquisition?

[Brad]

Hey Kevin,

The Smart Camera Acquisition step keeps 4 buffers in memory. Why don't you send your contact info to me at vbai.support@gmail.com and we can see if there might be another possibility to solve your application.

Thanks,

Brad

[Kevin_R]

I have made significant progress but can't quite call this problem solved yet.

I have successfully managed to programmatically assign the origin coordinates and thus eliminate one of the duplicated stages in the inspection.
Similarly, I figured out how to acquire an image that is only the necessary subset of the full camera field of view.

We are also going to update the older generation of carriers to match the newer ones, which will eliminate one of the alternate pattern matching stages.

I also figured out how to use a persistent session reference for each camera, so I am not burning resources connecting and disconnecting from the cameras every cycle of the calling LabVIEW vi.

At this point, I have all six cameras online and working most of the time. Every so often, however, one or another becomes unavailable. The strange thing is that while it vanishes from the visible network of remote devices (in both NI-MAX and VBAI) on the Windows 7 machine, but is still visible (and pingable) from a windows XP machine. Sometimes rebooting it from there will solve the problem, other times I have to actually cycle power on the camera.

I suspect this is still an issue with running out of memory in the camera. If VBAI 2009 has a reduced memory footprint and could solve the problem, I have no issue with trying a downgrade if it might solve the problem. I'd be much happier with enough space to run one inspection continuously that could do any of the 3 tasks depending on a user-set variable, as it would simplify things if my users need to restart the system when I am away.

All my cameras shipped with VBAI 2010; is VBAI 2009 available for download and will my inspections load in the 2009 version, or do I have to start over?

[Brad]

Hello Kevin,

Vision Builder AI inspections cannot currently be saved for previous, but if you email me your contact info at vbai.support@gmail.com I may be able to help you.

Thanks,

Brad

[Kevin_R]

The good news: thanks to some offline assistance from Brad, my cameras are no longer crashing.

The bad news: my labview VI using VBAI functions still fails for one or more cameras after a while (typically >12 hours of monitoring). 

It's very odd: the cameras are still visible online.

I can ping them.

I can connect to them via VBAI, run inspections that way (either in configuration mode or inspection mode), and disconnect successfully.
My calling VI, however insists that it can't connect (usually with a -354700 error from VBAI Connect.vi, occasionally with a -354705).

I am using shift registers in the monitor loop of the main VI so that I can pass the VBAI session reference for each camera back to the subvis that actually load and run the desired inspection once a connection has been made. After an indeterminate time, one or more of those appears to go bad, but if I attempt to reset and connect to the camera again (I did include a control to let me close and reconnect if necessary from the monitor loop), I still get the above errors.

The only way to get them back online within LabVIEW this morning proved to be to stop the main VI and reset all the controls to default, reboot the cameras, and then restart the VI. At that point it was able to connect to all 6 again and has been running happily for over 8 hours.

I set up the persistent sessions using the shift registers because I have observed in the past continually opening and closing resources like that can lead to memory problems. It also dramatically reduced my cycle time when all I want to do is get the latest images from the cameras to under 2 seconds.

Any ideas? Is there some subtlety to disconnecting/reconnecting to the VB in the cameras via Labview that I'm missing?

I am working on a slightly smarter version of the inspection that can be allowed to run continously in a camera and simply return the appropriate result when asked, rather than having to start and stop different inspections as I do now, but that is not ready for deployment yet. It's the symptom of running just fine for X hours and then losing one or more cameras that is baffling me right now.