IMAQ Start.vi hangs with many buffers

[zobfish]

Hey folks,

First and foremost, thanks for taking the time to take a peek and (hopefully!) help me out.

I have been using low level IMAQ vi's to do high speed acquisitions of a limited number of buffers from a cameralink camera (<~3000 buffers, stored on system memory, individually hardware triggered, 4x4 binning, 256 x 256 gray, @ 60fps) with a good deal of success for a little while now.  Check out the examples/imaq/IMAQ Low Level.llb/LL Ring.vi - it is essentially the same thing, except I use a IMAQ Copy Acquired Buffer.vi followed by an IMAQ Get Buffer.vi instead of the two IMAQ Extract Buffer.vi's (though saying that may be a mistake when it comes to our slippery friend, LabVIEW).  When I try to pump up the number of buffers, e.g. >5000, I start running into problems.

By highlighting the execution on the block diagram and using some indicators on the front panel, I have found that the process chugs along just fine until it gets to the IMAQ Start.vi, which, as you can see in the example, is right before and is responsible for beginning the acquisition.  Instead of passing along the the acquisition, the process just hangs there.  As I increase the number of buffers, IMAQ Start.vi takes longer and longer to pass to acquisition, taking minutes to pass between 3k-5k, if it does at all (I haven't waiting beyond 2 min).  The system does not seem to be overloaded (as per the windows task manager, and there is still plenty of RAM unused) and I have been unable to find any answers elsewhere online.

Potentially relevant details:

• Also doing some analog acquisition on a DAQ board (PCI-6259)
• Also acquiring from a usb cam via IMAQdx
• IMAQ card is a PCI-1428
• Camera is a Dalsa Pantera 1M60
• Quad core 2.8ghz
• 12gb ram
• Win7 64
• LabVIEW 2010 (v10.0) 64bit

[zobfish]

I've attached a picture of the relevent segment of the program.

[zobfish]

Anyone?

I'm happy to provide more details if need be...

[NateBraska]

Hello zobfish,

That is an interesting problem you are encountering. I would suggest to get your program to as simple as it can be and build from there. Is it only after you set the buffer to >5000 that this issue happens, and it works fine at <5000? What else is in the Sequence Structure of the IMAQ start VI? Are you sure that it is hanging on the start versus something else in that Sequence Structure? Also, you said the RAM is fine, but what about your computer's CPU usage?

Let me know about those points and we can see if we can get to the bottom of this.

Best Regards,

Nathan B

Applications Engineer

National Instruments

[zobfish]

Nathan,

Thank you for helping me out (regardless of the outcome)!

I'll work on stripping it down to just the IMAQ components.  In the meanwhile, answers to your questions that we can work from:

• 5000 buffers isn't a binary GOOD/BAD limit, so to say.  The delay at the IMAQ start vi increases in time almost exponentially.  When around 2 or 3k buffers, the delay is not all that noticeable.  Approaching 4k it starts to become a real nuisance to wait through, and becomes rapidly worse going up from 4k, ultimately taking some unknown time >2min at ~5k.
• Referring to the picture I posted, the imaq vi order is (names might be wrong, not at work computer):  initialize --> config acquisition? --> config camera --> config camera property (x4) --> grad Image Type property node --> --[for loop]--> imaq create --> config buffer list --[end for loop]--> --[sequence structure]--> start vi --> --[end sequence structure] --> --[while loop]-->copy acquired buffer --> grab Lost Frams & Frame Count property node --[end while loop]--> get buffer --> close
  
  The for loop at the far right is to write the images from the get buffer vi to arrays, then to a binary file (the format that is accessible to my analysis program)
• Like I said in my original post, I used the highlight execution to watch the program execute each vi.  There is one other output of the sequence structure, which completes and waits for the Start Vi to pass so the whole sequence can pass.  All the other little dots are at the while loop ready to execute once the sequence structure passes and send its dots to the while loop.  It is always the little dot hovering at the Start Vi that is not passing and holding everything up.
• I will check again on resource usages and try and supply some numbers (again, not at my work computer), but I seem to remember that the CPU usage was also fine.

Lastly, thanks again for taking the time to look into this.

Cheers,

zobfish

P.S. The spell check deleted all the spaces in my post?  I had to go back and put them all back in, yikes!

[zobfish]

Update:

I checked the resource monitor and made a few notes:

• For 2k, CPU load when reserving buffer list and while the start vi is passing is ~25%, pretty reasonable.  The total load doesn't change - stays at ~25% when loading in more buffers (i.e. 3k or 5k)
• It favors the second core of the processor (of 4 total cores) almost exclusively.  There is some activity on the first core while the buffers are loading, but ONLY the second core shows activity while the start vi is passing.  I checked the core affinity of the LabVIEW process (via the task manager) and it is indeed set to use all four coures.  Strange?
• Another strange thing with the memory...as the buffer reserve loop is running, the "In Use" memory usage does not change much.  Yet, when LabVIEW passes to the start vi, the "In Use" memory increases at about 1MB/s until the acquisition starts.  Once the acquisition is finished, the "In Use" memory that was build up during the start vi becomes "Modified" memory and then is converted to "Standby".

Hope these clues will help!

Best,

zob

[JÞB]

---

@zobfish wrote:

Update:

I checked the resource monitor and made a few notes:

• It favors the second core of the processor (of 4 total cores) almost exclusively.  There is some activity on the first core while the buffers are loading, but ONLY the second core shows activity while the start vi is passing.  I checked the core affinity of the LabVIEW process (via the task manager) and it is indeed set to use all four coures.  Strange?

Hope these clues will help!

Best,

zob

---

Not really- The loop is not set up as a parallel loop so each interation will be processed in the same thread.  You could try to run the loop parallized (depending on LV version its a recient feature). 

[zobfish]

Good point, Jeff - that makes sense.  I'll check that out and see what we get.

[zobfish]

I put a frame in the sequence structure before the frame with the start vi.  I put the buffer for loop in there and parallelized it.  I tried 2k - saw activity on all 4 cores, but not all that noticeable of a difference in time to buffer.  Tried 3k and the computer bluescreened about halfway through the buffer list with a page_fault_in_nonpaged_area error and referenced niimaqk.sys.  Yikes!  Any idea what caused that?

[zobfish]

My problem is still unresolved.  Anyone have any new ideas?