Why does releasing or creating IMAQ frames take so long?

[CSD]

I have a machine with a large amount of system RAM (32 GB) and I'm using this system RAM as a circular buffer for my machine vision camera (NI Test Image Generator, 1280x1024, 8 bits, 8 taps).  I ran into problem allocating and deallocating large amount of system RAM through IMAQ, so I ran some tests with the LL Ring example for this post.  I've modified the LL Ring example that ships with IMAQ to add 4 timers (code image below) and I've created an EXE from my modified LL Ring example.  I've listed my data below:

10 Frames:  2ms, 77ms, 5ms, 2ms

100 Frames:  24ms, 238ms, 71ms, 9ms

1000 Frames:  237ms, 2157ms, 4588ms, 110ms

2000 Frames:  701ms, 4198ms, 33499ms, 266ms

3000 Frames:  1463ms, 6714ms, 89297ms, 435ms

4000 Frames:  2486ms, 8483ms, 173576ms, 627ms

5000 Frames:  3812ms, 12154ms, 276378ms, 839ms

Why does it take so long to deallocate the data space allocated by IMAQ Start?  5 minutes for deallocating 5000 frames, that I never want to see again, seems to be excessive.

If I can't speed up the deallocation process, Is there any status indicator I can read in parallel to the IMAQ Close subroutine to get any insight into the deallocation progress?

I'm using LabVIEW 2009 64 bit and IMAQ 4.5 to develop the LL Ring exe, but I'm deploying the system to a NI-IMAQ 4.7.  Both machines have Windows 7 64 bit.  Both exhibit the problem.

[BlueCheese]

Hi,

This looks like a bug to me that some operation is behaving exponentially. Given that the allocation times appears roughly linear, I suspect there should be no reason the de-allocation can't be the same. I have created an internal bug tracking ticket (CAR #383827) that you can refer to in the future if you wish to request the status as well as if it is listed in future release notes when the issue is addressed.

Thanks for the detailed benchmarking data to identify the performance issue you were seeing.

Eric

[CSD]

Thank you for filing the CAR for me.  I'll watch it with interest.  Right now, it makes NI-IMAQ unusable compared to the Bitflow or Matrox framegrabber SDKs.

It may seem like the allocation code is ok, but it turns exponential as well.  I've included a few more data points that I didn't have run when I posted yesterday.

6000 Frames:  5400ms, 57144ms, 404532ms, 1068ms

7000 Frames:  6879ms, 295726ms, 563571ms, 1317ms

I'm also in the process of running 15000 frames, which consumes 22.6 GB of RAM, and it took 34.196 seconds to run through the allocation, and 3624.348 seconds (1 hour!!) to run through the IMAQ Start vi.  NI-IMAQ Close took 2643.414 seconds (45 minutes) to run, and the final deallocation process took 4.817 seconds to run. 

As a point of reference the Bitflow SDK takes about 1.5 minutes to allocate 70 GB of image capture data space in RAM in a similar PC.

While someone is working on the DLL behind NI-IMAQ Start and NI-IMAQ Close code, please ask them to provide some sort of progress indicator to the outside world.  I'd like to be able to display some sort of dialog to my users to let them know where they are in the memory allocation process.

[BlueCheese]

Hi,

I did a bit of profiling here and there are a few issues at play here...

The first is that by default, Vision images contain borders (the default if you don't wire the parameter is 3 pixels). When you acquire into those images with an IMAQ board, the board "skips" DMA'ing data in the border region (since with large border sizes it could be quite a bit of extra data). The downside here is that the driver has to create scatter-gather lists/links that are broken up quite a bit so that the hardware can skip data at the end of each line. If you have a border size on all your images set to "0" and your image width is 32-byte-aligned (yours appeared to be), you should get a fairly large improvement in your allocate/deallocate speed.

Now, given that the memory of the images might not always be physically contiguous (especially if you are using a large amount of memory), there still might be scenarios where there are excess links for each image (since the OS deals with things in 4K pages). There is a feature in IMAQ that can enable image buffers to be allocated as physically contiguous, but based on my testing with a system with tons of RAM it likely will not give you much improvement as each buffer is likely to be mostly contiguous anyway.

Now, there still is some non-linear behavior present that I'm still investigating, but my guess is that with the tweak above it should let you allocate a much larger number of images before you start hitting where the exponential growth becomes too high.

Eric

[CSD]

Setting a border size of 0 does help, at least at the 1280x1024 image size.  I reran all of my data points and saw dramatic improvements, a few of which I've listed below.  It appears that the IMAQ Close time does begin to go exponential after 10,000 frames.  In my case, I'd like to be able to be able to capture at least 5 times that number of frames (assuming I have the available RAM).

1000 Frames:  233ms, 417ms, 327ms, 105ms

5000 Frames:  3796ms, 1782ms, 7672ms, 811ms

10000 Frames:  14536ms, 4078ms, 60232ms, 2127ms

15000 Frames:  31281ms, 7049ms, 151613ms, 4722ms

Is there a way to turn on the IMAQ option you mention in the above post? 

I'm also using a camera that runs at 2048x2048, and it has IMAQ Start issues with 5000 frames.  @2048x2048 5000 frames takes 684 seconds to run through IMAQ Start.

[BlueCheese]

Here is the snippet to enable contiguous allocations. There is no property node for this particular attribute, so you have to use some of the internal VIs from the LLB.

[CSD]

I'll give this a shot before I go home for the holidays.  If I can get some results from my higher frame counts, I'll report them here.

[CSD]

I tried to put together the above snippet in my LabVIEW 2009 64 Bit install, but it didn't work, and produced an error.

In any case, my problem is not completly solved, but is better with setting a border size to 0.

At this point I have tested the following number of frames at an image size of 1024x1280.

10,000 frames

Allocation:  13.6 seconds

IMAQ Start:  4.2 seconds

IMAQ Close:  52.6 seconds

Release:  2.1 seconds

12,500 frames

Allocation:  23.3 seconds

IMAQ Start:  5.7 seconds

IMAQ Close:  101.0 seconds

Release:  3.4 seconds

It seems like the time to complete IMAQ Close has begun its exponential climb.  I've got a machine with 80+ GB of RAM that I'd like to be able to use with the 1433 framegrabber and a Basler 504k camera with this size image, but I can't make full use of the available RAM if I use code that utilizes IMAQ Close unless this bug is fixed.  Any other hints on how to get around the exponential time increase in IMAQ Close?

[BlueCheese]

Hi,

Thanks for testing this out and also for your patience. At this time I can't recommend any other workarounds that will make it any faster. We have identified where the issue lies and are still looking at some possible solutions. There are some memory management functions outside of our driver's direct control that are behaving exponentially in the specific use-case that IMAQ is exercising. As you saw, this only starts to be an issue when you get past several GB worth of memory, so it has not been a very common use-case until recently.

We are looking at some ways to mitigate this and I will update you when we have a fix for it.

Eric

[CSD]

Thank you for the help so far.  I look forward to the solution in a future version of the NI-IMAQ Vision Acquisition driver package.  Can I assume the original CAR # is still open, since a final solution hasn't been found?