64-bit LabVIEW - still major problems with large data sets

[DonRoth]

ps. The write portion should have the index wired into the 'replace array subset' fx and the image I have wired in would be different for each iteration of the loop.....Don

[Yamaeda]

I might be wrong, but the For loop in the Write is redundant and you shouldn't store the array in a front panel object, put it directly into the queue (or into the New data value reference).

I think this is the structure that was suggested, if as you comment on your picture you have dynamically loaded VI's it might be easier to just use the queue.

/Y

[DonRoth]

The array is initialized, then each element is replaced by a new image for each iteration of the for loop.  See my post after I used the VI snippet.  I forgot to wire in the index to show what I meant.  The empty clustered array represents an example image that is going to be different on each iteration....Don

[altenbach]

But, as Yamaeda said, you are using a front panel object to store the array data, this causes extra data copies. You should avoid any terminals for the dataset. Keep it on the diagram, all the front panel needs at any given time is exactly one image. 🙂

(I understand you are using arrays of clusters of arrays to avoid the need for contiguous memory which a 3D array would require.)

I was playing around a little bit and it seems using an array of DVRs seems to work pretty well and it is later easy to get any individual image by simply indexing into it. There seems to be exactly one data copy in memory. When I run the VI, the memory footprint increases by about 1.4GB, which is about as expected (Data size= 1920x1920 x 100 images x 4 bytes = ~ 1.5GB). Note that I am running this fine on 32bit LabVIEW 2010 under Windows 7 64bit. I have only 3GB of RAM. 🙂

After stopping the VI, the memory is released and we are back to where we started according to the task manager.

Note that I have not used DVRs in this manner and I am not sure if this method is sound. I will do a few more tests. 😉

[altenbach]

Another idea is to store the images inside a variant. Also here, we get exactly one copy in memory.

There are many ways to do all this. 😄

[EvenDeejay]

Hi guys!

I learned alot from reading and testing what you posted in this thread. I was really facinated by the program Altenbach created. But after watching the graph for five minutes, I now have a really bad headache

[Yamaeda]

That's common after Altenbach has some creative inspiration. 😄

/Y

[DFGray]

I would probably start with a single DVR and fill it with an array of clusters containing the images.  However, Altenbach's solution is more elegant.  It could have more memory issues with large numbers of images, since the wire you are passing around is, itself, an array, not a DVR.  But for a few hundred images, it should not be an issue.  I have never benchmarked the memory footprint of a DVR, but would be surprised if it were very large.

Altenbach's solution also gives easy access.  Placing an array of clusters of arrays into a DVR will give you two or three layers of In Place Element structure every time you extract or manipulate anything.  I have a hard time seeing how this would be faster (although I have been surprised before...).  Try it and find out.

Let us know if you have further questions.

[DonRoth]

OK - let me look into trying these approaches.  DVRs are definitely new to me.  Thanks again for everyone's help.  I may have a few more questions, and I'll post my results once I get this thing rearchitected.

Sincerely,

Don

[DonRoth]

Initially, I am using Damien's suggestion to use one DVR according to the following.

Removing IMAQ fxs, this approach seems to work fine.

If I add the IMAQ fxs that I need, I am running into trouble.  It seems to create more memory buffers.

There is improvement - I only went up to 9 gbyte RAM usage, down from 15 gbyte + disk swapping using my prior queuing approach.

Is there anyway we can address this further regarding the IMAQ fxs (I need to use these versus the conventional labview fxs for opening picture images since the IMAQ fxs provide more image type options)?

I see I have an extra SGL conversion here and it is unneeded, I am pulling that out right now.   It did not affect Memory statistics in Task Manager.