64-bit LabVIEW - still major problems with large data sets

[Yamaeda]

Regarding Altenbachs solution, how fast are Get Variant atributes compared to normal variant reading and convert variant to 2D array?

Maybe it doesn't work with big datasets at all, but when i read variants my first thought was to build an array of variants, converted from those 2D SGL arrays. Insted of Get attribute you'd ofc need to use Convert Variant.

Would that work, or since it's an array it'd be back to basic?

/Y

[altenbach]

---

DFGray wrote:

Altenbach's solution also gives easy access.  Placing an array of clusters of arrays into a DVR will give you two or three layers of In Place Element structure every time you extract or manipulate anything.  I have a hard time seeing how this would be faster (although I have been surprised before...).  Try it and find out.

---

Damien,

I was worried that there is another buffer allocation for the full 3D dataset when we use the single DVR. Specifically because we need to access much more data in order to slice out the desired image. With N images, I only touch 1/N of the full data for each access. I have not done any benchmarking. Maybe I do some more tests tonight.

It seems we can avoid one extra single-image buffer allocation by placing the intensity graph terminal inside the inplace structure. (see image). Does this make sense? (Isn't it typically recommended to place indicators on the outermost diagram?)

[altenbach]

---

@DonRoth wrote:

Initially, I am using Damien's suggestion to use one DVR according to the following.

Is there anyway we can address this further regarding the IMAQ fxs (I need to use these versus the conventional labview fxs for opening picture images since the IMAQ fxs provide more image type options)?

---

Sorry, I don't have IMAQ (never used it), but I wonder why you are keeping the image datatype around at all. Would it improve if you would dispose the image right in the reading loop? (I was under the impression that image data is similar to a DVR). What kind of image manipulation do you actually need to do?

Alternatively, have you tried to just store the image set in memory instead of the SGL arrays? Just wondering...

[DFGray]

Christian,

The In Place Element structure should remove problems with extra copies as you burrow into a large DVR.  It gives you the ability to avoid data copies when dereferencing elements buried deep in cluster/array complexes.  I do not know if it is actually faster than dereferencing an array from an array of DVRs.  In theory, the new compiler should make it faster.  In practice...

I believe your extra copy is more or less expected.  As soon as you leave an In Place Element structure, you incur a copy penalty.  As long as you keep it inside the structure, you may not.  You may or may not incur a copy penalty in your example if you use it as a subVI.  Keeping controls/indicators on the main block diagram enables LabVIEW to more efficiently process them and keep them in-place when being used as subVIs.  However, I have run into a case before where I made fewer copies when putting the indicator in a structure rather than routing it to the main block diagram.  In my case, it was in a functional global buried two deep in case structures.

Finally, have you looked at memory use as you execute this?  You could have a hidden copy that only shows up when you execute.  Indicators will do that sometimes.

[DonRoth]

I am watching the task manager while the programs runs.  It is really odd.  With the status bar being close to completion, my ~ 1.3 gbyte worth of images is at around 5 gbyte.  Then right at the end, before leaving the write case, it seems to take a jump to over 9 gbyte.

Disposing of the Imaq Image reference in each loop ultimately did not make a difference.

As far as image manipulations go, at times in my real application, I have to do manipulations such as reversing rows, columns, decimating, etc.

It almost seems like we need a function to deallocate memory right after specific operations.

Christian, I assume you are asking about storing images on disk in a binary file.  The whole point of me going to the 64-bit OS and 64-bit LabVIEW was so I could handle large data sets more readily right in RAM.  But yes, that is the obvious fallback if this does not prove fruitful.

Damien - it is a little disconcerting that we can't know for sure how memory is being allocated and that it is trial and error.  I understand that this is one of the foundations for LabVIEW not to have to worry about this kind of thing.  But I would sure like to have finer control in applications like this, and have a list of hard and fast best practices.

Sincerely,

Don

[altenbach]

---

@DonRoth wrote:

Initially, I am using Damien's suggestion to use one DVR according to the following.

---

Don, you just have way too much code for what is needed. Creaating the DVR could be done much simpler, for example like this.

Also all these inner shift registers are not needed. FOR loops allocate output arrays perfectly even for autoindexing output tunnels, so try something along the lines of the rewrite. See if it makes a difference. As I said, I don't have IMAQ so I had to disable that stuff).

I have also rewritten my example with this simpler way (DVR4.vi).

---

@DonRoth wrote:

Christian, I assume you are asking about storing images on disk in a binary file.  The whole point of me going to the 64-bit OS and 64-bit LabVIEW was so I could handle large data sets more readily right in RAM.  But yes, that is the obvious fallback if this does not prove fruitful.

---

I don't think I was ever talking about using files. Could you quote what I said and I will try to find out what I meant. 😄

[DonRoth]

Christian - That reduces by 50% the memory used to about 4 gbyte.  That is a phenomenal reduction.  So we must have been allocating extra copies of the data using the 'replace array subset' strategy (which by the way I always thought was the best way to do this since you allocate the memory up front).

So it is likely that only 2 more copies of the data are being made now.  It would still seem prudent to have some sort of memory deallocation function that operates within VI that can be implemented right after where you know buffers are being created.  Then, for functions that perform array manipulations, we might be able to operate less cautiously.

I misunderstood your quote: "Alternatively, have you tried to just store the image set in memory instead of the SGL arrays? Just wondering..."

I need to use SGL for reasons that are beyond the scope of this thread at the moment.

Wonderful work you have performed here again and I think you profusely.  I think this is enough of an improvement, since I have 16 gbyte RAM, that this is worth implementing in my real application.  I'll try to determine whether further improvements can be made and anyone else is welcome to do this also.

Sincerely,

Don

[DFGray]

Don,

Unfortunately, the in-place algorithm is such a complex piece of code, it is virtually impossible to give a complete set of hard and fast rules.  And they would change as the algorithm evolves from version to version.  The game of "hide the dots" using the buffer viewer is usually correct.  However, the black dots will sometimes not show you buffer allocations due to controls or split wires.  It will also sometimes show you buffers which are technically there, but practically are not.  An example would be a buffer on the output of a single element queue.  It is there in case of error condition, and is an empty data structure taking up little space.  It is not a full copy of data.  Experience can teach which are which.  For general use, however, correct use of the In Place Element structure will prevent most data copies.  When in doubt, step through your code with a lot of data on the wire and watch how much data LabVIEW is consuming at each point.  You can separate controls from their wires by enclosing them in a single-frame sequence, allowing you to see what the control is costing you.

Let me know if there is anything else I can do for you.

[DonRoth]

Damien & Christian - If I just look at the 'Write' case, it is not clear to me what would be creating the additional data set copies.  The IMAQ ImageToArray fx and conversion to SGL fx show buffer allocation.  What is not clear to me is why the two fxs would cost us so much memory.  Aren't their memory spaces reused in each iteration of the for loop?

Maybe there are hidden copies being made.

Sincerely,

Don

[DFGray]

Memory should be reused from iteration to iteration of the loop.  The buffer allocations there may be a red herring.  Post a picture of your current code showing the buffer allocations and we can try to figure it out.  "Hidden" allocations usually occur when splitting wires or going into indicators.  They also have a tendency to show up in places you don't expect.

Out of curiosity, did you try using the IMAQ references directly and converting only when you needed to?  This would probably result in some slowdown, but the IMAQ reference is already a by-reference object, so might make your coding somewhat easier.  It may also preserve the internal compression of the image file, although I have never worked with IMAQ, so could not say for sure.