How do I keep LabView from constantly allocating memory unnecessarily?

[kayotik0987]

I have lots of raw data collected from a wind tunnel test that needs to be used to calculate useful engineering data.  Typically the data is averaged and then reduced once, but this test needs to reduce each sample.  I wrote a LabView (7.1) VI that reads each sample, reduces it and saves it.  This is not a fast process and even on a fast computer, it was taking about 8 hours to reduce a 30 Mb file, of which there are about 200 files.  Other than speed, the real problem is that after one or two files, LabView has allocated all the memory and paging file space and the program terminates with the error message: Memory is full.  I have included the Request Deallocation VI in all my subVI's, but it doesn't seem to be working.

Is there any other way to try to manage LabView's memory allocation?  I know I could re-write the program in another language such as C++ and not have this issue, but it would take significant time and effort to translate the reduction code.  I have also tried to streamline the VI such that I'm not repeating tasks unnecessarily.

-Warren

[tst]

LabVIEW memory management is fairly complex, but the most frequent reasons:

• Resizing operations (Build array, Delete From Array, Concatenate String) create a new buffer. This is especially bad inside a loop.
• Locals and globals cause copies.

The way to get around these is to know how to create your buffer once and reuse it (for instance, using shift registers, for loops or Replace Array Subsets).

I can't think of any operation you could perform on a 30 MB file which would take that long. I would expect that it would take no more than 20-30 seconds at most.

If you can upload an example of your code, we should be able to help with more specific details.

If you want more details, you should read app note 168.

[Ben]

I have to second tst's comments and add

"... a 30 Mb file, of which there are about 200 files. "

If you mean that there are 200 files that are of size 20Mb, you are talking about 6Gig of raw data.

Windoze can barely handle 2Gig memory spaces so you will have to do this in peices.

Please post your code to allow us to be of further assistance.

Ben

[kayotik0987]

Thanks for the quick reply!

A big part of the reason it takes so long is that the data is all ASCII and there is a lot of text parsing that occurs during the reduction of each sample of which there are about 45000 in that 37 Mb file.

I'm not using any local or global variables, and I've tried to be fairly efficient with the use of for loops, shift registers, etc.  I was looking through more of my code and found some subVI's that don't have the request deallocation VI in them.

I would post an example, but the reduction code is sort of proprietary.

-Warren

[kayotik0987]

So far I've only tried to crunch 2-3 files in a row, but you're right, I have about 10.3 GB of data in total!

[Ben]

" ... would post an example, but the reduction code is sort of proprietary."

That is your call.

Can you toss the "proprietary" stuff and post a repesentative example?

Ben

[altenbach]

If I understand the problem right, the only real problem is the parsing operations. Your ASCII file seems to contain a lot of hot air if a 30+MB file only results in 45000 data points. 😉

Parsing a file can be done in portions if there is a memory issue, but 37MB should be OK to handle in one operation. Make absolutely sure you are not creating extra data copies (e.g. don't add a string indicator for your 37MB initial string, because it would have another copy of the data). For the same reason, you might want to flatten the code out to the main diagram so you don't need to shuttle 37MB between subVIs. Make sure to do things "in place" as much as possible. Operate on the original string by moving the index point instead of e.g. splitting it after each atomic parsing operation and then continue with the "after" substring in a shift register.

I am no expert here, but I would think that the "request deallocation" feature might actually hurt you. If a subVI hangs on to its memory between calls it means that the next time you call it with an equal amount of data, it can reuse the existing allocation. If you constantly deallocate, you need to constantly reallocate at the next call! (I have never played with this, so please somebody correct me if my understanding is flawed).

How complicated is the parsing operation? Are you basically moving linearly through the file or do you need to jump all over it.

[Ben]

" ... I would think that the "request deallocation" feature might actually hurt you. If a subVI hangs on to its memory between calls it means that the next time you call it with an equal amount of data, it can reuse the existing allocation. If you constantly deallocate, you need to constantly reallocate at the next call!  ..."

I'll confirm that.

The request allocation is good if you dynamically load a memory pig VI to do some work and then unload it after. I recall it only attempted to deallocate when the VI is removed from memory (at least the last time played with it).

Ben

[Jarrod_S.]

Here and here are a couple good links with really useful info and tips and tricks for managing large sets of memory efficiently in LabVIEW. Seriously, good reading. Hope it helps...