Waveform Arrays using massive amounts of memory

[NickG88]

Hello,

I'm having trouble with LabVIEW constantly running out of memory when I'm working with very large waveforms.

This is going to be a fairly long post, hopefully it doesn't scare people away, but I have no idea what's going wrong. I'll try to describe my situation as simply as possible. 

I'll try to explain the program I'm creating as best as I can. I created a program which reads a data file containing many waveform signals. The data files contain a couple hundred signals (some very short, some extremely long), and the files are usually 20-150MB in size. A subVI reads the file and takes out the signals the user wants and passes it to the main VI. The main program has a mixed signal graph on the front panel, and two enum controls which control which signals get displayed on the graph.

The subVI which reads the data file and loads the waveforms uses about 50MB of ram, but when I try to display the waveforms in the graph, LabVIEW uses 700MB-1.5GB of ram! The amount of memory used explodes when I try to index the waveform array and display the graphs on the front panel. The program has no trouble reading the small files, but it crashes when I try to load the larger waveforms. The biggest data set I have tried using has three waveforms with 2 million data points and six waveforms with 200k data points, for a total of 7.2M data points. With 7.2M double data points, shouldn't the program only use like ~58MB of ram?

Originally I thought the problem was because I was calling the graph using local variables a lot. I've read most of this article on memory usage and drastically reduced the number of local variables in my code. I added "request deallcation" to the end of all the subVIs which only run once during the program to reduce the memory. This has helped noticeably with smaller files, but the program still crashes when I try to load large files.

I can't post my main code, but I created a tiny VI which shows my memory issue pretty well. Here are some pictures:

Sample data before doing anything. It only takes up 366kB of space on the harddrive (which seems really small...). Right now the VI contains nothing but this array of waveforms. The first three waveforms contain 2 million data points, and the next six contain 200k data points, to accuratly mimic the data I'm having trouble with:

When I try to display the waveform with a front panel indicator LabVIEW takes 722MB of ram and the VI uses about 168MB of ram:

I want to display these as a graph though. So if I index the array and display one graph, the VI uses just slightly less memory than it did when it was displaying the entire waveform array indicator. The peak memory is much lower though. I guess because it only looks at the first waveform in the array rather than all of them?

I'm using the VI "Index Waveform Array" because I call some of the waveforms using a string rather than a number. The order the waveforms are read in is not fixed, and in order to do some of the numeric analysis I need to read certain signals, so I index them by name. I tried using both numbers and strings to index the waveforms in this test, but both had the same results:

What I really want to do is display four signals on a mixed signal graph. When I try do do that, LabVIEW uses about 800MB of ram and the VI uses about 430MB of ram:

Just displaying the data on the front panel uses ~700+ MB of ram. When I start to pass the data to subVIs to do data analysis, the amount of memory used can shoot up to 1.5GB, at which point my computer runs out of memory and LabVIEW stops working..

So... Why does LabVIEW use so much memory when displaying these waveforms?

I don't understand how apparently 366kB of data can take up SO much memory.

Any suggestions on how I can make this use less memory?

I'm beginning to think that the problem lies in indexing the arrays for some reason... if I pass the entire array of waveforms into a subVI but don't do anything with it, the subVI doesn't use much memory. However if index the array in a subVI, that subVI will also use ~100+MB of ram. Does the Index Waveform VI create copies of the waveform array?

Also, a side question: does a local variable of a graph create a copy of it in memory like a local variable of a string or array does? Do property nodes create copies?

Any help would be greatly appreciated!

Thanks for your time,

-Nick

[nathand]

A graph is a lot like an array indicator, so yes, using a local variable will generate an additional copy of the data.

If you haven't done so yet, search this forum for posts about inplaceness, or possibly just "in-place," which should help you understand when LabVIEW generates copies of data.

If you pass an array into a subVI and then return the same array unchanged, LabVIEW needs no additional memory because nothing happens to that array.  Let's say you then modify the VI so that it changes the array.  If the calling VI no longer needs the original array, then your subVI can operate on the original array "in-place" and doesn't need to make a copy.  However, if the calling VI does need the original array, then LabVIEW makes a copy of the array before calling the subVI.  If you have the front panel of the subVI open you may also generate additional copies of the array because LabVIEW will maintain a copy of the original array to display in the front panel control, and an additional copy in the output indicator.

It would help a lot of you could post parts of your code.  The "In Place Element Structure" may also help you, but it takes some experience to understand where it can help you.

You could probably save memory by decimating your waveforms before you display them on a graph.  Unless you need to zoom in on a very small portion of the graph, there's no way you'll ever need to display even 200k points of data, since your monitor is probably 2k pixels across at best.

[altenbach]

One problem is also your use of complicated data structures since all your array elements have potentially vastly different sizes. This means with each indexing operation, the indexed out element needs a new memory allocation because the existing allocation cannot be re-used.

If you use subVIs, make sure to keep the front panel closed and ensure they don't contain code elements that force the front panel to be in memory. If the panel is in memory, the controls and indicators also contain data copies, of course.

I would love to see a simplified program containg typical data. What is your LabVIEW version?