Memory leak??? or more likely not knowing proper cleanup

[gnzork]

Hello

As the title suggests I think I have a memory problem.  I wrote a VI in LabVIEW 2017 to process data generated from a scientific instrument.  This VI does not interact with any hardware, so it's purely a software VI.  Unfortunately I cannot show the code because it is proprietary, but I can show some screenshots and I think describing the problem might be enough.

The VI lets a user load a data set into the program and interact with it, correct for noise, outliers, ect.  It can also load multiple data sets together which allows for better correction of noise, outliers, ect.  Each data set is roughly 2000x300 floating point numbers and when I load 300 or so data sets I have the following issues I'm hoping to address:

1) it takes a very long time to load.

Each data set is a text file which I'm loading into an array with "Read delimited spreadsheet".  Then I display them on one graph and change the line colors to black and the symbol to a hollow circle.  20 or so data sets takes about 8 seconds to load, which is no problem.  300 data sets takes forever.  The loading part is probably as fast as it can be, but the changing line color takes almost just as long as the loading because I have to do it for every curve.  I there a way to set the line color and symbol before loading multiple curves, that will stay for every new curve loaded?

2) When I clear the data I still show lots of memory being used.

I think I'm clearing the data, but when I do, task manager still shows most of my ram being used (my laptop has 12GB ram).  Only when I stop the VI does my ram usage go back to normal.  Is there something I should be doing to clear a variable and clear a graph other than writing an empty set to them?  For reference 300 data sets loaded as double floats take about 5GB of memory. 

Thank you in advance for any help!

1.png shows one data set loaded

2.png shows multiple data sets loaded

3.png shows multiple data sets after some processing

4.png shows my clear data function.

[mcduff · Accepted solution]

Without seeing code, hard to diagnose. Local variables make copies of the data so do indicators and controls. Keep your data in a shift register and modify in place. To clear the data use the resize array function, set it to a 0 element vale, again this should be done in place on the shift register. You can also decimate you data before you display it.

mcduff

[gnzork]

---

@mcduff wrote:

Without seeing code, hard to diagnose.

---

Well if it helps, here is the load routine for multiple data sets.  I know it's not the cleanest of code.

[AYanez]

I see that the program uses a lot of local variables, these could cause race condition and often causes data copies, my suggestion would be replacing local variables by a shift registers with a cluster that contains your data elements for those that are related to controls.

To graph the data you could have a look into queues and using a producer consumer pattern. Your high use of RAM is probably due the data copies caused by using local variables as data transportation mechanism, with 300 sets of  2000x300 doubles you should have a RAM consumption of 1.44 GB aprox but copies could be generated because of local variables, you need to change the way of pass the data in your program.

For User Interface updates you can use defer update panels, see: https://forums.ni.com/t5/LabVIEW/Example-for-Defer-Panel-Update/td-p/3688130

I hope it helps you.

[bseguin]

First of all,

Do not display all the data. the reason is simple... the display resolution cannot show all of them, therefore your eyes cannot see it. So you are using a lots of memory and a lots of graphic processing time for something you cannot show.

Benoit

[bseguin]

I see as well a stacked frame structure.

I believe that you do not have a proper dataflow programming technique.

I believe that you do not have a producer consumer structure neither.

I believe that you are creating multiple copy of the same data... (use functional variable) This way, you will reduce the number of copy of data. Remember every time you split the "wire" you are creating a copy of data.

Benoit

[bseguin]

Another thought, difer front panel update when loading all your data and formatting your graph. this will increase dramatically the performance.

Benoit

[aptivo]

Why you are using a timed loop structure? Simply use a while loop.

Are you using the 32 bit version or the 64 bit version of LabVIEW?

As the other users already discussed: don't use local variables when possible and do not display all data sets if not really necessary.

For a program with so many controls I would recommand the usage of a producer/consumer strutcure. 

A brief overview is given at:

Example 1

More advanced structure

[altenbach]

I assume you are using LabVIEW 64bit. Loading formatted text files is inherently inefficient, maybe you can convert them to flat binary to speed up loading.

It seems unreasonable to put that many points in a graph, you have significantly more points than pixels (!), do some decimation first.

Yes, all data belongs in a single shift register, no locals needed.

[MaxJoseph]

Hi,

It looks like you've got an interesting system on your hands! It looks like you've gotten to a point of complexity with your program that a more advanced design pattern would be really useful. Would I be right in saying that the code is getting hard to work with? Its hard to juggle so many balls at once! Each piece of your code is fine in isolation, but together the whole becomes more complicated than the sum of its parts. When complexity gets to this point, you need a framework to work within that will help you organise your code. Conviniently using these frameworks also improves processing speed.

The advice from others already given is really valid, I would highlight some things:

• A producer-consumer architecture would allow you to split the functions of your program up into modular components.
• Using shift registers and in-place elements rather than local variables and array indexing will reduce your memory requirements.
• Use a binary file type (TDMS) when you need fast read/write. 
• Consider decimating your data. Or present a curated view of the whole; eg a single plot with error bars to represent variance of the background data at each point might give a better indication of what you want to see than plotting every point.

Plotting is generally a bottleneck, defering updates is a good way to minimise plot updates but this misses the wood for the trees, I think. It is better to avoid plotting lots of data then make the plotting of lots of data faster.