Memory leak Waveform Graph

[ohmjanssen]

Hi,

I have written a library (dll) in C# to do measurements using a NI acquisition system and store the data to disk. I have also written a GUI application (windows form) in C# to load the acquired data for analysis. From this application I open a new form application containing a WaveformGraph (Measurement studio version 8.1.1.377). A button is available on this second form to load data and create plots in a single graph.

During normal usage of the data is loaded multiple times and the WaveformGraph is zoomed and panned for analysis. However, the application crashes often due to a 'system out of memory exception'.

One of the things I observed is that when I open the second form, load data into the graph, then close the form and actively call the garbage collection, not all memory is freed.

For this situation I created a simple project (see attachment). The project consists of a main form with 2 buttons; 1 to spawn a second form and 1 to force garbage collection. The second form contains a WaveformGraph, a button to load data into the graph (10000000 dummy samples), a button to clear the graph and a button to force garbage collection.

The picture below shows what happens in 8 steps:

1. The main application form is started followed by opening the second form.

2. On the second form, the load button has been pressed to fill the graph.

3. On the second form, garbage collection is forced.

4. On the second form, the graph is cleared and garbage collection is forced.

5. Step 2, 3 and 4 are repeated.

6. The second form has been closed followed by forced garbage collection using the button on the main form. Thereafter the second form is reopened and steps 2 till 4 are repeated

7. Step 6 is repeated.

8. Both the second and main form are closed.

When step 6 is repeated multiple times at some point the 'system out of memory exception' will be thrown because there is not enough free memory left.

My question is, how do I completely free the memory used by the WaveformGraph (other than killing the full application)? Is this an issue in the used version of measurement studio (8.1.1) that has been solved in a newer version or am I doing something wrong?

Thanks for your advice.

Kind regards,

Oskar

[ohmjanssen]

No advices on how to fix / workaround this?

[IDCDeveloper]

In my application I had a memory issue too, It is very easy to use whole RAM nad virtual memory when you showing dozens of plots with millions points each. However my experience shows that NI library works without problems. Try to think, may be you don't erase all "connections" to the objects that you don't need anymore, before you launch GC. For exaple, dont forget to do the following:

foreach(var plot in _waveformGraph.Plots)

plot.ClearData();

or

_waveformGraph.Plots.Clear();

or

_waveformGraph.ClearData();

whatever suits your needs.

And then when you call GC, call it like this:

GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced);

It is the most time consuming, but the most resultative also.

[D_Biel]

I took a look at this example I could not reproduce a memory leak. However, I did see a lot of churn in the memory the application. But this seems in line with what I would consider normal for a .NET app. Part of the problem may be with the way the .NET garbage collector works. When you call the gc.collect, it does not necessarily collect all unreferenced objects. The .NET garbage collector uses "generations" to group objects based on likely they are to be unreferenced objects. The garbage collector uses a heuristic based on how many garbage collections the object has survived as an estimate of how likely the object is to be a long term object. This saves time by not considering older objects when doing garbage collection. Because of this, you sometimes need to force the garbage collector to consider these older objects. As IDCDeveloper suggests, call GC.Collect(GC.MaxGeneration, GCCollectionMode.Forced); to tell the garbage collector to consider all objects, and not just the youngest generation.

The reason this may help is because what may appear like a memory leak could just be that some objects that are no longer referenced are stuck in a higher generation. Eventually, the garbage collector would get to these on its own, but in the short term, it may cause spikes in your memory usage. What we really want to pay attention to is if you ever receive an OutOfMemory Exception. Before throwing this exception, the garbage collector will attempt to collect all generations (essentially calling the function IDCDeveloper suggested). If you receive this exception and are not currently using that much memory, then there is likely a memory leak.

I would suggest trying the suggestions from IDCDeveloper and see how the memory performs. Additionally, the Windows Taks Manager is not always the most helpful way to view memory usage of your system. I would recommend using ProcessExplorer which can provide more tools for monitoring memory on your system (http://technet.microsoft.com/en-us/sysinternals/bb896653).

[andreacer]

Good Mornig. I experieced the same comportament with my application. I use 3 waveformgraph with ten million points each. The points are obtained by the current acquisition (Dqa 6210) ad appended to the graphs. (10.000 samples/s). I am looking for rare events and i ought to view all the evolution.

Every time I append a packet of data, the memory will rapidly decrease and in certain situation (zooming ecc) the system crash.

(I have windows xp with 2 Gbyte RAM).

I will try the system you suggested and report the results.

[ohmjanssen]

Thanks all for the suggestions to solve the issue. I still have no leak free implementation, but with some explicit clearing of plot data and thorough garbage collection it is acceptable at the moment (but not fully satisfying).

The biggest problem is not just closing and reopening the form, but crashes after repetitive loading of graph data. I attached a screenshot in which my application crashes with a 'system out of memory exception' after 3 times succesfully loading data (ignoring the first cycle). It is clear that in this situation there is no excessive use of RAM.

[ohmjanssen]

One more thing worth noticing; I recently built the application in the first post against version 8.9 of measurement studio and that solves that memory leak (!). I still have to test the effect on my target application, maybe it solves the remaining issues described above too. Only 1 problem; I don't own the 8.9 license so I'd like to solve it with 8.1 if possible.

[IDCDeveloper]

Well... I can give you two more advices, however the require a lot of job to be done. But they work, and actually this is what you have to do from the beginning. We all work under time requirements and it makes most of a problems. But if you have time - you can solve your problem by doing the following:

1) Think of important fields in your graph containing user control, that must by initialized. Write good initialization function and understand the code flow. Then catch out of memory exception. Write those important field values to some external object. Dispose your graph containing control and create and initialize it from the beginning. Do you understand what I mean? Don't hesitate to ask if not. Now look, actually this technique is quite general and it might be a good idea to create ExtendedUserControl class, that inherits from UserControl and contains all this trick. And then all your controls should inherit from him and all of them will be bulletproof to OOM exceptions. A lot of work, but works.

2)It is old issue and a lot is already written on this. But again. You have your plot area for example 100x100 pixels. And you want to plot 1000000 points on it? This makes 1000 points at list falling on one pixel, meaning they will not contribute to user's understanding. I say you have to go and choose about only one point from every 1000 and display it. Few things that you have to do are:

•  You keep your data aside from the graph. Meaning you have some array of float-pairs for points on the graph. This array contains all points and you add points from your source to it, not straight to the graph.
•  You will need to find good algorithm for choosing "good" points. For example you have to take Min and Max from every 1000. There are many such algorithms written.
•  You have to do some things while zooming. Zoom and pan actions change the set of points that you have to choose good points from. So you have to calculate how many points you have to take now. Example: Let's assume the plot area contains 1000 pixels. So we want to show 1000 points as maximum. We have 1M points and we are in full zoom (seeing whole graph), so we divide 1M by 1000 and have 1000 as a result. This means that from every 1000 we have to choose one good point. Now we decide to zoom by two. Meaning we will see only half of the graph. So we have 500000 visible points that should be shown on 1000 pixels. This gives us the result that we have to choose 1 good point from every 500 points... So lets clean old plot from the graph and lets insert new good point set. Yep, a lot of work, as I have told you from the beginning.
• If you are brave enough to try to implement this, don't harry to erase old code! You will need it. Those calculations are time consuming. All this optimizes our application only if the number of points is much larger that the number of pixels. So you have to have both code flows in your programm. Having some data set that you have to show on the graph you have to calculate this points to pixels ratio and decide at runtime whch flow are you going to use.

Well... that's it. I bet you say "No way! I will better burn in hell for irresponsive application coding, but I will note implement this!" 🙂
It is your job, your code, your time deadlines.... you decide. But good luck anyway!
Regards.

[andreacer]

I Spend a night to test waveformgraph and my problem seem resolved in this way:

1) initialize the waveformgraph obj with the maximum value of plots/point per plot (It is necessary because i use plotappend).

I dont use immediateUpdates but it can help.

The principal channals use all the data, the others use a decimated version.

2) AFTER plotappend, zoom ecc, I call this simple function:

void FreeMem

{long TotalUsed = GC.getTotalMemory(true);

if (TotalUsed > MyMax)

{System.GC.Collect();

System.GC.WaitForPendingFinalizers();

}

note that funct is time expensive and have priority -  if you have realtime acq you can loss data.

Place it with care. ( I call it between two readData  and use a data buffer).

I have : Xp/VisualStudio10/Measurement studio 9.1.0.204

nb if you print TotalUsed every second, it seems that Zoom, Pan and every change in xAxis.Range are memory exensive.

Memory is used and immediatly released but if you have multiple channals you can have OOM. 

A stupid solution maybe (next version of Measuement):

- use a <float> version of PlotAppend instead of <double>  buffer for waveformgraph - when you plot 10 million data the difference can't be seen;

- separate the updates of multichannels plots in order to optimize memory swaps.

I hope it can works on your jobs. 

[ohmjanssen]

Once again, thanks for the good advice.

However, I found something else interesting that might be useful to others as well;

I use annotations in my graph and originally I updated them every time I loaded new data into the graph. This involved creating a few hunderds of new XYPointAnnotation objects each time. When I disabled this functionality my System out of memory exeptions where gone! Next step was to only initialize a List of annotations at start-up and then reusing these annotations whenever new data is loaded. Trial and error learned me that I can only initialize a few 1000 annotations before the System out of memory exception pops up. For my graph, only a few hundred are needed, so that is not an issue.

Note that I saw this issue working with version 8.1 of the Measurment studio libraries. Other versions may not show this behavior.