topic Re: Memory usage/ memory leak in Measurement Studio for .NET Languages

Memory usage/ memory leak

pbrowntwi — Tue, 24 Oct 2006 21:22:04 GMT

Are there any know problems with using 8.0.1 and releasing memory ?

We currently have an application that creates lots of graphs, however it eats memory and never seems to release it. So I've tried to create a small test app that shows the problem and found that doing ClearData (which I assumed would remove data from memory), and then doing a manual GC collect left residual memory behind. This was a basic form with a scatter plot and 2 buttons that added 210,000 items to the plot. (We typically have 15 plots with ~210000 points each on them in 7 lines).

This means that when I first add the data to the plot the memory jumps by ~14Mb, removing the data drops the memory back by ~2mb and then seems to stabilise aroung that number. Is there something else that needs to be done to free up memory ?

Also is the a list of know issues with 8.0.1 somewhere ?

This is the guts of the code:

        private void button1_Click(object sender, EventArgs e)
        {
            Random rnd = new Random();
            List<double> x = new List<double>();
            List<double> y = new List<double>();
            for (int i = 0; i < 210000; i++)
            {
                x.Add( i );
                y.Add( rnd.NextDouble() );
            }
            scatterGraph1.PlotXY(x.ToArray(), y.ToArray());
        }

        private void button2_Click(object sender, EventArgs e)
        {
            scatterGraph1.ClearData();
            GC.Collect();
            GC.WaitForPendingFinalizers();
        }

Thanks

Paul

Re: Memory usage/ memory leak

Mike J — Fri, 27 Oct 2006 23:45:25 GMT

I am having the same problem. I have tried ClearData(), and tried clearing plots and axis. I also tried to force the GC, but when I used the Ants memory profiler, it became obvious that some internal NI classes are holding data. It looks like NI obfuscated some of the internal classes and methods so it is hard to get a clue as to what is going on. It is possible that some collection classes are being held by events. When an object gets tied to an event, it is not garbage collected unless it is manually removed. It is a commong mistake, and there are several articles on the net about solutions to this problem.

Can someone at NI look at their code and see what is going on? My app can't ship until this is fixed.

Mike

Re: Memory usage/ memory leak

breeve — Mon, 30 Oct 2006 17:00:37 GMT

When you plot data on a plot for the first time, we (Measurement Studio) cache screen points and other things so drawing will be faster. The plot object holds on to this data. When you call ClearData, the plot only clears the raw data and not the screen points etc associated with the data (we will look into this).

If you plot data over and over in a loop the memory remains constant. Each time new data is plotted the old data is released and when it comes time to draw the new screen points the old screen points are released. I verified this with the code above by plotting the data in a loop.

Because calling ClearData does not clear the screen points (and other cached values), we need to have the plot object get garbage collected. Here is some code that works. Let me know if this works out for you.

public partial class Form1 : Form
    {
        private ScatterPlot _plot;

        public Form1()
        {
            InitializeComponent();
        }

        private void button1_Click(object sender, EventArgs e)
        {
            Random rnd = new Random();
            List x = new List();
            List y = new List();
            for (int i = 0; i < 210000; i++)
            {
                x.Add(i);
                y.Add(rnd.NextDouble());
            }

            _plot = new ScatterPlot(xAxis1, yAxis1);
            scatterGraph1.Plots.Add(_plot);
            _plot.PlotXY(x.ToArray(), y.ToArray());
        }

        private void button2_Click(object sender, EventArgs e)
        {
            if (_plot != null)
            {
                scatterGraph1.Plots.Remove(_plot);
                _plot.Dispose();
                _plot = null;
            }

            GC.Collect();
            GC.WaitForPendingFinalizers();
        }
    }

Re: Memory usage/ memory leak

Mike J — Tue, 31 Oct 2006 17:24:00 GMT

Calling Dispose() seems to work. I was already clearing the plots, but I was not calling Dispose().

Thanks,

Mike

Re: Memory usage/ memory leak

Sof — Tue, 31 Oct 2006 17:59:08 GMT

breeve,

I had a question for you. What kind of performance do you get plotting 210000 points on a plot? We had such bad performance we had to go back to the ActiveX control.

Scott

Re: Memory usage/ memory leak

pbrowntwi — Tue, 31 Oct 2006 20:54:21 GMT

Performance isn't too bad, we only have problems when adding annotations to the charts and then it goes like treacle.

Re: Memory usage/ memory leak

breeve — Thu, 02 Nov 2006 17:52:49 GMT

ScatterPlot

Here is what I get from running the code below using the Stopwatch class. I set both axes to fixed; [0-1] for the YAxis and [0-210000] for the XAxis at design time. I timed both the ImmediateUpdate case (Graph.ImmediateUpdates=true to time the drawing as well) and non-immediate update case, which is the default. I timed the code 4 times for each and got the average, throwing out the first time.

ImmediateUpdates=false (default)

.0637 seconds

ImmediateUpdates = true (times the drawing as well)

1.2185 seconds

SpeedPlot

Now, I was curious if I could make that time faster by not copying the data. When you call ScatterPlot.PlotXY, all your data is copied to an internal buffer. We do that to maintain the integrity of the plotted data as well as to support charting. I created a SpeedPlot object in the code below. This keeps a reference to the data and maps and draws the points itself, keeping the same timing process as above the following times where obtained:

ImmediateUpdates=false (default)

.0000755 seconds

ImmediateUpdates = true (times the drawing as well)

.9413 seconds

Now before we get all excited about the speedup in the ImmedateUpdates=false case, we need to remember what we are actually timing here. All we are timing is a field set of an address. This does not take long to do. A more accurate estimate of the speedup would be the second case where ImmediateUpdates=true because this times the drawing operation as well. Also remember that with the SpeedPlot, I am doing all the drawing of the plot myself. This means certain features like PointStyles, LineStyles, etc will not be effective. This plot is purely for speed where speed is really needed. I don't recommend using it for anything else, because certain features of the graph will not work.

Give it a try.

 public partial class Form1 : Form
    {
        private SpeedPlot _plot;
        public Form1()
        {
            InitializeComponent();
            _plot = new SpeedPlot(new Range(0, 210000), new Range(0, 1), xAxis1, yAxis1);
            scatterGraph1.Plots.Add(_plot);
        }

        private void button1_Click(object sender, EventArgs e)
        {
            Random rnd = new Random();
            List x = new List();
            List y = new List();
            for (int i = 0; i < 210000; i++)
            {
                x.Add(i);
                y.Add(rnd.NextDouble());
            }
            double[] xData = x.ToArray();
            double[] yData = y.ToArray();
           
            Stopwatch watch = Stopwatch.StartNew();
            //switch line below to _plot from scatterGraph1 to test NoCopyPlot.
            scatterGraph1.PlotXY(xData, yData);
            //_plot.PlotData(xData, yData);
            watch.Stop();
            textBox1.Text = watch.Elapsed.TotalSeconds.ToString();
        }

        private class SpeedPlot : ScatterPlot
        {
            private double[] _xData;
            private double[] _yData;

            public SpeedPlot(Range xRange, Range yRange, XAxis xAxis, YAxis yAxis)
                : base(xAxis, yAxis)
            {
                XAxis.Mode = AxisMode.Fixed;
                YAxis.Mode = AxisMode.Fixed;
                XAxis.Range = xRange;
                YAxis.Range = yRange;
            }

            public void PlotData(double[] xData, double[] yData)
            {
                _xData = xData;
                _yData = yData;
                //Need to redraw new points.
                Invalidate();
            }

            protected override void OnBeforeDraw(BeforeDrawXYPlotEventArgs e)
            {
                base.OnBeforeDraw(e);
                if (_xData != null && _yData != null)
                {
                    PointF[] points = MapDataPoints(e.Bounds, _xData, _yData);
                    using (Pen pen = new Pen(LineColor))
                        e.Graphics.DrawLines(pen, points);
                    e.Cancel = true;
                }
            }
        }
    }

Re: Memory usage/ memory leak

bilalD — Fri, 03 Nov 2006 15:48:36 GMT

I wanted to add that there is a large performance difference between using GDI (native Win32 drawing API) vs. GDI+ (.NET drawing API). GDI+ lacks hardware support and with Vista coming, it doesn't look like video drivers will be adding support for it anytime soon. On the other hand, video card drivers do provided hardware acceleration for GDI.

GDI+ is what we use for all of our .NET UI controls since these are managed controls and GDI+ is the Microsoft recommended managed drawing API. We use GDI for the ActiveX controls which is why there is a difference in their performance.

Re: Memory usage/ memory leak

pbrowntwi — Sun, 03 Dec 2006 12:43:39 GMT

I don't know if I'm missing something very obvious, however I've been trying to get the charts to dispose correctly and ended up comming to the conclusion that the only way to do this is to unsubscribe all events manually. i.e. if I attach a handler to say the AfterMove event of a cursor, before disposing of the the form I end up having to do a -= on that cursor.

I've ended up finding this out by using Son of strike and getting gcroot on the undisposed form and finding the following

DOMAIN(0014C0C0):HANDLE(Pinned):ba13ec:Root:02564b68(System.Object[])->
0166b720(System.ComponentModel.WeakHashtable)->
01764618(System.Collections.Hashtable+bucket[])->
0166b9f4(System.ComponentModel.TypeDescriptor+TypeDescriptionNode)->
0166b9e0(System.ComponentModel.ReflectTypeDescriptionProvider)->
0166ba34(System.Collections.Hashtable)->
0195a36c(System.Collections.Hashtable+bucket[])->
019755c0(System.ComponentModel.ReflectTypeDescriptionProvider+ReflectedTypeData)->
0197a710(System.ComponentModel.PropertyDescriptorCollection)->
0197a68c(System.Object[])->
01979e9c(System.ComponentModel.ReflectPropertyDescriptor)->
019802f0(System.Collections.Hashtable)->
01c33f24(System.Collections.Hashtable+bucket[])->
01b5ad28(NationalInstruments.UI.XYCursor)->
01b5ad68(NationalInstruments.Restricted.CallbackManager)->
01b5ad7c(NationalInstruments.Restricted.MultiKeyDictionary`3[[System.Object, mscorlib],[System.Delegate, mscorlib],[NationalInstruments.Restricted.CallbackManager+a, NationalInstruments.Common]])->
01b5ad88(System.Collections.Generic.Dictionary`2[[System.Object, mscorlib],[System.Collections.Generic.Dictionary`2[[System.Delegate, mscorlib],[NationalInstruments.Restricted.CallbackManager+a, NationalInstruments.Common]], mscorlib]])->
01b5b0c8(System.Collections.Generic.Dictionary`2+Entry[[System.Object, mscorlib],[System.Collections.Generic.Dictionary`2[[System.Delegate, mscorlib],[NationalInstruments.Restricted.CallbackManager+a, NationalInstruments.Common]], mscorlib]][])->
01b5b82c(System.Collections.Generic.Dictionary`2[[System.Delegate, mscorlib],[NationalInstruments.Restricted.CallbackManager+a, NationalInstruments.Common]])->
01b5b878(System.Collections.Generic.Dictionary`2+Entry[[System.Delegate, mscorlib],[NationalInstruments.Restricted.CallbackManager+a, NationalInstruments.Common]][])->
01b5b7e8(NationalInstruments.UI.BeforeMoveXYCursorEventHandler)->
01becab4(Teletest.Views.FrmGeneralDispersionCurve)
DOMAIN(0014C0C0):HANDLE(WeakSh):ba290c:Root:01becd6c(System.Windows.Forms.Control+ControlNativeWindow)->
01becab4(Teletest.Views.FrmGeneralDispersionCurve)
DOMAIN(0014C0C0):HANDLE(WeakLn):ba312c:Root:01c4ce2c(System.Windows.Forms.NativeMethods+WndProc)->
01becd6c(System.Windows.Forms.Control+ControlNativeWindow)

I can only assume that this one is left as I add it manually on the form, I remove all of the +='s that are assigned in the designer before truing to dispose and have the following method to remove everything else

        internal static void Remover(ScatterGraph graph)
        {
            while (graph.Plots.Count > 0)
            {
                ScatterPlot plt = graph.Plots[graph.Plots.Count - 1];
                graph.Plots.RemoveAt(graph.Plots.Count - 1);
                plt.Dispose();
                plt = null;
            }

            while (graph.XAxes.Count > 0)
            {
                XAxis xa = graph.XAxes[graph.XAxes.Count - 1];
                graph.XAxes.Remove(xa);
                xa.Dispose();
                xa = null;
            }

            while (graph.YAxes.Count > 0)
            {
                YAxis ya = graph.YAxes[graph.YAxes.Count - 1];
                graph.YAxes.Remove(ya);
                ya.Dispose();
                ya = null;
            }

            while (graph.Annotations.Count > 0)
            {
                Annotation ann = graph.Annotations[graph.Annotations.Count - 1];
                graph.Annotations.RemoveAt(graph.Annotations.Count - 1);
                ann.Dispose();
                ann = null;
            }

            while (graph.Cursors.Count > 0)
            {
                XYCursor csr = graph.Cursors[graph.Cursors.Count - 1];
                graph.Cursors.RemoveAt(graph.Cursors.Count - 1);
                csr.Dispose();
                csr = null;
            }
        }

Is this really required, or is there a way to have the graph dispose correctly ?

Also is there a timescale for a patch to address current issues ?

Thanks

Paul

Re: Memory usage/ memory leak

breeve — Mon, 04 Dec 2006 16:24:15 GMT

Can you post a simple project? Some details are unclear.

Are you trying to Dispose the entire form and then recreate another form?
What version of MStudio

Re: Memory usage/ memory leak

Abhishek Ghuwalewala — Mon, 04 Dec 2006 16:31:28 GMT

The implementation of your Remover method is already built into the graph. The graph will dispose all its sub-objects such as plots, cursors, axes, and annotations when the graph is disposed, so you should not have to do this manually. It is essential to remove any event handlers when disposing objects. After removing your event handlers there should not be any memory being used by the graph.

Re: Memory usage/ memory leak

pbrowntwi — Mon, 04 Dec 2006 16:55:03 GMT

So even if I explicitly call dispose on the graph object I still need to track every event that has been bound during it's lifetime and unsubscribe them ?

This seems like something the graph should do itself (especially if it's events that have been bound in the designer)

Re: Memory usage/ memory leak

pbrowntwi — Mon, 04 Dec 2006 17:25:21 GMT

This attachement shows the memory creep being experienced (pressing Creator + Once + Destroy should give a baseline for running multiple times, however if I then press 200 times my memory goes from 26Mb upto 35Mb

and the highlights of my stackdump are

03d2efd4      201        24120 NationalInstruments.UI.Internal.b3
03d2e8b8      201        24120 NationalInstruments.UI.Internal.do
03d2ed5c      201        24924 NationalInstruments.UI.Internal.m
049fd108      201        25728 NationalInstruments.UI.Internal.b2
03c1ccfc      201        26532 NationalInstruments.UI.WindowsForms.Legend
03d2e36c      201        28944 NationalInstruments.UI.Internal.r
7a74d9c8      247        29640 System.ComponentModel.ReflectPropertyDescriptor
03c1742c      201        31356 NationalInstruments.UI.WindowsForms.WaveformGraph
049fcd94      201        32160 NationalInstruments.Restricted.LegendManager
7912c444        1        40016 System.Double[]
03d2d314      402        51456 NationalInstruments.UI.Internal.aq
03d2b160      402        51456 NationalInstruments.UI.Internal.cz
03d2ae20      402        51456 NationalInstruments.UI.Internal.ag
03d24dac      201        57888 NationalInstruments.Restricted.WaveformGraphManager
790fa3e0      932       214284 System.String
79124228      513       306324 System.Object[]
0018abf0       19       483648      Free
791242ec     2867      1069416 System.Collections.Hashtable+bucket[]

The finalizer is commented out but was being called for each instance.

Memory will eventually come back but it takes time (almost as if there's a timer holding onto it) and then it'll drop back to ~32Mb which is still a 6Mb increase (measured using Process Explorer).

Can someone tell me what I'm doing wrong ?

Thanks

Regards Paul

Re: Memory usage/ memory leak

Abhishek Ghuwalewala — Mon, 04 Dec 2006 19:08:02 GMT

We have looked at the project you have provided. I am going to file a bug report for the issue. I will let you know when the problem has been investigated and fixed.

Re: Memory usage/ memory leak

pbrowntwi — Mon, 04 Dec 2006 20:13:55 GMT

Is there a viewable list of currently know bugs/issues anywhere ? We are nearing shipping and it would be good to know of any other issues that are likely to bite us, annotations were the previous problem to hit us and now this one...

Thanks

Paul

Re: Memory usage/ memory leak

Abhishek Ghuwalewala — Mon, 04 Dec 2006 20:53:40 GMT

Yes. The list of known issues at the time of release are documented in the readme. You can find the readme through your Start menu

Start >> All Programs >> National Instruments >> Measurement Studio 8.0.1 for [Visual Studio .NET 2003] | [Visual Studio 2005] >> Readme

Re: Memory usage/ memory leak

pbrowntwi — Mon, 04 Dec 2006 21:05:10 GMT

I was more interested in issues that have been found since release.

Paul

Re: Memory usage/ memory leak

Abhishek Ghuwalewala — Sat, 09 Dec 2006 01:06:51 GMT

The problem has been investigated and fixed. The fix will be available in the next release of Measurement Studio after 8.1. Thank you very much for reporting the issue and providing us with code to reproduce it.

Re: Memory usage/ memory leak

pbrown — Sun, 10 Dec 2006 10:04:39 GMT

What is the timescale for this ? As I said we are about to ship a product and it has much more serious consequences in that than in the test application.

Re: Memory usage/ memory leak

bilalD — Mon, 11 Dec 2006 23:39:43 GMT

Hello

We're going to release a patch for this issue. The patch will be available in a week or two.

Sorry for the inconvienience.