Reading a table causes memory usage to skyrocket?

[E30Alex]

Hi, I have a program here that is used for data acquisition. Everytime the user presses a button, it records a set of info onto a table. The problem is that when the table gets to the size of about 2Mb, the computer starts acting up.

I checked the memory usage, and for every megabyte that the data table is, the ram used is 90 megabytes! So by the time the table, hits 2 megabytes, the computer that I am running this program on is out of memory.

Is this a common problem?

I included a little test program to show what happens. When I open up this program, memory usage is 26 megabytes. But when I load the 250k data file, memory usage jumps up to 34 megabytes.

Alex

[Andre_Saller]

You can not compare the amount of memory data takes up in a file and in an application. You read the 250k Ascii file into a 1656 x 96 2D-Array of Strings, these are nearlyx 100000 single strings. Every string has some overhead, because LabView stores not only the single characters, but also the length. The same counts for the array. So this 2D array will have alredy some MB, because the data is not in the same way stored in LabView memory as it is on the harddrive.
Second, the Table on the frontpanel will use a second buffer of this array, so it has a complete copy of this array in memory, which will double the amount of memory labview needs again..
So in general this is how LabView works, and you could see similar memory usage in other programming languages.

To reduce memory usage, you could reduce the size of the array, and only load the data you are interested in.

Hope this helps!

[shoneill]

In addition, by not programming properly, you can end up with a LOT of unneccessary copies of the data in memory.

I would imagine this is what's happening.  Have you actually looked at the REAL size of the data in the table?  Try flattening to a string and getting the string length.  This will tell you approximately the size of the table in memory.  Everything else is most likely unneccessary copying of the data.

Shane.

[chrisger]

hi there,

please check the first line in your datafile. there's a EOL missing at the first set of data. this will lead to an extra number of (empty) cells in your table.

[chrisger]

there's also a number of tabs at the end of your lines. remove them to minimize memory usage. i blew up your file to 32.000 lines, when loaded with the original header memory usage was ~190MB, with a corrected header and removed extra tabs usage was ~100MB.

[chrisger]

it seems to me that a MultiColumnListbox uses less memory than a table (use "ItemNames" property to set cell values)

[E30Alex]

I understand that improper programming will get you tons of unnecessary memory usage, but for that test program, all I have is a case structure inside a while loop. There are no local variables or anything of the sort.

I did flatten the table to see how much memory it uses and it says it is about 140% the size of the notepad file. So for the 2Mb file I'm loading, the table is only about 2.8Mb.

What I don't understand is why labview increases in memory usage by 50Mb for loading a 2.8Mb table. Is there any other way to get around this?

Thanks a lot,

Alex

[E30Alex]

Thanks for the suggestion. That definitely does decrease the file size, but I'd still like to be able to load a table over ~2Mb.

[E30Alex]

I just built a simple application to load the table. I also put an indicator in to tell me how big the table file is. I ran the application and loaded the table.

he memory usage jumped up 71.7Mb while the table is only 3.4Mb. That is a 21x increase. I can understand 2x or 3x, but 21?

[DFGray]

There are a lot of ways to create memory copies in LabVIEW.  Check out the tutorial Managing Large Data Sets in LabVIEW for help.  Let us know if you need more help.  You should be able to get your memory usage down to something reasonable.